TEAM LinG

Encyclopedia of
Data Warehousing
and Mining

John Wang
Montclair State University, USA

IDEA GROUP REFERENCE

Hershey London Melbourne Singapore

TEAM LinG
Acquisitions Editor: Rene Davies
Development Editor: Kristin Roth
Senior Managing Editor: Amanda Appicello
Managing Editor: Jennifer Neidig
Copy Editors: Eva Brennan, Alana Bubnis, Rene Davies and Sue VanderHook
Typesetters: Diane Huskinson, Sara Reed and Larissa Zearfoss
Support Staff: Michelle Potter
Cover Design: Lisa Tosheff
Printed at: Yurchak Printing Inc.

Published in the United States of America by

Idea Group Reference (an imprint of Idea Group Inc.)
701 E. Chocolate Avenue, Suite 200
Hershey PA 17033
Tel: 717-533-8845
Fax: 717-533-8661
E-mail: cust@idea-group.com
Web site: http://www.idea-group-ref.com

and in the United Kingdom by

Idea Group Reference (an imprint of Idea Group Inc.)
3 Henrietta Street
Covent Garden
London WC2E 8LU
Tel: 44 20 7240 0856
Fax: 44 20 7379 3313
Web site: http://www.eurospan.co.uk

Copyright 2006 by Idea Group Inc. All rights reserved. No part of this publication may be reproduced, stored or distributed in
any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher.
Product or company names used in this set are for identification purposes only. Inclusion of the names of the products or
companies does not indicate a claim of ownership by IGI of the trademark or registered trademark.

Library of Congress Cataloging-in-Publication Data

Encyclopedia of data warehousing and mining / John Wang, editor.

p. cm.
Includes bibliographical references and index.
ISBN 1-59140-557-2 (hard cover) -- ISBN 1-59140-559-9 (ebook)
1. Data warehousing. 2. Data mining. I. Wang, John, 1955-
QA76.9.D37E52 2005
005.74--dc22
2005004522

British Cataloguing in Publication Data

A Cataloguing in Publication record for this book is available from the British Library.

All work contributed to this encyclopedia set is new, previously-unpublished material. The views expressed in this encyclopedia set
are those of the authors, but not necessarily of the publisher.

TEAM LinG
Editorial Advisory Board

Floriana Esposito, Universitadi Bari, Italy

Chandrika Kamath, Lawrence Livermore National Laboratory, USA

Huan Liu, Arizona State University, USA

Sach Mukherjee, University of Oxford, UK

Alan Oppenheim, Montclair State University, USA

Marco Ramoni, Harvard University, USA

Mehran Sahami, Stanford University, USA

Michele Sebag, Universite Paris-Sud, France

Alexander Tuzhilin, New York University, USA

Ning Zhong, Maebashi Institute of Technology, Japan

TEAM LinG
List of Contributors

Abdulghani, Amin A. / Quantiva, USA

Abidin, Taufik / North Dakota State University, USA
Abou-Seada, Magda / Middlesex University, UK
Agresti, William W. / Johns Hopkins University, USA
Amoretti, Maria Suzana Marc / Federal University of Rio Grande do Sul (UFRGS), Brasil
An, Aijun / York University, Canada
Aradhye, Hrishikesh B. / SRI International, USA
Artz, John M. / The George Washington University, USA
Ashrafi, Mafruz Zaman / Monash University, Australia
Athappilly, Kuriakose / Western Michigan University, USA
Awad, Mamoun / University of Texas at Dallas, USA
Bala, Pradip Kumar / IBAT, Deemed University, India
Banerjee, Protima / Drexel University, USA
Banerjee, Rabindra Nath / Indian Institute of Technology, Kharagpur, India
Barko, Christopher D. / The University of North Carolina at Greensboro, USA
Bashir, Ahmad / University of Texas at Dallas, USA
Becerra, Victor M. / University of Reading, UK
Bellatreche, Ladjel / LISI/ENSMA, France
Besemann, Christopher / North Dakota State University, USA
Betz, Andrew L. / Progressive Insurance, USA
Beynon, Malcolm J. / Cardiff University, UK
Bhatnagar, Shalabh / Indian Institute of Science, India
Bhowmick, Sourav Saha / Nanyang Technological University, Singapore
Bickel, Steffen / Humboldt-Universitt zu Berlin, Germany
Bittanti, Sergio / Politecnico di Milano, Italy
Borges, Thyago / Catholic University of Pelotas, Brazil
Boros, Endre / RUTCOR, Rutgers University, USA
Bose, Indranil / The University of Hong Kong, Hong Kong
Boulicaut, Jean-Francois / INSA de Lyon, France
Breault, Joseph L. / Ochsner Clinic Foundation, USA
Bretschneider, Timo R. / Nanyang Technological University, Singapore
Brown, Marvin L. / Grambling State University, USA
Bruha, Ivan / McMaster University, Canada
Bruining, Nico / Erasmus Medical Thorax Center, The Netherlands
Buccafurri, Francesco / University Mediterranea of Reggio Calabria, Italy
Burr, Tom / Los Alamos National Laboratory, USA
Butler, Shane M. / Monash University, Australia
Cadot, Martine / University of Henri Poincar/LORIA, Nancy, France
Caelli, Terry / Australian National University, Australia
Calvo, Roberto Wolfler / Universit de Technologie de Troyes, France
Caramia, Massimiliano / Istituto per le Applicazioni del Calcolo IAC-CNR, Italy
Cardoso, Jorge / University of Madeira, Portugal

TEAM LinG
Carneiro, Sofia / University of Minho, Portugal
Cerchiello, Paola / University of Pavia, Italy
Chakravarty, Indrani / Indian Institute of Technology, India
Chalasani, Suresh / University of Wisconsin-Parkside, USA
Chang, Chia-Hui / National Central University, Taiwan
Chen, Qiyang / Montclair State University, USA
Chen, Shaokang / The University of Queensland, Australia
Chen, Yao / University of Massachusetts Lowell, USA
Chen, Zhong / Shanghai JiaoTong University, PR China
Chien, Chen-Fu / National Tsing Hua University, Taiwan
Cho, Vincent / The Hong Kong Polytechnic University, Hong Kong
Chu, Feng / Nanyang Technological University, Singapore
Chu, Wesley / University of California - Los Angeles, USA
Chung, Seokkyung / University of Southern California, USA
Chung, Soon M. / Wright State University, USA
Conversano, Claudio / University of Cassino, Italy
Cook, Diane J. / University of Texas at Arlington, USA
Cook, Jack / Rochester Institute of Technology, USA
Cunningham, Colleen / Drexel University, USA
Dai, Honghua / Deakin University, Australia
Daly, Olena / Monash University, Australia
Dardzinska, Agnieszka / Bialystok Technical University, Poland
Das, Gautam / The University of Texas at Arlington, USA
de Campos, Luis M. / Universidad de Granada, Spain
de Luigi, Fabio / University of Ferrara, Italy
De Meo, Pasquale / Universit Mediterranea di Reggio Calabria, Italy
DeLorenzo, Gary J. / Robert Morris University, USA
Delve, Janet / University of Portsmouth, UK
Denoyer, Ludovic / University of Paris VI, France
Denton, Anne / North Dakota State University, USA
Dhaenens, Clarisse / LIFL, University of Lille 1, France
Diday, Edwin / University of Dauphine, France
Dillon, Tharam / University of Technology Sydney, Australia
Ding, Qiang / Concordia College, USA
Ding, Qin / Pennsylvania State University, USA
Domeniconi, Carlotta / George Mason University, USA
Dorado de la Calle, Julin / University of A Corua, Spain
Dorai, Chitra / IBM T. J. Watson Research Center, USA
Drew, James H. / Verizon Laboratories, USA
Dumitriu, Luminita / Dunarea de Jos University, Romania
Ester, Martin / Simon Fraser University, Canada
Fan, Weiguo / Virginia Polytechnic Institute and State University, USA
Felici, Giovanni / Istituto di Analisi dei Sistemi ed Informatica (IASI-CNR), Italy
Feng, Ling / University of Twente, The Netherlands
Fernndez, Vctor Fresno / Universidad Rey Juan Carlos, Spain
Fernndez-Luna, Juan M. / Universidad de Granada, Spain
Fischer, Ingrid / Friedrich-Alexander University Erlangen-Nrnberg, Germany
Fu, Ada Wai-Chee / The Chinese University of Hong Kong, Hong Kong
Fu, Li M. / University of Florida, USA
Fu, Yongjian / Cleveland State University, USA
Fung, Benjamin C. M. / Simon Fraser University, Canada
Fung, Benny Yiu-ming / The Hong Kong Polytechnic University, Hong Kong
Gallinari, Patrick / University of Paris VI, France
Galvo, Roberto Kawakami Harrop / Instituto Tecnolgico de Aeronutica, Brazil

TEAM LinG
Ganguly, Auroop R. / Oak Ridge National Laboratory, USA
Garatti, Simone / Politecnico di Milano, Italy
Garrity, Edward J. / Canisius College, USA
Ge, Nanxiang / Aventis, USA
Gehrke, Johannes / Cornell University, USA
Georgieva, Olga / Institute of Control and System Research, Bulgaria
Giudici, Paolo / University of Pavia, Italy
Goodman, Kenneth W. / University of Miami, USA
Greenidge, Charles / University of the West Indies, Barbados
Grzymala-Busse, Jerzy W. / University of Kansas, USA
Gunopulos, Dimitrios / University of California, USA
Guo, Hong / Southern Illinois University, USA
Gupta, Amar / University of Arizona, USA
Gupta, P. / Indian Institute of Technology, India
Haastrup, Palle / European Commission, Italy
Hamdi, Mohamed Salah / UAE University, UAE
Hamel, Lutz / University of Rhode Island, USA
Hamers, Ronald / Erasmus Medical Thorax Center, The Netherlands
Hammer, Peter L. / RUTCOR, Rutgers University, USA
Han, Hyoil / Drexel University, USA
Harms, Sherri K. / University of Nebraska at Kearney, USA
Hogo, Mofreh / Czech Technical University, Czech Republic
Holder, Lawrence B. / University of Texas at Arlington, USA
Hong, Yu / BearingPoint Inc, USA
Horiguchi, Susumu / Tohoku University, Japan
Hou, Wen-Chi / Southern Illinois University, USA
Hsu, Chun-Nan / Institute of Information Science, Academia Sinica, Taiwan
Hsu, William H. / Kansas State University, USA
Hu, Wen-Chen / University of North Dakota, USA
Hu, Xiaohua / Drexel University, USA
Huang, Xiangji / York University, Canada
Huete, Juan F. / Universidad de Granada, Spain
Hwang, Sae / University of Texas at Arlington, USA
Ibaraki, Toshihide / Kwansei Gakuin University, Japan
Ito, Takao / Ube National College of Technology, Japan
Jahangiri, Mehrdad / University of Southern California, USA
Jrvelin, Kalervo / University of Tampere, Finland
Jha, Neha / Indian Institute of Technology, Kharagpur, India
Jin, Haihao / University of Kentucky, USA
Jourdan, Laetitia / LIFL, University of Lille 1, France
Jun, Jongeun / University of Southern California, USA
Kanapady, Ramdev / University of Minnesota, USA
Kao, Odej / University of Paderborn, Germany
Karakaya, Murat / Bilkent University, Turkey
Katsaros, Dimitrios / Aristotle University, Greece
Kern-Isberner, Gabriele / University of Dortmund, Germany
Khan, Latifur / University of Texas at Dallas, USA
Khan, M. Riaz / University of Massachusetts Lowell, USA
Khan, Shiraj / University of South Florida, USA
Kickhfel, Rodrigo Branco / Catholic University of Pelotas, Brazil
Kim, Han-Joon / The University of Seoul, Korea
Klawonn, Frank / University of Applied Sciences Braunschweig/Wolfenbuettel, Germany
Koeller, Andreas / Montclair State University, USA
Kokol, Peter / University of Maribor, FERI, Slovenia

TEAM LinG
Kontio, Juha / Turku Polytechnic, Finland
Koppelaar, Henk / Delft University of Technology, The Netherlands
Kroeze, Jan H. / University of Pretoria, South Africa
Kros, John F. / East Carolina University, USA
Kryszkiewicz, Marzena / Warsaw University of Technology, Poland
Kusiak, Andrew / The University of Iowa, USA
la Tendresse, Ingo / Technical University of Clausthal, Germany
Lax, Gianluca / University Mediterranea of Reggio Calabria, Italy
Layos, Luis Magdalena / Universidad Politcnica de Madrid, Spain
Lazarevic, Aleksandar / University of Minnesota, USA
Lee, Chung-Hong / National Kaohsiung University of Applied Sciences, Taiwan
Lee, Chung-wei / Auburn University, USA
Lee, JeongKyu / University of Texas at Arlington, USA
Lee, Tzai-Zang / National Cheng Kung University, Taiwan, ROC
Lee, Zu-Hsu / Montclair State University, USA
Lee-Post, Anita / University of Kentucky, USA
Leni, Mitja / University of Maribor, FERI, Slovenia
Levary, Reuven R. / Saint Louis University, USA
Li, Tao / Florida International University, USA
Li, Wenyuan / Nanyang Technological University, Singapore
Liberati, Diego / Consiglio Nazionale delle Ricerche, Italy
Licthnow, Daniel / Catholic University of Pelotas, Brazil
Lim, Ee-Peng / Nanyang Technological University, Singapore
Lin, Beixin (Betsy) / Montclair State University, USA
Lin, Tsau Young / San Jose State University, USA
Lindell, Yehuda / Bar-Ilan University, Israel
Lingras, Pawan / Saint Marys University, Canada
Liu, Chang / Northern Illinois University, USA
Liu, Huan / Arizona State University, USA
Liu, Li / Aventis, USA
Liu, Xiaohui / Brunel University, UK
Liu, Xiaoqiang / Delft University of Technology, The Netherlands, and Donghua University, China
Lo, Victor S.Y. / Fidelity Personal Investments, USA
Lodhi, Huma / Imperial College London, UK
Loh, Stanley / Catholic University of Pelotas, Brazil, and Lutheran University of Brasil, Brazil
Long, Lori K. / Kent State University, USA
Lorenzi, Fabiana / Universidade Luterana do Brasil, Brazil
Lovell, Brian C. / The University of Queensland, Australia
Lu, June / University of Houston-Victoria, USA
Lu, Xinjian / California State University, Hayward, USA
Lutu, Patricia E.N. / University of Pretoria, South Africa
Ma, Sheng / IBM T.J. Watson Research Center, USA
Maj, Jean-Baptiste / LORIA/INRIA, France
Maloof, Marcus A. / Georgetown University, USA
Mangamuri, Murali / Wright State University, USA
Mani, D. R. / Massachusetts Institute of Technology, USA, and Harvard University, USA
Maniezzo, Vittorio / University of Bologna, Italy
Manolopoulos, Yannis / Aristotle University, Greece
Marchetti, Carlo / Universit di Roma La Sapienza, Italy
Mart, Rafael / Universitat de Valncia, Spain
Masseglia, Florent / INRIA Sophia Antipolis, France
Mathieu, Richard / Saint Louis University, USA
McLeod, Dennis / University of Southern California, USA
Mecella, Massimo / Universit di Roma La Sapienza, Italy

TEAM LinG
Meinl, Thorsten / Friedrich-Alexander University Erlangen-Nrnberg, Germany
Meo, Rosa / Universit degli Studi di Torino, Italy
Mishra, Nilesh / Indian Institute of Technology, India
Mladeni , Dunja / Jozef Stefan Institute, Slovenia
Mobasher, Bamshad / DePaul University, USA
Mohania, Mukesh / IBM India Research Lab, India
Morantz, Brad / Georgia State University, USA
Moreira, Adriano / University of Minho, Portugal
Motiwalla, Luvai / University of Massachusetts Lowell, USA
Muhlenbach, Fabrice / EURISE, Universit Jean Monnet - Saint-Etienne, France
Mukherjee, Sach / University of Oxford, UK
Murty, M. Narasimha / Indian Institute of Science, India
Muruzbal, Jorge / University Rey Juan Carlos, Spain
Muselli, Marco / Italian National Research Council, Italy
Musicant, David R. / Carleton College, USA
Muslea, Ion / SRI International, USA
Nanopoulos, Alexandros / Aristotle University, Greece
Nasraoui, Olfa / University of Louisville, USA
Nayak, Richi / Queensland University of Technology, Australia
Nemati, Hamid R. / The University of North Carolina at Greensboro, USA
Ng, Vincent To-yee / The Hong Kong Polytechnic University, Hong Kong
Ng, Wee-Keong / Nanyang Technological University, Singapore
Nicholson, Scott / Syracuse University School of Information Studies, USA
ODonnell, Joseph B. / Canisius College, USA
Oh, JungHwan / University of Texas at Arlington, USA
Oppenheim, Alan / Montclair State University, USA
Owens, Jan / University of Wisconsin-Parkside, USA
Oza, Nikunj C. / NASA Ames Research Center, USA
Pang, Les / National Defense University, USA
Paquet, Eric / National Research Council of Canada, Canada
Pasquier, Nicolas / Universit de Nice-Sophia Antipolis, France
Pathak, Praveen / University of Florida, USA
Perlich, Claudia / IBM Research, USA
Perrizo, William / North Dakota State University, USA
Peter, Hadrian / University of the West Indies, Barbados
Peterson, Richard L. / Montclair State University, USA
Pharo, Nils / Oslo University College, Norway
Piltcher, Gustavo / Catholic University of Pelotas, Brazil
Poncelet, Pascal / Ecole des Mines dAls, France
Portougal, Victor / The University of Auckland, New Zealand
Povalej, Petra / University of Maribor, FERI, Slovenia
Primo, Tiago / Catholic University of Pelotas, Brazil
Provost, Foster / New York University, USA
Psaila, Giuseppe / Universit degli Studi di Bergamo, Italy
Quattrone, Giovanni / Universit Mediterranea di Reggio Calabria, Italy
Rabual Dopico, Juan R. / University of A Corua, Spain
Rahman, Hakikur / SDNP, Bangladesh
Rakotomalala, Ricco / ERIC, Universit Lumire Lyon 2, France
Ramoni, Marco F. / Harvard Medical School, USA
Ras, Zbigniew W. / University of North Carolina, Charlotte, USA
Rea, Alan / Western Michigan University, USA
Rehm, Frank / German Aerospace Center, Germany
Ricci, Francesco / eCommerce and Tourism Research Laboratory, ITC-irst, Italy
Rivero Cebrin, Daniel / University of A Corua, Spain

TEAM LinG
Sacharidis, Dimitris / University of Southern California, USA
Saldaa, Ramiro / Catholic University of Pelotas, Brazil
Sanders, G. Lawrence / State University of New York at Buffalo, USA
Santos, Maribel Yasmina / University of Minho, Portugal
Saquer, Jamil M. / Southwest Missouri State University, USA
Sayal, Mehmet / Hewlett-Packard Labs, USA
Saygin, Ycel / Sabanci University, Turkey
Scannapieco, Monica / Universit di Roma La Sapienza, Italy
Schafer, J. Ben / University of Northern Iowa, USA
Scheffer, Tobias / Humboldt-Universitt zu Berlin, Germany
Schneider, Michel / Blaise Pascal University, France
Scime, Anthony / State University of New York College Brockport, USA
Sebastiani, Paola / Boston University School of Public Health, USA
Segall, Richard S. / Arkansas State University, USA
Shah, Shital C. / The University of Iowa, USA
Shahabi, Cyrus / University of Southern California, USA
Shen, Hong / Japan Advanced Institute of Science and Technology, Japan
Sheng, Yihua Philip / Southern Illinois University, USA
Siciliano, Roberta / University of Naples Federico II, Italy
Simitsis, Alkis / National Technical University of Athens, Greece
Simes, Gabriel / Catholic University of Pelotas, Brazil
Sindoni, Giuseppe / ISTAT - National Institute of Statistics, Italy
Singh, Richa / Indian Institute of Technology, India
Smets, Philippe / Universit Libre de Bruxelles, Belgium
Smith, Kate A. / Monash University, Australia
Song, Il-Yeol / Drexel University, USA
Song, Min / Drexel University, USA
Sounderpandian, Jayavel / University of Wisconsin-Parkside, USA
Souto, Nieves Pedreira / University of A Corua, Spain
Stanton, Jeffrey / Syracuse University School of Information Studies, USA
Sundaram, David / The University of Auckland, New Zealand
Sural, Shamik / Indian Institute of Technology, Kharagpur, India
Talbi, El-Ghazali / LIFL, University of Lille 1, France
Tan, Hee Beng Kuan / Nanyang Technological University, Singapore
Tan, Rebecca Boon-Noi / Monash University, Australia
Taniar, David / Monash University, Australia
Teisseire, Maguelonne / University of Montpellier II, France
Terracina, Giorgio / Universit della Calabria, Italy
Thelwall, Mike / University of Wolverhampton, UK
Theodoratos, Dimitri / New Jersey Institute of Technology, USA
Thomasian, Alexander / New Jersey Institute of Technology, USA
Thuraisingham, Bhavani / The MITRE Corporation, USA
Tininini, Leonardo / CNR - Istituto di Analisi dei Sistemi e Informatica Antonio Ruberti, Italy
Troutt, Marvin D. / Kent State University, USA
Truemper, Klaus / University of Texas at Dallas, USA
Tsay, Li-Shiang / University of North Carolina, Charlotte, USA
Tzacheva, Angelina / University of North Carolina, Charlotte, USA
Ulusoy, zgr / Bilkent University, Turkey
Ursino, Domenico / Universit Mediterranea di Reggio Calabria, Italy
Vardaki, Maria / University of Athens, Greece
Vargas, Juan E. / University of South Carolina, USA
Vatsa, Mayank / Indian Institute of Technology, India
Viertl, Reinhard / Vienna University of Technology, Austria
Viktor, Herna L. / University of Ottawa, Canada

TEAM LinG
Virgillito, Antonino / Universit di Roma La Sapienza, Italy
Viswanath, P. / Indian Institute of Science, India
Walter, Jrg Andreas / University of Bielefeld, Germany
Wang, Dajin / Montclair State University, USA
Wang, Hai / Saint Marys University, Canada
Wang, Ke / Simon Fraser University, Canada
Wang, Lipo / Nanyang Technological University, Singapore
Wang, Shouhong / University of Massachusetts Dartmouth, USA
Wang, Xiong / California State University at Fullerton, USA
Webb, Geoffrey I. / Monash University, Australia
Wen, Ji-Rong / Microsoft Research Asia, China
West, Chad / IBM Canada Limited, Canada
Wickramasinghe, Nilmini / Cleveland State University, USA
Wieczorkowska, Alicja A. / Polish-Japanese Institute of Information Technology, Poland
Winkler, William E. / U.S. Bureau of the Census, USA
Wong, Raymond Chi-Wing / The Chinese University of Hong Kong, Hong Kong
Woon, Yew-Kwong / Nanyang Technological University, Singapore
Wu, Chien-Hsing / National University of Kaohsiung, Taiwan, ROC
Xiang, Yang / University of Guelph, Canada
Xing, Ruben / Montclair State University, USA
Yan, Feng / Williams Power, USA
Yan, Rui / Saint Marys University, Canada
Yang, Hsin-Chang / Chang Jung University, Taiwan
Yang, Hung-Jen / National Kaohsiung Normal University, Taiwan
Yang, Ying / Monash University, Australia
Yao, James E. / Montclair State University, USA
Yao, Yiyu / University of Regina, Canada
Yavas, Gkhan / Bilkent University, Turkey
Yeh, Jyh-haw / Boise State University, USA
Yoo, Illhoi / Drexel University, USA
Yu, Lei / Arizona State University, USA
Zendulka, Jaroslav / Brno University of Technology, Czech Republic
Zhang, Bin / Hewlett-Packard Research Laboratories, USA
Zhang, Chengqi / University of Technology Sydney, Australia
Zhang, Shichao / University of Technology Sydney, Australia
Zhang, Yu-Jin / Tsinghua University, Beijing, China
Zhao, Qiankun / Nanyang Technological University, Singapore
Zhao, Yan / University of Regina, Canada
Zhao, Yuan / Nanyang Technological University, Singapore
Zhou, Senqiang / Simon Fraser University, Canada
Zhou, Zhi-Hua / Nanjing University, China
Zhu, Dan / Iowa State University, USA
Zhu, Qiang / University of Michigan, USA
Ziad, Tarek / NUXEO, France
Ziarko, Wojciech / University of Regina, Canada
Zorman, Milan / University of Maribor, FERI, Slovenia
Zou, Qinghua / University of California - Los Angeles, USA

TEAM LinG
Contents
by Volume

VOLUME I

Action Rules / Zbigniew W. Ras, Angelina Tzacheva, and Li-Shiang Tsay ........................................................... 1

Active Disks for Data Mining / Alexander Thomasian ........................................................................................... 6

Active Learning with Multiple Views / Ion Muslea ................................................................................................. 12

Administering and Managing a Data Warehouse / James E. Yao, Chang Liu, Qiyang Chen, and June Lu .......... 17

Agent-Based Mining of User Profiles for E-Services / Pasquale De Meo, Giovanni Quattrone,
Giorgio Terracina, and Domenico Ursino ......................................................................................................... 23

Aggregate Query Rewriting in Multidimensional Databases / Leonardo Tininini ................................................. 28

Aggregation for Predictive Modeling with Relational Data / Claudia Perlich and Foster Provost ....................... 33

API Standardization Efforts for Data Mining / Jaroslav Zendulka ......................................................................... 39

Application of Data Mining to Recommender Systems, The / J. Ben Schafer ........................................................ 44

Approximate Range Queries by Histograms in OLAP / Francesco Buccafurri and Gianluca Lax ........................ 49

Artificial Neural Networks for Prediction / Rafael Mart ......................................................................................... 54

Association Rule Mining / Yew-Kwong Woon, Wee-Keong Ng, and Ee-Peng Lim ................................................ 59

Association Rule Mining and Application to MPIS / Raymond Chi-Wing Wong and Ada Wai-Chee Fu ............. 65

Association Rule Mining of Relational Data / Anne Denton and Christopher Besemann ..................................... 70

Association Rules and Statistics / Martine Cadot, Jean-Baptiste Maj, and Tarek Ziad ..................................... 74

Automated Anomaly Detection / Brad Morantz ..................................................................................................... 78

Automatic Musical Instrument Sound Classification / Alicja A. Wieczorkowska ................................................... 83

Bayesian Networks / Ahmad Bashir, Latifur Khan, and Mamoun Awad ............................................................... 89

TEAM LinG
Best Practices in Data Warehousing from the Federal Perspective / Les Pang ....................................................... 94

Bibliomining for Library Decision-Making / Scott Nicholson and Jeffrey Stanton ................................................. 100

Biomedical Data Mining Using RBF Neural Networks / Feng Chu and Lipo Wang ................................................ 106

Building Empirical-Based Knowledge for Design Recovery / Hee Beng Kuan Tan and Yuan Zhao ...................... 112

Business Processes / David Sundaram and Victor Portougal ............................................................................... 118

Case-Based Recommender Systems / Fabiana Lorenzi and Francesco Ricci ....................................................... 124

Categorization Process and Data Mining / Maria Suzana Marc Amoretti ............................................................. 129

Center-Based Clustering and Regression Clustering / Bin Zhang ........................................................................... 134

Classification and Regression Trees / Johannes Gehrke ........................................................................................ 141

Classification Methods / Aijun An ........................................................................................................................... 144

Closed-Itemset Incremental-Mining Problem / Luminita Dumitriu ......................................................................... 150

Cluster Analysis in Fitting Mixtures of Curves / Tom Burr ..................................................................................... 154

Clustering Analysis and Algorithms / Xiangji Huang ............................................................................................. 159

Clustering in the Identification of Space Models / Maribel Yasmina Santos, Adriano Moreira,
and Sofia Carneiro .............................................................................................................................................. 165

Clustering of Time Series Data / Anne Denton ......................................................................................................... 172

Clustering Techniques / Sheng Ma and Tao Li ....................................................................................................... 176

Clustering Techniques for Outlier Detection / Frank Klawonn and Frank Rehm ................................................. 180

Combining Induction Methods with the Multimethod Approach / Mitja Leni, Peter Kokol, Petra Povalej
and Milan Zorman ............................................................................................................................................... 184

Comprehensibility of Data Mining Algorithms / Zhi-Hua Zhou .............................................................................. 190

Computation of OLAP Cubes / Amin A. Abdulghani .............................................................................................. 196

Concept Drift / Marcus A. Maloof ............................................................................................................................ 202

Condensed Representations for Data Mining / Jean-Francois Boulicaut ............................................................. 207

Content-Based Image Retrieval / Timo R. Bretschneider and Odej Kao ................................................................. 212

Continuous Auditing and Data Mining / Edward J. Garrity, Joseph B. ODonnell,

and G. Lawrence Sanders ................................................................................................................................... 217

Data Driven vs. Metric Driven Data Warehouse Design / John M. Artz ................................................................. 223

Data Management in Three-Dimensional Structures / Xiong Wang ........................................................................ 228

TEAM LinG
Data Mining and Decision Support for Business and Science / Auroop R. Ganguly, Amar Gupta,
and Shiraj Khan .................................................................................................................................................. 233

Data Mining and Warehousing in Pharma Industry / Andrew Kusiak and Shital C. Shah .................................... 239

Data Mining for Damage Detection in Engineering Structures / Ramdev Kanapady

and Aleksandar Lazarevic .................................................................................................................................. 245

Data Mining for Intrusion Detection / Aleksandar Lazarevic ................................................................................. 251

Data Mining in Diabetes Diagnosis and Detection / Indranil Bose ........................................................................ 257

Data Mining in Human Resources / Marvin D. Troutt and Lori K. Long ............................................................... 262

Data Mining in the Federal Government / Les Pang ................................................................................................ 268

Data Mining in the Soft Computing Paradigm / Pradip Kumar Bala, Shamik Sural,
and Rabindra Nath Banerjee .............................................................................................................................. 272

Data Mining Medical Digital Libraries / Colleen Cunningham and Xiaohua Hu ................................................... 278

Data Mining Methods for Microarray Data Analysis / Lei Yu and Huan Liu ......................................................... 283

Data Mining with Cubegrades / Amin A. Abdulghani ............................................................................................. 288

Data Mining with Incomplete Data / Hai Wang and Shouhong Wang .................................................................... 293

Data Quality in Cooperative Information Systems / Carlo Marchetti, Massimo Mecella, Monica Scannapieco,
and Antonino Virgillito ...................................................................................................................................... 297

Data Quality in Data Warehouses / William E. Winkler .......................................................................................... 302

Data Reduction and Compression in Database Systems / Alexander Thomasian .................................................. 307

Data Warehouse Back-End Tools / Alkis Simitsis and Dimitri Theodoratos ......................................................... 312

Data Warehouse Performance / Beixin (Betsy) Lin, Yu Hong, and Zu-Hsu Lee ..................................................... 318

Data Warehousing and Mining in Supply Chains / Richard Mathieu and Reuven R. Levary ............................... 323

Data Warehousing Search Engine / Hadrian Peter and Charles Greenidge ......................................................... 328

Data Warehousing Solutions for Reporting Problems / Juha Kontio ..................................................................... 334

Database Queries, Data Mining, and OLAP / Lutz Hamel ....................................................................................... 339

Database Sampling for Data Mining / Patricia E.N. Lutu ....................................................................................... 344

DEA Evaluation of Performance of E-Business Initiatives / Yao Chen, Luvai Motiwalla, and M. Riaz Khan ...... 349

Decision Tree Induction / Roberta Siciliano and Claudio Conversano ............................................................... 353

Diabetic Data Warehouses / Joseph L. Breault ....................................................................................................... 359

TEAM LinG
Discovering an Effective Measure in Data Mining / Takao Ito ............................................................................... 364

Discovering Knowledge from XML Documents / Richi Nayak .............................................................................. 372

Discovering Ranking Functions for Information Retrieval / Weiguo Fan and Praveen Pathak ............................ 377

Discovering Unknown Patterns in Free Text / Jan H. Kroeze ................................................................................. 382

Discovery Informatics / William W. Agresti ............................................................................................................. 387

Discretization for Data Mining / Ying Yang and Geoffrey I. Webb .......................................................................... 392

Discretization of Continuous Attributes / Fabrice Muhlenbach and Ricco Rakotomalala .................................. 397

Distributed Association Rule Mining / Mafruz Zaman Ashrafi, David Taniar, and Kate A. Smith ........................ 403

Distributed Data Management of Daily Car Pooling Problems / Roberto Wolfler Calvo, Fabio de Luigi,
Palle Haastrup, and Vittorio Maniezzo ............................................................................................................. 408

Drawing Representative Samples from Large Databases / Wen-Chi Hou, Hong Guo, Feng Yan,
and Qiang Zhu ..................................................................................................................................................... 413

Efficient Computation of Data Cubes and Aggregate Views / Leonardo Tininini .................................................. 421

Embedding Bayesian Networks in Sensor Grids / Juan E. Vargas .......................................................................... 427

Employing Neural Networks in Data Mining / Mohamed Salah Hamdi .................................................................. 433

Enhancing Web Search through Query Log Mining / Ji-Rong Wen ....................................................................... 438

Enhancing Web Search through Web Structure Mining / Ji-Rong Wen ................................................................. 443

Ensemble Data Mining Methods / Nikunj C. Oza ................................................................................................... 448

Ethics of Data Mining / Jack Cook ......................................................................................................................... 454

Ethnography to Define Requirements and Data Model / Gary J. DeLorenzo ......................................................... 459

Evaluation of Data Mining Methods / Paolo Giudici ............................................................................................. 464

Evolution of Data Cube Computational Approaches / Rebecca Boon-Noi Tan ...................................................... 469

Evolutionary Computation and Genetic Algorithms / William H. Hsu ..................................................................... 477

Evolutionary Data Mining For Genomics / Laetitia Jourdan, Clarisse Dhaenens, and El-Ghazali Talbi ............ 482

Evolutionary Mining of Rule Ensembles / Jorge Muruzbal .................................................................................. 487

Explanation-Oriented Data Mining / Yiyu Yao and Yan Zhao ................................................................................. 492

Factor Analysis in Data Mining / Zu-Hsu Lee, Richard L. Peterson, Chen-Fu Chien, and Ruben Xing ............... 498

Financial Ratio Selection for Distress Classification / Roberto Kawakami Harrop Galvo, Victor M. Becerra,
and Magda Abou-Seada ..................................................................................................................................... 503

TEAM LinG
Flexible Mining of Association Rules / Hong Shen ................................................................................................. 509

Formal Concept Analysis Based Clustering / Jamil M. Saquer .............................................................................. 514

Fuzzy Information and Data Analysis / Reinhard Viertl ......................................................................................... 519

General Model for Data Warehouses, A / Michel Schneider .................................................................................. 523

Genetic Programming / William H. Hsu .................................................................................................................... 529

Graph Transformations and Neural Networks / Ingrid Fischer ............................................................................... 534

Graph-Based Data Mining / Lawrence B. Holder and Diane J. Cook .................................................................... 540

Group Pattern Discovery Systems for Multiple Data Sources / Shichao Zhang and Chengqi Zhang ................... 546

Heterogeneous Gene Data for Classifying Tumors / Benny Yiu-ming Fung and Vincent To-yee Ng .................... 550

Hierarchical Document Clustering / Benjamin C. M. Fung, Ke Wang, and Martin Ester ....................................... 555

High Frequency Patterns in Data Mining / Tsau Young Lin .................................................................................... 560

Homeland Security Data Mining and Link Analysis / Bhavani Thuraisingham ..................................................... 566

Humanities Data Warehousing / Janet Delve .......................................................................................................... 570

Hyperbolic Space for Interactive Visualization / Jrg Andreas Walter ................................................................... 575

VOLUME II

Identifying Single Clusters in Large Data Sets / Frank Klawonn and Olga Georgieva ......................................... 582

Immersive Image Mining in Cardiology / Xiaoqiang Liu, Henk Koppelaar, Ronald Hamers,
and Nico Bruining ............................................................................................................................................... 586

Imprecise Data and the Data Mining Process / Marvin L. Brown and John F. Kros .............................................. 593

Incorporating the People Perspective into Data Mining / Nilmini Wickramasinghe .............................................. 599

Incremental Mining from News Streams / Seokkyung Chung, Jongeun Jun and Dennis McLeod ........................ 606

Inexact Field Learning Approach for Data Mining / Honghua Dai ......................................................................... 611

Information Extraction in Biomedical Literature / Min Song, Il-Yeol Song, Xiaohua Hu, and Hyoil Han .............. 615

Instance Selection / Huan Liu and Lei Yu ............................................................................................................... 621

Integration of Data Sources through Data Mining / Andreas Koeller .................................................................... 625

Intelligence Density / David Sundaram and Victor Portougal .............................................................................. 630

Intelligent Data Analysis / Xiaohui Liu ................................................................................................................... 634

TEAM LinG
Intelligent Query Answering / Zbigniew W. Ras and Agnieszka Dardzinska ........................................................ 639

Interactive Visual Data Mining / Shouhong Wang and Hai Wang .......................................................................... 644

Interscheme Properties Role in Data Warehouses / Pasquale De Meo, Giorgio Terracina,

and Domenico Ursino .......................................................................................................................................... 647

Inter-Transactional Association Analysis for Prediction / Ling Feng and Tharam Dillon .................................... 653

Interval Set Representations of Clusters / Pawan Lingras, Rui Yan, Mofreh Hogo, and Chad West .................... 659

Kernel Methods in Chemoinformatics / Huma Lodhi .............................................................................................. 664

Knowledge Discovery with Artificial Neural Networks / Juan R. Rabual Dopico, Daniel Rivero Cebrin,
Julin Dorado de la Calle, and Nieves Pedreira Souto .................................................................................... 669

Learning Bayesian Networks / Marco F. Ramoni and Paola Sebastiani ............................................................... 674

Learning Information Extraction Rules for Web Data Mining / Chia-Hui Chang and Chun-Nan Hsu .................. 678

Locally Adaptive Techniques for Pattern Classification / Carlotta Domeniconi and Dimitrios Gunopulos ........ 684

Logical Analysis of Data / Endre Boros, Peter L. Hammer, and Toshihide Ibaraki .............................................. 689

Lsquare System for Mining Logic Data, The / Giovanni Felici and Klaus Truemper ............................................ 693

Marketing Data Mining / Victor S.Y. Lo .................................................................................................................. 698

Material Acquisitions Using Discovery Informatics Approach / Chien-Hsing Wu and Tzai-Zang Lee ................. 705

Materialized Hypertext View Maintenance / Giuseppe Sindoni .............................................................................. 710

Materialized Hypertext Views / Giuseppe Sindoni ................................................................................................... 714

Materialized View Selection for Data Warehouse Design / Dimitri Theodoratos and Alkis Simitsis ..................... 717

Methods for Choosing Clusters in Phylogenetic Trees / Tom Burr ........................................................................ 722

Microarray Data Mining / Li M. Fu .......................................................................................................................... 728

Microarray Databases for Biotechnology / Richard S. Segall ................................................................................ 734

Mine Rule / Rosa Meo and Giuseppe Psaila ........................................................................................................... 740

Mining Association Rules on a NCR Teradata System / Soon M. Chung and Murali Mangamuri ....................... 746

Mining Association Rules Using Frequent Closed Itemsets / Nicolas Pasquier ................................................... 752

Mining Chat Discussions / Stanley Loh, Daniel Licthnow, Thyago Borges, Tiago Primo,
Rodrigo Branco Kickhfel, Gabriel Simes, Gustavo Piltcher, and Ramiro Saldaa ..................................... 758

Mining Data with Group Theoretical Means / Gabriele Kern-Isberner .................................................................. 763

Mining E-Mail Data / Steffen Bickel and Tobias Scheffer ....................................................................................... 768

TEAM LinG
Mining for Image Classification Based on Feature Elements / Yu-Jin Zhang ......................................................... 773

Mining for Profitable Patterns in the Stock Market / Yihua Philip Sheng, Wen-Chi Hou, and Zhong Chen ......... 779

Mining for Web-Enabled E-Business Applications / Richi Nayak ......................................................................... 785

Mining Frequent Patterns via Pattern Decomposition / Qinghua Zou and Wesley Chu ......................................... 790

Mining Group Differences / Shane M. Butler and Geoffrey I. Webb ....................................................................... 795

Mining Historical XML / Qiankun Zhao and Sourav Saha Bhowmick .................................................................. 800

Mining Images for Structure / Terry Caelli ............................................................................................................. 805

Mining Microarray Data / Nanxiang Ge and Li Liu ................................................................................................ 810

Mining Quantitative and Fuzzy Association Rules / Hong Shen and Susumu Horiguchi ..................................... 815

Model Identification through Data Mining / Diego Liberati .................................................................................. 820

Modeling Web-Based Data in a Data Warehouse / Hadrian Peter and Charles Greenidge ................................. 826

Moral Foundations of Data Mining / Kenneth W. Goodman ................................................................................... 832

Mosaic-Based Relevance Feedback for Image Retrieval / Odej Kao and Ingo la Tendresse ................................. 837

Multimodal Analysis in Multimedia Using Symbolic Kernels / Hrishikesh B. Aradhye and Chitra Dorai ............ 842

Multiple Hypothesis Testing for Data Mining / Sach Mukherjee ........................................................................... 848

Music Information Retrieval / Alicja A. Wieczorkowska ......................................................................................... 854

Negative Association Rules in Data Mining / Olena Daly and David Taniar ....................................................... 859

Neural Networks for Prediction and Classification / Kate A. Smith ......................................................................... 865

Off-Line Signature Recognition / Indrani Chakravarty, Nilesh Mishra, Mayank Vatsa, Richa Singh,
and P. Gupta ........................................................................................................................................................ 870

Online Analytical Processing Systems / Rebecca Boon-Noi Tan ........................................................................... 876

Online Signature Recognition / Indrani Chakravarty, Nilesh Mishra, Mayank Vatsa, Richa Singh,
and P. Gupta ........................................................................................................................................................ 885

Organizational Data Mining / Hamid R. Nemati and Christopher D. Barko .......................................................... 891

Path Mining in Web Processes Using Profiles / Jorge Cardoso ............................................................................. 896

Pattern Synthesis for Large-Scale Pattern Recognition / P. Viswanath, M. Narasimha Murty,

and Shalabh Bhatnagar ..................................................................................................................................... 902

Physical Data Warehousing Design / Ladjel Bellatreche and Mukesh Mohania .................................................. 906

Predicting Resource Usage for Capital Efficient Marketing / D. R. Mani, Andrew L. Betz, and James H. Drew .... 912

TEAM LinG
Privacy and Confidentiality Issues in Data Mining / Ycel Saygin ......................................................................... 921

Privacy Protection in Association Rule Mining / Neha Jha and Shamik Sural ..................................................... 925

Profit Mining / Senqiang Zhou and Ke Wang ......................................................................................................... 930

Pseudo Independent Models / Yang Xiang ............................................................................................................ 935

Reasoning about Frequent Patterns with Negation / Marzena Kryszkiewicz ......................................................... 941

Recovery of Data Dependencies / Hee Beng Kuan Tan and Yuan Zhao ................................................................ 947

Reinforcing CRM with Data Mining / Dan Zhu ........................................................................................................ 950

Resource Allocation in Wireless Networks / Dimitrios Katsaros, Gkhan Yavas, Alexandros Nanopoulos,
Murat Karakaya, zgr Ulusoy, and Yannis Manolopoulos ............................................................................ 955

Retrieving Medical Records Using Bayesian Networks / Luis M. de Campos, Juan M. Fernndez-Luna,
and Juan F. Huete ............................................................................................................................................... 960

Robust Face Recognition for Data Mining / Brian C. Lovell and Shaokang Chen ............................................... 965

Rough Sets and Data Mining / Jerzy W. Grzymala-Busse and Wojciech Ziarko .................................................... 973

Rule Generation Methods Based on Logic Synthesis / Marco Muselli .................................................................. 978

Rule Qualities and Knowledge Combination for Decision-Making / Ivan Bruha .................................................... 984

Sampling Methods in Approximate Query Answering Systems / Gautam Das ...................................................... 990

Scientific Web Intelligence / Mike Thelwall ............................................................................................................ 995

Search Situations and Transitions / Nils Pharo and Kalervo Jrvelin .................................................................. 1000

Secure Multiparty Computation for Privacy Preserving Data Mining / Yehida Lindell .......................................... 1005

Semantic Data Mining / Protima Banerjee, Xiaohua Hu, and Illhoi Yoo ............................................................... 1010

Semi-Structured Document Classification / Ludovic Denoyer and Patrick Gallinari ............................................ 1015

Semi-Supervised Learning / Tobias Scheffer ............................................................................................................ 1022

Sequential Pattern Mining / Florent Masseglia, Maguelonne Teisseire, and Pascal Poncelet ............................ 1028

Software Warehouse / Honghua Dai ....................................................................................................................... 1033

Spectral Methods for Data Clustering / Wenyuan Li ............................................................................................... 1037

Statistical Data Editing / Claudio Conversano and Roberta Siciliano .................................................................. 1043

Statistical Metadata in Data Processing and Interchange / Maria Vardaki ........................................................... 1048

Storage Strategies in Data Warehouses / Xinjian Lu .............................................................................................. 1054

TEAM LinG
Subgraph Mining / Ingrid Fischer and Thorsten Meinl .......................................................................................... 1059

Support Vector Machines / Mamoun Awad and Latifur Khan ............................................................................... 1064

Support Vector Machines Illuminated / David R. Musicant .................................................................................... 1071

Survival Analysis and Data Mining / Qiyang Chen, Alan Oppenheim, and Dajin Wang ...................................... 1077

Symbiotic Data Mining / Kuriakose Athappilly and Alan Rea .............................................................................. 1083

Symbolic Data Clustering / Edwin Diday and M. Narasimha Murthy .................................................................... 1087

Synthesis with Data Warehouse Applications and Utilities / Hakikur Rahman .................................................... 1092

Temporal Association Rule Mining in Event Sequences / Sherri K. Harms ........................................................... 1098

Text Content Approaches in Web Content Mining / Vctor Fresno Fernndez and Luis Magdalena Layos ....... 1103

Text Mining-Machine Learning on Documents / Dunja Mladeni ........................................................................ 1109

Text Mining Methods for Hierarchical Document Indexing / Han-Joon Kim ......................................................... 1113

Time Series Analysis and Mining Techniques / Mehmet Sayal .............................................................................. 1120

Time Series Data Forecasting / Vincent Cho ........................................................................................................... 1125

Topic Maps Generation by Text Mining / Hsin-Chang Yang and Chung-Hong Lee ............................................. 1130

Transferable Belief Model / Philippe Smets ............................................................................................................ 1135

Tree and Graph Mining / Dimitrios Katsaros and Yannis Manolopoulos ............................................................. 1140

Trends in Web Content and Structure Mining / Anita Lee-Post and Haihao Jin .................................................. 1146

Trends in Web Usage Mining / Anita Lee-Post and Haihao Jin ............................................................................ 1151

Unsupervised Mining of Genes Classifying Leukemia / Diego Liberati, Sergio Bittanti,

and Simone Garatti ............................................................................................................................................. 1155

Use of RFID in Supply Chain Data Processing / Jan Owens, Suresh Chalasani,
and Jayavel Sounderpandian ............................................................................................................................. 1160

Using Dempster-Shafer Theory in Data Mining / Malcolm J. Beynon .................................................................... 1166

Using Standard APIs for Data Mining in Prediction / Jaroslav Zendulka .............................................................. 1171

Utilizing Fuzzy Decision Trees in Decision Making / Malcolm J. Beynon .............................................................. 1175

Vertical Data Mining / William Perrizo, Qiang Ding, Qin Ding, and Taufik Abidin .............................................. 1181

Video Data Mining / JungHwan Oh, JeongKyu Lee, and Sae Hwang .................................................................... 1185

Visualization Techniques for Data Mining / Herna L. Viktor and Eric Paquet ...................................................... 1190

TEAM LinG
Wavelets for Querying Multidimensional Datasets / Cyrus Shahabi, Dimitris Sacharidis,
and Mehrdad Jahangiri ...................................................................................................................................... 1196

Web Mining in Thematic Search Engines / Massimiliano Caramia and Giovanni Felici .................................... 1201

Web Mining Overview / Bamshad Mobasher ......................................................................................................... 1206

Web Page Extension of Data Warehouses / Anthony Scime ................................................................................... 1211

Web Usage Mining / Bamshad Mobasher ............................................................................................................... 1216

Web Usage Mining and Its Applications / Yongjian Fu ......................................................................................... 1221

Web Usage Mining Data Preparation / Bamshad Mobasher ................................................................................... 1226

Web Usage Mining through Associative Models / Paolo Giudici and Paola Cerchiello .................................... 1231

World Wide Web Personalization / Olfa Nasraoui ................................................................................................. 1235

World Wide Web Usage Mining / Wen-Chen Hu, Hung-Jen Yang, Chung-wei Lee, and Jyh-haw Yeh ............... 1242

TEAM LinG
 xxi

Foreword

There has been much interest developed in the data mining field both in the academia and the industry over the past
10-15 years. The number of researchers and practitioners working in the field and the number of scientific papers
published in various data mining outlets increased drastically over this period. Major commercial vendors incorporated
various data mining tools into their products, and numerous applications in many areas, including life sciences, finance,
CRM, and Web-based applications, have been developed and successfully deployed.
Moreover, this interest is no longer limited to the researchers working in the traditional fields of statistics, machine
learning and databases, but has recently expanded to other fields, including operations research/management science
(OR/MS) and mathematics, as evidenced from various data mining tracks organized at different INFORMS meetings,
special issues of OR/MS journals and the recent conference on Mathematical Foundations of Learning Theory
organized by mathematicians.
As the Encyclopedia of Data Warehousing and Mining amply demonstrates, all these diverse interests from
different groups of researchers and practitioners helped to shape data mining as a broad and multi-faceted discipline
spanning a large class of problems in such diverse areas as life sciences, marketing (including CRM and e-commerce),
finance, telecommunications, astronomy, and many other fields (the so called data mining and X phenomenon, where
X constitutes a broad range of fields where data mining is used for analyzing the data). This also resulted in a process
of cross-fertilization of ideas generated by these diverse groups of researchers interacting across the traditional
boundaries of their disciplines.
Despite all this progress, data mining still faces several challenges that make the field ripe with future research
opportunities. First, despite the cross-fertilization of ideas spanning various disciplines, the convergence among
different disciplines proceeds gradually, and more work is required to arrive at a unified view of data mining widely
accepted by different groups of researchers. Second, despite a considerable progress, still more work is required on
the theoretical foundations of data mining, as was recently stated by the participants of the Dagstuhl workshop Data
Mining: The Next Generation organized by R. Agrawal, J.-C. Freytag and R. Ramakrishnan and also expressed by
various other data mining researchers. Third, the data mining community must address the privacy and security
problems for data mining to be accepted by the privacy advocates and the Congress. Fourth, as the field advances, so
is the scope of data mining applications. The challenge to the field is to develop more advanced data mining methods
that would work in these increasingly demanding applications. Fifth, despite a considerable progress in developing more
user-friendly data mining tools, more work is required in this area with the goal of making these tools accessible to a
large audience of nave data mining users. In particular, one of the challenges is to devise methods that would
smoothly embed data mining tools into corresponding applications on the front-end and would integrate these tools
with databases on the back-end. Achieving such capabilities is very important since this would allow data mining to
cross the chasm (using Geoffrey Moores terminology) and become a mainstream technology utilized by millions of
users. Finally, more work is required on actionability and on the development of better methods for discovering
actionable patterns in the data. Currently, discovering actionable patterns in data constitutes a laborious and
challenging process. It is important to streamline and simplify this process and make it more efficient.
Given significant and rapid advancements in data mining and data warehousing, it is important to take periodic
snapshots of the field every few years. The data mining community addressed this issue by producing publications
covering the state of the art of the field every few years starting with the first volume Advances in Knowledge
Discovery and Data Mining (edited by U. Fayyad, G. Piatetsky-Shapiro, P. Smyth and R. Uthurusamy) published by
AAAI/MIT Press in 1996. This encyclopedia provides the latest snapshot of the field and surveys a broad array of
topics ranging from the basic theories to the recent advancements in the field and covers a diverse range of problems

TEAM LinG
xxii

from the analysis of microarray data to the analysis of multimedia and Web data. It also identifies future directions and
trends in data mining and data warehousing. Therefore, this volume should become an excellent guide to researchers
and practitioners.

Alexander Tuzhilin
New York University, USA

TEAM LinG
 xxiii

Preface

How can a data-flooded manager get out of the mire? How can a confused decision maker pass through a maze?
How can an over-burdened problem solver clean up a mess? How can an exhausted scientist decipher a myth?
The answer is an interdisciplinary subject and a powerful tool known as data mining (DM). DM can turn data into
dollars; transform information into intelligence; change pattern into profit; and convert relationship into resources.
As the third branch of operations research and management science (OR/MS) and the third milestone of data
management, DM can help attack the third category of decision making by elevating our raw data into the third stage
of knowledge creation.
The term third has been mentioned four times above. Lets go backward and look at the three stages of knowledge
creation. Managers are often drowning in data (the first stage) but starving for knowledge. A collection of data is not
information (the second stage); and a collection of information is not knowledge. Data begets information which begets
knowledge. The whole subject of DM has a synergy of its own and represents more than the sum of its parts.
There are three categories of decision making: structured, semi-structured and unstructured. Decision making
processes fall along a continuum that ranges from highly structured decisions (sometimes called programmed) to highly
unstructured (non-programmed) decisions (Turban et al., 2005, p. 12).
At one end of the spectrum, structured processes are routine and typically repetitive problems for which standard
solutions exist. Unfortunately, rather than being static, deterministic and simple, the majority of real world problems
are dynamic, probabilistic, and complex. Many professional and personal problems are classified as unstructured, or
marginally as semi-structured, or even in between, since the boundaries between them may not be crystal-clear.
In addition to developing normative models (such as linear programming, economic order quantity) for solving
structured (or programmed) problems, operation researchers and management scientists have created many descriptive
models, such as simulation and goal programming, to deal with semi-structured alternatives. Unstructured problems,
however, fall in a gray areas for which there are no cut-and-dry solution methods. The current two branches of OR/MS
hit a dead end with unstructured problems.
To gain knowledge, one must understand the patterns that emerge from information. Patterns are not just simple
relationships among data; they exist separately from information, as archetypes or standards to which emerging
information can be compared so that one may draw inferences and take action. Over the last 40 years, the tools and
techniques used to process data and information have continued to evolve from databases (DBs) to data warehousing
(DW) and further to DM. DW applications, the middle of these three stages, have become business-critical. However,
DM can help deliver even more value from these huge repositories of information.
Certainly, there are many statistical models that have emerged over time. Machine learning has marked a milestone
in the evolution of computer science (Fayyad, Piatetsky-Shapiro, Smyth & Uthurusamy, 1996). Although DM is still
in its infancy, it is now being used in a wide range of industries and for a range of tasks in a variety of contexts (Wang,
2003). DM is synonymous with knowledge discovery in databases, knowledge extraction, data/pattern analysis, data
archeology, data dredging, data snooping, data fishing, information harvesting, and business intelligence (Giudici,
2003; Hand et al., 2001; Han & Kamber, 2000). There are unprecedented opportunities in the future to utilize DM.
Data warehousing and mining (DWM) is the science of managing and analyzing large datasets and discovering novel
patterns. In recent years, DWM has emerged as a particularly exciting and industrially relevant area of research.
Prodigious amounts of data are now being generated in domains as diverse and elusive as market research, functional
genomics and pharmaceuticals. Intelligently analyzing data to discover knowledge with the aim of answering crucial
questions and helping make informed decisions is the challenge that lies ahead.
The Encyclopedia of Data Warehousing and Mining provides theories, methodologies, functionalities, and
applications to decision makers, problem solvers, and data miners in business, academia, and government. DWM lies

TEAM LinG
xxiv

at the junction of database systems, artificial intelligence, machine learning and applied statistics, which makes it a
valuable area for researchers and practitioners. With a comprehensive overview, The Encyclopedia of Data Warehous-
ing and Mining offers a thorough exposure to the issues of importance in this rapidly changing field. The encyclopedia
also includes a rich mix of introductory and advanced topics while providing a comprehensive source of technical,
functional and legal references to DWM.
After spending more than a year preparing this book, with a strictly peer-reviewed process, I am delighted to see
it published. The standard for selection was very high. Each article went through at least three peer reviews; additional
third-party reviews were sought in cases of controversy. There have been innumerable instances where this feedback
has helped to improve the quality of the content, and even influenced authors in how they approach their topics.
The primary objective of this encyclopedia is to explore the myriad of issues regarding DWM. A broad spectrum
of practitioners, managers, scientists, educators, and graduate students who teach, perform research, and/or implement
these discoveries, are the envisioned readers of this encyclopedia.
The encyclopedia contains a collection of 234 articles, written by an international team of 361 experts representing
leading scientists and talented young scholars from 34 countries. They have contributed great effort to create a source
of solid, practical information, informed by sound underlying theory that should become a resource for all people
involved in this dynamic new field. Lets take a peek at a few articles:
The evaluation of DM methods requires a great deal of attention. A valid model evaluation and comparison can
improve considerably the efficiency of a DM process. Paolo Giudici has presented several ways to perform model
comparison, in which each has its advantages and disadvantages.
According to Zbigniew W. Ras, the main object of action rules is to generate special types of rules for a database
that point the direction for re-classifying objects with respect to some distinguishing attributes (called decision
attributes). This creates flexible attributes that form a basis for action rules construction.
With the constraints imposed by computer memory and mining algorithms, we can experience selection pressures
more than ever. The main point of instance selection is approximation. Our task is to achieve as good mining results
as possible by approximating the whole dataset with the selected instances and hope to do better in DM with instance
selection as it is possible to remove noisy and irrelevant data in the process. Huan Liu and Lei Yu have presented an
initial attempt to review and categorize the methods of instance selection in terms of sampling, classification, and
clustering.
Shichao Zhang and Chengqi Zhang introduce a group of pattern discovery systems for dealing with the multiple
data source (MDS) problem, mainly including a logical system for enhancing data quality; a logical system for resolving
conflicts; a data cleaning system; a database clustering system; a pattern discovery system and a post-mining system.
Based on his extensive experience, Gautam Das surveys recent state-of-the-art solutions to the problem of
approximate query answering in databases, in which ballpark answers(i.e., approximate answers) to queries can be
provided within acceptable time limits. These techniques sacrifice accuracy to improve running time; typically through
some sort of lossy data compression. Also, Han-Joon Kim (the holder of two patents on text mining applications)
discusses a comprehensive text-mining solution to document indexing problems on topic hierarchies (taxonomy).
Condensed representations have been proposed as a useful concept for the optimization of typical DM tasks. It
appears as a key concept within the emerging inductive DB framework where inductive query evaluation needs for
effective constraint-based DM techniques. Jean-Franois Boulicaut introduces this research domain, its achievements
in the context of frequent itemset mining from transactional data and its future trends.
Zhi-Hua Zhou discusses complexity issues in DM. Although we still have a long way to go in order to produce
patterns that can be understood by most people involved with DM tasks, endeavors on improving the comprehensibility
of complicated algorithms have proceeded at a promising pace.
Pattern classification poses a difficult challenge in finite settings and high dimensional spaces caused by the issue
of dimensionality. Carlotta Domeniconi and Dimitrios Gunopulos discuss classification techniques, including the
authors own work, to mitigate the problem of dimensionality and reduce bias, by estimating local feature relevance and
selecting features accordingly. This issue has both theoretical and practical relevance, since learning tasks abound in
which data are represented as a collection of a very large numbers of features. Thus, many applications can benefit from
improvements in predicting error.
Qinghua Zou proposes using pattern decomposition algorithms to find frequent patterns in large datasets. Pattern
decomposition is a DM technology that uses known, frequent or infrequent patterns to decompose long itemsets to
many short ones. It identifies frequent patterns in a dataset using a bottom-up methodology and reduces the size of
the dataset in each step. The algorithm avoids the process of candidate set generation and decreases the time for
counting supports due to the reduced dataset.

TEAM LinG
 xxv

Perrizo, Ding, et al. review a category of DM approaches using vertical data structures. They demonstrate their
applications in various DM areas, such as association rule mining and multi-relational DM. Vertical DM strategy aims
at addressing scalability issues by organizing data in vertical layouts and conducting logical operations on vertically
partitioned data instead of scanning the entire DB horizontally.
Integration of data sources refers to the task of developing a common schema, as well as data transformation
solutions, for a number of data sources with related content. The large number and size of modern data sources makes
manual approaches to integration increasingly impractical. Andreas Koeller provides a comprehensive overview over
DM techniques which can help to partially or fully automate the data integration process.
DM applications often involve testing hypotheses regarding thousands or millions of objects at once. The statistical
concept of multiple hypothesis testing is of great practical importance in such situations, and an appreciation of the
issues involved can vastly reduce errors and associated costs. Sach Mukherjee provides an introductory look at multiple
hypothesis testing in the context of DM.
Maria Vardaki illustrates the benefits of using statistical metadata by information systems, depicting also how such
standardization can improve the quality of statistical results. She proposes a common, semantically rich, and object-
oriented data/metadata model for metadata management that integrates the main steps of data processing and covers
all aspects of DW that are essential for DM requirements. Finally, she demonstrates how a metadata model can be
integrated in a web-enabled statistical information system to ensure quality of statistical results.
A major obstacle in DM applications is the gap between statistic-based pattern extraction and value-based decision-
making. Profit mining aims at reducing this gap. The concept and techniques proposed by Ke Wang and Senqiang Zhou
are applicable to applications under a general notion of utility.
Although a tremendous amount of progress has been made in DM over the last decade or so, many important
challenges still remain. For instance, there are still no solid standards of practice; it is still too easy to misuse DM
software; secondary data analysis without appropriate experimental design is still common; and it is still hard to choose
right kind of analysis methods for the problem in hand. Xiao Hui Liu points out that intelligent data analysis (IDA)
is an interdisciplinary study concerning the effective analysis of data, which may help advance the state of art in the
field.
In recent years, the need to extract complex tree-like or graph-like patterns in massive data collections (e.g., in
bioinformatics, semistructured or Web DBs) has become a necessity. This has led to the emergence of the research field
of graph and tree mining. This field provides many promising topics for both theoretical and engineering achievements,
and many expect this to be one of the key fields in DM research in the years ahead. Katsaros and Manolopoulos review
the most important strategic application-domains where frequent structure mining (FSM) provides significant results.
A survey is presented of the most important algorithms that have been proposed for mining graph-like and tree-like
substructures in massive data collections.
Lawrence B. Holder and Diane J. Cook are among the pioneers in the field of graph-based DM and have developed
the widely-disseminated Subdue graph-based DM system (http://ailab.uta.edu/subdue). They have directed multi-
million dollar government-funded projects in the research, development and application of graph-based DM in real-
world tasks ranging from bioinformatics to homeland security.
Graphical models such as Bayesian networks (BNs) and decomposable Markov networks (DMNs) have been widely
applied to probabilistic reasoning in intelligent systems. Automatic discovery of such models from data is desirable,
but is NP-hard in general. Common learning algorithms use single-link look-ahead searches for efficiency. However,
pseudo-independent (PI) probabilistic domains are not learnable by such algorithms. Yang Xiang introduces funda-
mentals of PI domains and explains why common algorithms fail to discover them. He further offers key ideas as to how
they can efficiently be discovered, and predicts advances in the near future.
Semantic DM is a novel research area that used graph-based DM techniques and ontologies to identify complex
patterns in large, heterogeneous data sets. Tony Hus research group at Drexel University is involved in the
development and application of semantic DM techniques to the bioinformatics and homeland security domains.
Yu-Jin Zhang presents a novel method for image classification based on feature element through association rule
mining. The feature elements can capture well the visual meanings of images according to the subjective perception
of human beings, and are suitable for working with rule-based classification models. Techniques are adapted for mining
the association rules which can find associations between the feature elements and class attributes of the image, and
the mined rules are applied to image classifications.
Results of image DB queries are usually presented as a thumbnail list. Subsequently, each of these images can be
used for refinement of the initial query. This approach is not suitable for queries by sketch. In order to receive the desired
images, the user has to recognize misleading areas of the sketch and modify these images appropriately. This is a non-

TEAM LinG
xxvi

trivial problem, as the retrieval often is based on complex, non-intuitive features. Therefore, Odej Kao presents a mosaic-
based technique for sketch feedback, which combines the best sections contained in an image DB into a single query
image.
Andrew Kusiak and Shital C. Shah emphasize the need for an individual-based paradigm, which may ensure the well-
being of patients and the success of pharmaceutical industry. The new methodologies are illustrated with various
medical informatics research projects on topics such as predictions for dialysis patients, significant gene/SNP
identifications, hypoplastic left heart syndrome for infants, and epidemiological and clinical toxicology. DWM and data
modeling will ultimately lead to targeted drug discovery and individualized treatments with minimum adverse effects.
The use of microarray DBs has revolutionized the way in which biomedical research and clinical investigation can
be conducted in that high-density arrays of specified DNA sequences can be fabricated onto a single glass slide or
chip. However, the analysis and interpretation of the vast amount of complex data produced by this technology poses
an unprecedented challenge. LinMin Fu and Richard Segall present a state-of-the-art review of microarray DM problems
and solutions.
Knowledge discovery from genomic data has become an important research area for biologists. An important
characteristic of genomic applications is the very large amount of data to be analyzed, and most of the time, it is not
possible to apply only classical statistical methods. Therefore, Jourdan, Dhaenens and Talbi propose to model
knowledge discovery tasks associated with such problems as combinatorial optimization tasks, in order to apply
efficient optimization algorithms to extract knowledge from those large datasets.
Founded on the work of Indrani Chakravarty et al.s research, handwritten signature is a behavioral biometric. There
are two methods used for recognition of handwritten signatures offline and online. While offline methods extract static
features of signature instances by treating them as images, online methods extract and use temporal or dynamic features
of signatures for recognition purposes. Temporal features are difficult to imitate, and hence online recognition methods
offer higher accuracy rates than offline methods.
Neurons are small processing units that are able to store some information. When several neurons are connected,
the result is a neural network, a model inspired by biological neural networks like the brain. Kate Smith provides useful
guidelines to ensure successful learning and generalization of the neural network model. Also, a special version in the
form of probabilistic neural networks (PNNs) is explained by Ingrid Fischer with the help of graphic transformations.
The sheer volume of multimedia data available has exploded on the Internet in the past decade in the form of webcasts,
broadcast programs and streaming audio and video. Automated content analysis tools for multimedia depend on face
detectors and recognizers; videotext extractors; speech and speaker identifiers; people/vehicle trackers; and event
locators resulting in large sets of multimodal features that can be real-valued, discrete, ordinal, or nominal. Multimedia
metadata based on such a multimodal collection of features, poses significant difficulties to subsequent tasks such as
classification, clustering, visualization and dimensionality reduction which traditionally deal only with continuous-
valued data. Aradhye and Dorai discuss mechanisms that extend tasks traditionally limited to continuous-valued feature
spaces to multimodal multimedia domains with symbolic and continuous-valued features, including (a) dimensionality
reduction, (b) de-noising, (c) visualization, and (d) clustering.
Brian C. Lovell and Shaokang Chen review the recent advances in the application of face recognition for multimedia
DM. While the technology for mining text documents in large DBs could be said to be relatively mature, the same cannot
be said for mining other important data types such as speech, music, images and video. Yet these forms of multimedia
data are becoming increasingly common on the Internet and intranets.
The goal of Web usage mining is to capture, model, and analyze the behavioral patterns and profiles of users
interacting with a Web site. Bamshad Mobasher and Yongjian Fu provide an overview of the three primary phases of
the Web mining process: data preprocessing, pattern discovery, and pattern analysis. The primary focus of their articles
is on the types of DM and analysis tasks most commonly used in Web usage mining, as well as some their typical
applications in areas such as Web personalization and Web analytics. Ji-Rong Wen explores the ways of enhancing
Web search using query log mining and Web structure mining.
In line with Mike Thelwalls opinion, scientific Web intelligence (SWI) is a research field that combines techniques
from DM, Web intelligence and scientometrics to extract useful information from the links and text of academic-related
Web pages, using various clustering, visualization and counting techniques. SWI is a type of Web mining that combines
Web structure mining and text mining. Its main uses are in addressing research questions concerning the Web, or Web-
related phenomena, rather than in producing commercially useful knowledge.
Web-enabled electronic business is generating massive amount of data on customer purchases, browsing patterns,
usage times and preferences at an increasing rate. DM techniques can be applied to all the data being collected. Richi
Nayak presents issues associated with DM for Web-enabled electronic-business.

TEAM LinG
 xxvii

Tobias Scheffer gives an overview of common email mining tasks including email filing, spam filtering and mining
communication networks. The main section of his work focuses on recent developments in mining email data for support
of the message creation process. Approaches to mining question-answer pairs and sentences are also reviewed.
Stanley Loh describes a computer-supported approach to mine discussions that occurred in chat rooms. Dennis
Mcleod explores incremental mining from news streams. JungHwan Oh summarizes the current status of video DM. J.
Ben Schafer addresses the technology used to generate recommendations.
In the abstract, a DW can be seen as a set of materialized views defined over source relations. During the initial design
of a DW, the designer faces the problem of deciding which views to materialize in the DW. This problem has been
addressed in the literature for different classes of queries and views, and with different design goals. Theodoratos and
Simitsis identify the different design goals used to formulate alternative versions of the problem and highlight the
techniques used to solve it.
Michel Schneider addresses the problem of designing a DW schema. He suggested a general model for this purpose
that integrates a majority of existing models: the notion of a well-formed structure is proposed to help design the process;
a graphic representation is suggested for drawing well-formed structures; and the classical star-snowflake structure
is represented.
Anthony Scime presents a methodology for adding external information from the World Wide Web to a DW, in
addition to the DWs domain information. The methodology assures decision makers that the added Web based data
are relevant to the purpose and current data of the DW.
Privacy and confidentiality of individuals are important issues in the information technology age. Advances in DM
technology have increased privacy concerns even more. Jack Cook and Ycel Sayg1n highlight the privacy and
confidentiality issues in DM, and survey state of the art solutions and approaches for achieving privacy preserving
DM.
Ken Goodman provides one of the first overviews of ethical issues that arise in DM. He shows that while privacy
and confidentiality often are paramount in discussions of DM, other issues including the characterization of
appropriate uses and users, and data miners intentions and goals must be considered. Machine learning in genomics
and in security surveillance are set aside as special issues requiring attention.
Increased concern about privacy and information security has led to the development of privacy preserving DM
techniques. Yehuda Lindell focuses on the paradigms for defining security in this setting, and the need for a rigorous
approach. Shamik Sural et al. present some of important approaches to privacy protection in association rule mining.
Human-computer interaction is crucial in the knowledge discovery process in order to accomplish a variety of novel
goals of DM. In Shou Hong Wangs opinion, interactive visual DM is human-centered DM, implemented through
knowledge discovery loops coupled with human-computer interaction and visual representations.
Symbiotic DM is an evolutionary approach that shows how organizations analyze, interpret, and create new
knowledge from large pools of data. Symbiotic data miners are trained business and technical professionals skilled in
applying complex DM techniques and business intelligence tools to challenges in a dynamic business environment.
Athappilly and Rea opened the discussion on how businesses and academia can work to help professionals learn, and
fuse the skills of business, IT, statistics, and logic to create the next generation of data miners.
Yiyu Yao and Yan Zhao first make an immediate comparison between scientific research and DM and add an
explanation construction and evaluation task to the existing DM framework. Explanation-oriented DM offers a new
perspective, which has a significant impact on the understanding of the complete process of DM and effective
applications of DM results.
Traditional DM views the output from any DM initiative as a homogeneous knowledge product. Knowledge
however, always is a multifaceted construct, exhibiting many manifestations and forms. It is the thesis of Nilmini
Wickramasinghes discussion that a more complete and macro perspective, and a more balanced approach to knowledge
creation, can best be provided by taking a broader perspective of the knowledge product resulting from the KDD
process: namely, by incorporating a people-based perspective into the traditional KDD process, and viewing knowledge
as the multifaceted construct it is. This in turn will serve to enhance the knowledge base of an organization, and facilitate
the realization of effective knowledge.
Fabrice Muhlenbach and Ricco Rakotomalala are the authors of an original supervised multivariate discretization
method called HyperCluster Finder. Their major contributions to the research community are present in a DM software
called TANAGRA, which is freely available on Internet.
Recently there have been many efforts to apply DM techniques to security problems, including homeland security
and cyber security. Bhavani Thuraisingham (the inventor of three patents for MITRE) examines some of these
developments in DM in general and link analysis in particular, and shows how DM and link analysis techniques may
be applied for homeland security applications. Some emerging trends are also discussed.

TEAM LinG
xxviii

In order to reduce financial statement errors and fraud, Garrity, ODonnell and Sanders proposed an architecture
that provides auditors with a framework for an effective continuous auditing environment that utilizes DM.
The applications of DWM are everywhere: from Kernel Methods in Chemoinformatics to Data Mining for Damage
Detection in Engineering Structures; from Predicting Resource Usage for Capital Efficient Marketing to Mining for
Profitable Patterns in the Stock Market; from Financial Ratio Selection for Distress Classification to Material
Acquisitions Using Discovery Informatics Approach; from Resource Allocation in Wireless Networks to Reinforcing
CRM with Data Mining; from Data Mining Medical Digital Libraries to Immersive Image Mining in Cardiology; from
Data Mining in Diabetes Diagnosis and Detection to Distributed Data Management of Daily Car Pooling Problems;
and from Mining Images for Structure to Automatic Musical Instrument Sound ClassificationThe list of DWM
applications is endless and the future of DWM is promising.
Knowledge explosion pushes DWM, a multidisciplinary subject, to ever-expanding regions. Inclusion, omission,
emphasis, evolution and even revolution are part of our professional life. In spite of our efforts to be careful, should
you find any ambiguities or perceived inaccuracies, please contact me at wangj@mail.montclair.edu.

REFERENCES

Fayyad, U., Piatetsky-Shapiro, G., Smyth, P., & Uthurusamy, R. (1996). Advances in knowledge discovery and data
mining. AAAI/MIT Press.
Giudici, P. (2003). Applied data mining: Statistical methods for business and industry. John Wiley.
Han, J., & Kamber, M. (2000). Data mining: Concepts and techniques. Morgan Kaufmann Publishers.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of data mining. MIT Press.
Turban, E., Aronson, J. E., & Liang, T. P. (2005). Decision support systems and intelligent systems. Upper Saddle River,
NJ: Pearson Prentice Hall.
Wang, J. (2003). Data mining: Opportunities and challenges. Hershey, PA: Idea Group Publishing.

TEAM LinG
 xxix

Acknowledgments

The editor would like to thank all of the authors for their insights and excellent contributions to this book. I also want
to thank the group of anonymous reviewers who assisted me in the peer-reviewing process and provided comprehen-
sive, critical and, constructive reviews. Each Editorial Advisory Board member has made a big contribution in guidance
and assistance.
The editor wishes to acknowledge the help of all involved in the development process of this book, without whose
support the project could not have been satisfactorily completed. Linxi Liao and MinSun Ku, two graduate assistants,
are hereby graciously acknowledged for their diligent work. I owe my thanks to Karen Dennis for lending a hand in the
tedious process of proof-reading. A further special note of thanks goes to the staff at Idea Group Inc., whose
contributions have been invaluable throughout the entire process, from inception to final publication. Particular thanks
go to Sara Reed, Jan Travers, and Rene Davies, who continuously prodded via e-mail to keep the project on schedule,
and to Mehdi Khosrow-Pour, whose enthusiasm motivated me to accept his invitation to join this project.
My appreciation is also due the Global Education Center at MSU for awarding me a Global Education Fund. I would
also like to extend my thanks to my brothers Zhengxian, Shubert (an artist, http://www.portraitartist.com/wang/bio.htm),
and sister Jixian, who stood solidly behind me and contributed in their own sweet little ways. Thanks go to all Americans,
since it would not have been possible for the four of us to come to the U.S. without their support of different scholarships.
Finally, I want to thank my family: my parents, Houde Wang and Junyan Bai for their encouragement; my wife Hongyu
for her unfailing support, and my son Leigh for being without a dad during this project. By the way, our second boy
Leon was born on August 4, 2004. Like a baby, DWM has a bright and promising future.

John Wang, Ph.D.

Wangj@mail.montclair.edu
Dept. Information and Decision Sciences
School of Business
Montclair State University
Montclair, New Jersey
USA

TEAM LinG
xxx

About the Editor

John Wang, Ph.D., is a full professor in the Department of Information and Decision Sciences at Montclair State
University (MSU), USA. Professor Wang has published 89 refereed papers and three books. He is on the editorial board
of the International Journal of Cases on Electronic Commerce and has been a guest editor and referee for Operations
Research, IEEE Transactions on Control Systems Technology, and many other highly prestigious journals. His long-
term research goal is on the synergy of operations research, data mining and cybernetics.

TEAM LinG

Action Rules
Zbigniew W. Ras
University of North Carolina, Charlotte, USA
Angelina Tzacheva
University of North Carolina, Charlotte, USA
Li-Shiang Tsay
University of North Carolina, Charlotte, USA
INTRODUCTION
There are two aspects of interestingness of rules that
have been studied in data mining literature, objective
and subjective measures (Liu, 1997; Adomavicius &
Tuzhilin, 1997; Silberschatz & Tuzhilin, 1995, 1996).
Objective measures are data-driven and domain-inde-
pendent. Generally, they evaluate the rules based on
their quality and similarity between them. Subjective
measures, including unexpectedness, novelty and ac-
tionability, are user-driven and domain-dependent.
A rule is actionable if user can do an action to his/her
advantage based on this rule (Liu, 1997). This defini-
tion, in spite of its importance, is too vague and it leaves
open door to a number of different interpretations of
actionability. In order to narrow it down, a new class of
rules (called action rules) constructed from certain
pairs of association rules, has been proposed in Ras &
Wieczorkowska (2000). A formal definition of an ac-
tion rule was independently proposed in Geffner &
Wainer (1998). These rules have been investigated fur-
ther in Tsay & Ras (2004) and Tzacheva & Ras (2004).
To give an example justifying the need of action rules,
let us assume that a number of customers have closed
their accounts at one of the banks. We construct, possi-
bly the simplest, description of that group of people and
next search for a new description, similar to the one we
have, with a goal to identify a new group of customers
from which no one left that bank. If these descriptions
have a form of rules, then they can be seen as actionable
rules. Now, by comparing these two descriptions, we
may find the cause why these accounts have been closed
and formulate an action, which if undertaken by the bank,
may prevent other customers from closing their ac-
counts. Such actions are stimulated by action rules and
they are seen as precise hints for actionability of rules.
For example, an action rule may say that by inviting
people from a certain group of customers for a glass of
wine by the bank, it is guaranteed that these customers
will not close their accounts and they do not move to
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
1
A
another bank. Sending invitations by regular mail to all
these customers or inviting them personally by giving
them a call are examples of an action associated with
that action rule.
In paper by Ras & Gupta (2002), authors assume that
information system is distributed and its sites are au-
tonomous. They show that it is wise to search for action
rules at remote sites when action rules extracted at the
client site cannot be implemented in practice (sug-
gested actions are too expensive or too risky). The
composition of two action rules, not necessary ex-
tracted at the same site, was defined in Ras & Gupta
(2002). Authors gave assumptions guaranteeing the cor-
rectness of such a composition. One of these assump-
tions requires that semantics of attributes, including the
interpretation of null values, have to be the same at both
sites. This assumption is relaxed in Tzacheva & Ras
(2004) since authors allow different granularities of the
same attribute at involved sites. In the same paper, they
introduce the notion of a cost and feasibility of an action
rule. Usually, a number of action rules or chains of
action rules can be applied to reclassify a certain set of
objects. The cost associated with changes of values
within one attribute is usually different than the cost
associated with changes of values within another at-
tribute. The strategy for replacing the initially extracted
action rule by a composition of new action rules, dy-
namically built, was proposed in the paper by Tzacheva
& Ras (2004). This composition of rules uniquely de-
fines a new action rule and it was built with a goal to
lower the cost of reclassifying objects supported by the
initial action rule.
BACKGROUND
In the paper by Ras & Wieczorkowska (2000), the
notion of an action rule was introduced. The main idea
was to generate, from a database, special type of rules
which basically form a hint to users showing a way to
TEAM LinG

reclassify objects with respect to some distinguished
attribute (called a decision attribute). Clearly, each
relational schema gives a list of attributes used to
represent objects stored in a database. Values of some
of these attributes, for a given object, can be changed
and this change can be influenced and controlled by
user. However, some of these changes (for instance
profit) cannot be done directly to a decision attribute.
In such a case, definitions of this decision attribute in
terms of other attributes (called classification attributes)
have to be learned. These new definitions are used to
construct action rules showing what changes in values of
some attributes, for a given class of objects, are needed
to reclassify objects the way users want. But users may
still be either unable or unwilling to proceed with ac-
tions leading to such changes. In all such cases, we may
search for definitions of values of any classification
attribute listed in an action rule. By replacing a value of
such attribute by its definition extracted either locally
or at remote sites (if system is distributed), we con-
struct new action rules, which might be of more interest
to business users than the initial rule.
MAIN THRUST
The technology dimension will be explored to clarify
the meaning of actionable rules including action rules
and extended action rules.
Action Rules Discovery in a Stand-
alone Information System
An information system is used for representing
knowledge. Its definition, given here, is due to Pawlak
(1991).
By an information system we mean a pair S = (U, A),
where:
1. U is a nonempty, finite set of objects (object
identifiers),
2. A is a nonempty, finite set of attributes, that is,
a:U Va for a A, where Va is called the domain
of a.
Information systems can be seen as decision tables.
In any decision table together with the set of attributes
a partition of that set into conditions and decisions is
given. Additionally, we assume that the set of conditions
is partitioned into stable and flexible conditions (Ras &
Wieczorkowska, 2000).
Attribute a A is called stable for the set U if its values
assigned to objects from U can not change in time.
Action Rules
Otherwise, it is called flexible. Date of Birth is an example
of a stable attribute. Interest rate on any customer
account is an example of a flexible attribute. For simplicity
reasons, we will consider decision tables with only one
decision. We adopt the following definition of a decision
table:
By a decision table we mean an information system
S = (U, A1 A2 {d}), where d A1 A2 is a distin-
guished attribute called decision. The elements of A1 are
called stable conditions, whereas the elements of A2
{d} are called flexible conditions. Our goal is to change
values of attributes in A1 for some objects from U so the
values of the attribute d for these objects may change as
well. Certain relationships between attributes from A1 and
the attribute d will have to be discovered first.
By Dom(r) we mean all attributes listed in the IF part
of a rule r extracted from S. For example, if r = [
(a1,3)*(a2,4) (d,3)] is a rule, then Dom(r) = {a1,a2}.
By d(r) we denote the decision value of rule r. In our
example d(r) = 3.
If r1, r2 are rules and B A1 A2 is a set of attributes,
then r1/B = r2/B means that the conditional parts of rules
r1, r2 restricted to attributes B are the same.
For example, if r1 = [(a1,3) (d,3)], then r1/{a1} = r/
{a1}.
Assume also that (a, v w) denotes the fact that the
value of attribute a has been changed from v to w.
Similarly, the term (a, v w)(x) means that a(x)=v has
been changed to a(x)=w. Saying another words, the
property (a,v) of an object x has been changed to prop-
erty (a,w). Assume now that rules r1, r2 have been
extracted from S and r1/A1 = r2/A1, d(r1)=k1, d(r2)=k2
and k1< k2. Also, assume that (b1, b2,, bp) is a list of
all attributes in Dom(r1) Dom(r2) A2 on which r1,
r2 differ and r1(b1)= v1, r1(b2)= v2,, r1(bp)= vp,
r2(b1)= w1, r2(b2)= w2,, r2(bp)= wp.
By (r1,r2)-action rule on xU we mean a statement:
[ (b1, v1 w1) (b2, v2 w2) (bp, vp
wp)](x) [(d, k1 k2)](x).
If the value of the rule on x is true then the rule is
valid. Otherwise it is false.
Let us denote by U<r1> the set of all customers in U
supporting the rule r1. If (r1,r2)-action rule is valid on
xU <r1> then we say that the action rule supports the new
profit ranking k2 for x.
To define an extended action rule (Ras & Tsay, 2003),
let us assume that two rules are considered. We present
Table 1.
A (St) B (Fl) C (St) E (Fl) G (St) H (Fl) D (Decision)
a1 b1 c1 e1 d1
a1 b2 g2 h2 d2
TEAM LinG
Action Rules
them in Table 1 to better clarify the process of construct-
ing extended action rules. Here, St means stable classi-
fication attribute and Fl means flexible one.
In a classical representation, these two rules will have
a form:
r1 = [ a1 * b1 * c1 * e1 d1 ] ,
r2 = [ a1 * b2 * g2 * h2 d2 ].
Assume now that object x supports rule r1, which
means that it is classified as d1. In order to reclassify x
to class d2, we need to change its value B from b1 to b2
but also we have to require that G(x)=g2 and that the
value H for object x has to be changed to h2. This is the
meaning of the extended (r1,r2)-action rule given below:
[(B, b1 b2) (G = g2) (H, h2)](x) (D, defined as an information system coupled with a set of
d1 d2)(x).
Assume now that by Sup(t) we mean the number of
tuples having property t .
By the support of the extended (r1,r2)-action rule
(given above) we mean:
Sup[(A=a1)*(B=b1)*(G=g2)]
By the confidence of the extended (r1,r2)-action rule
(given above) we mean:
[Sup[(A=a1)*(B=b1)*(G=g2)*(D=d1)]/
Sup[(A=a1)*(B=b1)*(G=g2)]]
[Sup[(A=a1)*(B=b2)*(C=c1)*(D=d2)]/
Sup[(A=a1)*(B=b2)*(C=c1)]].
To give another example of extended action rule,
assume that S=(U,A1 A2 {d}) is a decision table repre-
sented by Table 2. Assume that A1= {c, b}, A 2 = {a}.
For instance, rules r1=[(a,1)*(b,1) (d,L)], r2=[(c,2) *
(a,2) (d,H)] can be extracted from S, where U <r1> = {x1,
x4}. Extended (r1,r2)-action rule
[ (a, 1 2) (c = 2)](x) [(d, L H)](x)
is only supported by object x1. The corresponding (r1,r2)-
action rule
Table 2.
c a b d
x1 2 1 1 L
x2 1 2 2 L
x3 2 2 1 H
x4 1 1 1 L
[(a, 1 2)](x) [(d, L H)](x) is supported by x1
and x4. A
The confidence of an extended action rule is higher
than the confidence of the corresponding action rule
because all objects making the confidence of that ac-
tion rule lower have been removed from its set of
support.
Actions Rules Discovery in Distributed
Autonomous Information System
In Ras & Dardzinska (2002), the notion of a Distributed
Autonomous Knowledge System (DAKS) framework
was introduced. DAKS is seen as a collection of knowl-
edge systems where each knowledge system is initially
rules (called a knowledge base) extracted from that
system. These rules are transferred between sites due
to the requests of a query answering system associated
with the client site. Each rule transferred from one site
of DAKS to another remains at both sites.
Assume now that information system S represents
one of DAKS sites. If rules extracted from S = (U, A1 A2
{d}), describing values of attribute d in terms of
attributes from A 1 A2, do not lead to any useful action
rules (user is not willing to undertake any actions sug-
gested by rules), we may:
1) search for definitions of flexible attributes listed
in the classification parts of these rules in terms
of other local flexible attributes (local mining
for rules),
2) search for definitions of flexible attributes listed
in the classification parts of these rules in terms
of flexible attributes from another site (mining
for rules at remote sites),
3) search for definitions of decision attributes of
these rules in terms of flexible attributes from
another site (mining for rules at remote sites).
Another problem, which has to be taken into consid-
eration, is the semantics of attributes that are common
for a client site and some of the remote sites. This
semantics may easily differ from site to site. Some-
time, such a difference in semantics can be repaired
quite easily. For instance, if Temperature in Celsius is
used at one site and Temperature in Fahrenheit at the
other, a simple mapping will fix the problem. If infor-
mation systems are complete and two attributes have
the same name and differ only in their granularity level,
a new hierarchical attribute can be formed to fix the
problem. If databases are incomplete, the problem is more
TEAM LinG

complex because of the number of options available to
interpret incomplete values (including null vales). The
problem is especially difficult in a distributed framework
when chase techniques based on rules extracted at the
client and at remote sites are used by a client site to impute
current values by values which are less incomplete. These
problems are presented and partial solutions given in Ras
& Dardzinska (2002).
Now, let us assume that the action rule
r = [(b1, v1 w1) (b 2, v2 w2) (bp, vp
wp)](x) (d, k 1 k2)(x),
extracted from system S, does not provide any useful
hint to a user for its actionability.
In this case we may look for a new action rule
(extracted either from S or from some of its remote
sites)
r1 = [(bj1, vj1 wj1)(bj2, vj2 wj2)(bjq, vjq
wjq)](y) (bj, vj w j)(y)
which concatenated with r may provide better hint for its
actionability. For simplicity reason, we assume that the
semantics and the granularity levels of all attributes
listed in both information systems are the same.
Concatenation [r1 r] is a new action rule (called
global), defined as:
[(b1, v1 w 1) [(b1j, v1j w 1j) (bj2, vj2 wj2)
( b jq, vjq wjq)] ( bp, vp wp)](x) (d, FUTURE TRENDS
k1 k2)(x)
where x is an object in S = (X,A,V).
Some of the attributes in {bj1, bj2,, bjq} may not
belong to A. Also, the support of r1 is calculated in the
information system from which r 1 was extracted. Let us
denote that system by Sm = (Xm, Am, Vm) and the set of
objects in m supporting r1 by SupSm(r1). Assume that
SupS(r) is the set of objects in S supporting rule r. The
domain of [r1 r] is the same as the domain of r, which is
equal to SupS(r). Before we define the notion of a similarity
between two objects belonging to two different informa-
tion systems, we assume that A={b1, b2, b3, b4}, Am={b1,
b2, b3, b5, b6}, and objects xX, yXm are defined by Table
3 given below.
The similarity (x, y) between x and y is defined as:
[1+0+0+1/2+1/2+1/2]=5/12.
Table 3.
b1 b2 b3 b4 b5 b6
x v1 v2 v3 v4
y v1 w2 w3 w5 w6
Action Rules
To give more formal definition of similarity, we assume
that:
(x,y) = [{(bi(x), bi(y)) : bi (A Am)}] / card(A
Am), where:
(bi(x), bi(y)) = 0, if bi(x) bi(y)
(bi(x), bi(y)) = 1, if bi(x) = bi(y)
(bi(x), bi(y)) = 1/2, if either bi(x) is undefined.
Also, assume that
(x, SupSm(r1)) = max{(x, y) : y SupSm(r1)}, for
each x Sup S(r)
By the confidence of the action rule [r 1 r] we mean
[{(x, SupSm(r1)) : x SupS(r)} / card(Sup S(r))]
Conf(r 1) Conf(r)
where Conf(r) is the confidence of the rule r in S and
Conf(r 1) is the confidence of the rule r1 in Sm.
If we allow to concatenate action rules extracted
from S with action rules extracted at other sites of
DAKS, we are increasing the total number of generated
action rules and the same our chance to find more
suitable action rules for reclassifying objects in S is
also increased.
Business user may be either unable or unwilling to
proceed with actions leading to desired reclassifica-
tions of objects. Undertaking the actions may be trivial,
feasible to an acceptable degree, or may be practically
very difficult. Therefore, the notion of a cost of an
action rule is of very great importance. New strategies
for discovering action rules of the lowest cost in DAKS,
based on ontologies, will be investigated.
CONCLUSION
Attributes are divided into two groups: stable and flex-
ible. By stable attributes we mean attributes which val-
ues cannot be changed (for instance, age or maiden
name). On the other hand attributes (like percentage rate
or loan approval to buy a house) which values can be
changed are called flexible. Rules are extracted from a
decision table, using standard KD methods, with prefer-
ence given to flexible attributes so mainly they are
listed in a classification part of rules. Most of these rules
TEAM LinG
Action Rules
can be seen as actionable rules and the same used to
construct action-rules.
REFERENCES
Adomavicius, G., & Tuzhilin, A. (1997). Discovery of
actionable patterns in databases: The action hierarchy
approach. In Proceedings of KDD97 Conference, New-
port Beach, CA. Menlo Park, CA: AAAI Press.
Geffner, H., & Wainer, J. (1998). Modeling action,
knowledge and control. In H. Prade (Ed.), ECAI 98, 13th
European Conference on AI (pp. 532-536). New York:
John Wiley & Sons.
Liu, B., Hsu, W., & Chen, S. (1997). Using general
impressions to analyze discovered classification rules.
In Proceedings of KDD97 Conference, Newport Beach,
CA. Menlo Park, CA: AAAI Press.
Pawlak, Z. (1991). Rough sets: Theoretical aspects of
reasoning about data. Kluwer.
Ras, Z., & Dardzinska, A. (2002). Handling semantic
inconsistencies in query answering based on distributed
knowledge mining. In Foundations of Intelligent Sys-
tems, Proceedings of ISMIS02 Symposium (pp. 66-
74). LNAI (No. 2366). Berlin: Springer-Verlag.
Ras, Z., & Gupta, S. (2002). Global action rules in distrib-
uted knowledge systems. Fundamenta Informaticae Jour-
nal, 51(1-2), 175-184.
Ras, Z., & Wieczorkowska, A. (2000). Action rules:
How to increase profit of a company. In D.A. Zighed, J.
Komorowski, & J. Zytkow (Eds.), Principles of Data
Mining and Knowledge Discovery. (Proceedings of
PKDD00 (pp. 587-592), LNAI (No. 1910), Lyon, France.
Berlin: Springer-Verlag.
Silberschatz, A., & Tuzhilin, A. (1995). On subjective
measures of interestingness in knowledge discovery. In
Proceedings of KDD95 Conference. Menlo Park, CA:
AAAI Press.
Silberschatz, A., & Tuzhilin, A. (1996). What makes
patterns interesting in knowledge discovery systems.
IEEE Transactions on Knowledge and Data Engineer-
ing, 5(6).
Tsay, L.-S., & Ras, Z.W. (2005). Action rules discovery
system DEAR, method and experiments. Journal of Ex-
perimental and Theoretical Artificial Intelligence, Spe-
cial Issue on Knowledge Discovery, 17(1-2), 119-128. A
Tzacheva, A., & Ras, Z.W. (2003). Discovering non-
standard semantics of semi-stable attributes. In I. Russell
& S. Haller (Eds.), Proceedings of FLAIRS-2003 (pp. 330-
334), St. Augustine, Florida. Menlo Park, CA: AAAI
Press.
Tzacheva, A., & Ras, Z.W. (2005). Action rules mining.
International Journal of Intelligent Systems, Special
Issue on Knowledge Discovery (In press).
KEY TERMS
Actionable Rule: A rule is actionable if user can do
an action to his/her advantage based on this rule.
Autonomous Information System: Information
system existing as an independent entity.
Domain of Rule: Attributes listed in the IF part of a
rule.
Flexible Attribute: Attribute is called flexible if
its value can be changed in time.
Knowledge Base: A collection of rules defined as
expressions written in predicate calculus. These rules
have a form of associations between conjuncts of values
of attributes.
Ontology: An explicit formal specification of how
to represent objects, concepts and other entities that are
assumed to exist in some area of interest and relation-
ships holding among them. Systems that share the same
ontology are able to communicate about domain of
discourse without necessarily operating on a globally
shared theory. System commits to ontology if its ob-
servable actions are consistent with definitions in the
ontology.
Semantics: The meaning of expressions written in
some language, as opposed to their syntax, which de-
scribes how symbols may be combined independently
of their meaning.
Stable Attribute: Attribute is called stable for the
set U if its values assigned to objects from U cannot
change in time.
TEAM LinG
6
Active Disks for Data Mining
Alexander Thomasian
New Jersey Institute of Technology, USA
INTRODUCTION
Active disks allow the downloading of certain types of
processing from the host computer onto disks, more
specifically the disk controller, which has access to a
limited amount of local memory serving as the disk
cache. Data read from a disk may be sent directly to the
host computer as usual or processed locally at the disk.
Only filtered information is uploaded to the host com-
puter. In this article, We am interested in downloading
data mining and database applications to the disk control-
ler.
This article is organized as follows. A section on
background information is followed by a discussion of
data-mining operations and hardware technology trends.
The more influential active disk projects are reviewed
next. Future trends and conclusions appear last.
BACKGROUND
Data mining has been defined as the use of algorithms to
extract information and patterns as part of Knowledge
Discovery in Databases (KDD) (Dunham, 2003). Cer-
tain aspects of data mining introduced a decade ago were
computationally challenging for the computer systems
at that time. This was partially due to the high cost of
computing and the uncertainty associated with the value
of information extracted by data mining. Data mining
has become more viable economically with the advent
of cheap computing power based on UNIX, Linux, and
Windows operating systems.
The past decade has shown a tremendous increase in
the processing power of computer systems, which has
made possible the previously impossible. On the other
hand, with higher disk transfer rates, a single processor
is required to process the incoming data from one disk,
but the number of disks associated with a multiproces-
sor server usually exceeds the number of processors,
that is, the data transfer rate is not matched by the
processing power of the host computer.
Computer systems have been classified into shared-
everything systems (multiple processors sharing the
main memory and several disks), shared-nothing sys-
tems (multiple computers with connectivity via an inter-
connection network), and shared-disk systems (several
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
computers sharing a set of disks). Parallel data mining is
appropriate for large-scale data mining (Zaki, 1999) and
the active disk paradigm can be considered as a low-cost
scheme for parallel data mining.
In active disks the host computer partially or fully
downloads an application, such as data mining, onto the
microprocessors serving as the disk controllers. These
microprocessors have less computing power than those
associated with servers, but because servers tend to have
a large number of disks, the raw computing power asso-
ciated with the disk controllers may easily exceed the
(raw) computing power of the server. Computing power
is estimated as the sum of the MIPS (millions of in-
structions per second) ratings of the microprocessors.
The computing power associated with the disks comes
in the form of a shared-nothing system, with connectiv-
ity limited to indirect communication via the host.
Amdahls law on the efficiency of parallel processing is
applicable both to multiprocessors and disk control-
lers: If a fraction F of the processing can be carried out
in parallel with a degree of parallelism P, then the
effective degree of parallelism is 1/(1F+F/P)
(Hennessy & Patterson, 2003). There is a higher degree
of interprocessor communication cost for disk control-
lers.
A concise specification of active disks is as fol-
lows:
A number of important I/O intensive applications can
take advantage of computational power available
directly at storage devices to improve their overall
performance, more effectively balance their
consumption of system wide resources, and provide
functionality that would not be otherwise available
(Riedel, 1999).
Active disks should be helpful from the following
viewpoint. The size of the database and the processing
requirements for decision support systems (DSS) is
growing rapidly. This is attributable to the length of the
history, the level of the detail being saved, and the
increased number of users and queries (Keeton,
Patterson, & Hellerstein, 1998). The first two factors
contribute to the capacity requirement, while all three
factors contribute to the processing requirement.
With the advent of relational databases (in the mid-
1970s) perceived actual inefficiencies associated with
TEAM LinG
Active Disks for Data Mining
the processing of (SQLstructured query language)
queries, as compared to low-level data manipulation lan-
guages (DMLs) for hierarchical and network databases,
led to numerous proposals for database machines (Stanley
& Su, 1983). The following categorization is given: (a)
intelligent secondary storage devices, proposed to speed
up text retrieval but later modified to handle relational
algebra operators, (b) database filters to accelerate table
scan, such as content-addressable file store (CAFS) from
International Computers Limited (ICL), (c) associative
memory systems, which retrieve data by content, and (d)
database computers, which are mainly multicomputers.
Intelligent secondary storage devices can be further
classified (Riedel, 1999): (a) processor per track
(PPT), (b) processor per head (PPH), and (c) proces-
sor per disk (PPD). Given that modern disks have
thousands of tracks, the first solution is out of the
question. The second solution may require the R/W
heads to be aligned simultaneously to access all the
tracks on a cylinder, which is not feasible. NCRs
Teradata DBC/1012 database machine (1985) is a
multicomputer PPD system.
To summarize, according to the active disk para-
digm, the host computer offloads the processing of
data-warehousing and data-mining operators onto the
embedded microprocessor controller in the disk drive.
There is usually a cache associated with each disk drive,
which is used to hold prefetched data but can be also
used as a small memory, as mentioned previously.
MAIN THRUST
Data mining, which requires high data access band-
widths and is computationally intensive, is used to illus-
trate active disk applications.
Data-Mining Applications
The three main areas of data mining are (a) classifica-
tion, (b) clustering, and (c) association rule mining
(Dunham, 2003). A brief review is given of the methods
discussed in this article.
Classification assigns items to appropriate classes
by using the attributes of each item. When regression is
used for this purpose, the input values are the item
attributes, and the output is its class. The k-nearest-
neighbor (k-NN) method uses a training set, and a new
item is placed in the set, whose entries appear most
among the k-NNs of the target item.
K-NN queries are also used in similarity search, for
example, content based image retrieval (CBIR). Ob-
jects (images) are represented by feature vectors in the
areas of object color, texture, and so forth. Similarity of
objects is determined by the distance of their feature
vectors. k-NN queries find the k objects with the smallest A
(squared) Euclidean distances.
Clustering methods group items, but unlike classifica-
tion, the groups are not predefined. A distance measure,
such as the Euclidean distance between feature vectors of
the objects, is used to form the clusters. The agglomerative
clustering algorithm is a hierarchical algorithm, which
starts with as many clusters as there are data items.
Agglomerative clustering tends to be expensive.
Non-hierarchical or partitional algorithms compute
the clusters more efficiently. The popular k-means clus-
tering method is in the family of squared-error cluster-
ing algorithms, which can be implemented as follows:
(1) Designate k randomly selected points from n points
as the centroids of the clusters; (2) assign a point to the
cluster, whose centroid is closest to it, based on Euclid-
ean or some other distance measure; (3) recompute the
centroids for all clusters based on the items assigned to
them; (4) repeat steps (2 through 3) with the new cen-
troids until there is no change in point membership. One
measure of the quality of clustering is the sum of
squared distances (SSD) of points in each cluster with
respect to its centroid. The algorithm may be applied
several times and the results of the iteration with the
smallest SSD selected. Clustering of large disk-resi-
dent datasets is a challenging problem (Dunham, 2003).
Association rule mining (ARM) considers market-
basket or shopping-cart data, that is, the items purchased
on a particular visit to the supermarket. ARM first
determines the frequent sets, which have to meet a
certain support level. For example, s% support for two
items A and B, such as bread and butter, implies that they
appear together in s percent of transactions. Another
measure is the confidence level, which is the ratio of the
support for the set intersection of A and B divided by the
support for A by itself. If bread and butter appear to-
gether in most market-basket transactions, then there is
high confidence that customers who buy bread also buy
butter. On the other hand, this is meaningful only if a
significant fraction of customers bought bread, that is,
the support level is high. Multiple passes over the data
are required to find all association rules (with a lower
bound for the support) when the number of objects is
large.
Algorithms to reduce the cost of ARM include sam-
pling, partitioning (the argument for why this works is
that a frequent set of items must be frequent in at least
one partition), and parallel processing (Zaki, 1999).
Hardware Technology Trends
A computer system, which may be a server, a worksta-
tion, or a PC, has three components most affecting its
TEAM LinG

performance: one or more microprocessors, a main memory,
and magnetic disk storage. My discussion of technology
trends is based on Patterson and Keeton (1998).
The main memory consists of dynamic random access
memory (DRAM) chips. Memory chip capacity is qua-
drupled every three years so that the memory capacity
versus cost ratio is increasing 25% per year. The memory
access latency is about 150 nanoseconds and, in the case
of RAMBUS DRAM, the bandwidth is 800 to 1600 MB per
second. The access latency is dropping 7% per year, and
the bandwidth is increasing 20% per year. According to
Moores law, processor speed increases 60% per year
versus 7% for main memories, so that the processor-
memory performance gap grows 50% per year. Multilevel
cache memories are used to bridge this gap and reduce the
number of processor cycles per instruction (Hennessy
& Patterson, 2003).
According to Greg Papadopulos (at Sun
Microsystems), the demand for database applications
exceeds the increase in central processing unit (CPU)
speed according to Moores law (Patterson & Keeton,
1998). Both are a factor of two, but the former is in 9 to
12 months, and the latter is in 18 months. Consequently,
the so-called database gap is increasing with time.
Disk capacity is increasing at the rate of 60% per
year. This is due to dramatic increases in magnetic re-
cording density. Disk access time consists of queueing
time, controller time, seek time, rotational latency, and
transfer time. The increased disk RPMs (rotations per
minute) and especially increased linear recording densi-
ties have resulted in very high transfer rates, which are
increasing 60% annually. The sum of seek time and
rotational latency, referred to as positioning time, is of
the order of 10 milliseconds and is decreasing very
slowly (8% per year). Utilizing disk access bandwidth is
a very important consideration and is the motivation
behind freeblock scheduling (Lumb, Schindler, Ganger,
Nagle, & Riedel, 2000; Riedel, Faloutsos, Ganger, &
Nagle, 2000).
Active Disk Projects
There have been several concurrent activities in the area
of active disks at Carnegie Mellon University, the Uni-
versity of California at Santa Barbara, University of
Maryland, University of California at Berkeley, and in
the SmartSTOR project at IBMs Almaden Research
Center.
Active1 Disk Projects at CMU
Most of this effort is summarized in (Riedel, Gibson, &
Faloutsos, 1998).
Active Disks for Data Mining
Table scan is a costly operation in relational data-
bases, which is applied selectively when the search
argument is not sargable, that is, not indexed. The data
filtering for table scans can be carried out in parallel if the
data is partitioned across several disks. A GROUP BY
SQL statement computes a certain value, such as the
minimum, the maximum, or the mean, based on some
classification of the rows in the table under consider-
ation, for example, undergraduate standing. To compute
the overall average GPA, each disk sends its mean GPA
and the number of participating records to the host
computer.
A table scan outperforms indexing in processing k-
NN queries for high dimensions. There is the additional
cost of building and maintaining the index. A synthetic
dataset associated with the IBM Almadens Quest
project for loan applications is used to experiment with
k-NN queries. The relational table contains the follow-
ing attributes: age, education, salary, commission, zip
code, make of car, cost of house, loan amount, years
owned. In the case of categorical attributes, an exact
match is required.
The Apriori algorithm for ARM is also considered
in determining whether customers who purchase a par-
ticular set of items also purchase an additional item,
but this is meaningful at a certain level of support.
More rules are generated for lower values of support,
so the limited size of the disk cache may become a
bottleneck.
Analytical models and measurement results from a
prototype show that active disks scale beyond the point
where the server saturates.
Freeblock Scheduling
This CMU project emphasizes disk performance from
the viewpoint of maximizing disk arm utilization (Lumb
et al., 2000). Freeblock scheduling utilizes opportu-
nistic reading of low-priority blocks of data from disk,
while the arm having completed the processing of a
high priority request is moving to process another such
request. Opportunities for freeblock scheduling di-
minish as more and more blocks are being read. This is
because blocks located centrally will be accessed right
away, although other disk blocks located at extreme
disk cylinders may require an explicit access.
In one scenario a disk processes requests by an OLTP
(online transaction processing) application and a back-
ground ARM application. OLTP requests have a higher
priority because transaction response time should be as
low as possible, but ARM requests are processed as
freeblock requests. OLTP requests access specific
records, but ARM requires multiple passes over the
TEAM LinG
Active Disks for Data Mining
dataset in any order. This is a common feature of algo-
rithms suitable for freeblock scheduling.
In the experimental study, OLTP requests are gener-
ated by transactions running at a given multiprogramming
level (MPL) with a certain think time, that is, the time
before the transaction generates its next request. Re-
quests are to 8 KB blocks, and the Read:Write ratio is 2:1,
as in the TPC-C benchmark (see http://www.tpc.org),
while the background process accesses 4 KB blocks. The
bandwidth due to freeblock requests increases with in-
creasing MPL. Initiating low-priority requests when the
disk has a low utilization is not considered in this study,
because such accesses would result in an increase in the
response time of disk accesses on behalf of the OLTP
application. Additional seeks are required to access the
remaining blocks so that the last 5% of requests takes 30%
of the time of a full scan.
Active Disks at UCSB/Maryland
The host computer acts as a coordinator, scheduler, and
combiner of results, while the bulk of processing is
carried out at the disks (Acharya, Uysal, & Saltz, 1998).
The computation at the host initiates disklets at the
disks. Disklets are disallowed to initiate disk accesses
and to allocate and free memory in the disk cache. All
these functions are carried out by the host computers
operating system (OS). Disklets can only access memory
locations (in a disks cache) within certain bounds speci-
fied by the host computers OS.
Implemented are SELECT, GROUP BY, and
DATACUBE operator, which computes GROUP BYs
for all possible combinations of a list of attributes
(Dunham, 2003), external sort, image convolution, and
generating composite satellite images.
The SmartSTOR Project2
SmartSTOR is significantly different from the previous
projects. It is argued in Hsu, Smith, and Young (2000)
that enhancing the computing power of individual disks
is not economically viable due to stringent constraints
on the power budget and the cost of individual disks. The
low market share for such high-cost, high-performance,
and high-functionality disks would further increase their
cost.
The following alternatives are considered (Hsu et
al., 2000): (a) no offloading, (b) offloading operations
on single tables, and (c) offloading operations on mul-
tiple tables. The third approach, which makes the most
sense, is implemented in SmartSTOR, which is the
controller of multiple disks and can coordinate the
joining of relational tables residing on the disks.
The host, which runs application programs, can offload
processing to SmartSTORs, which then deliver results A
back to the host. Experimental results with the TPC-D
benchmark are presented (see http://www.tpc.org).
FUTURE TRENDS
The offloading of activities from the host to peripheral
devices has been carried out successfully in the past.
SmartSTOR is an intermediate step, and active disk is a
more distant possibility, which would require standard-
ization activities such as object-based storage devices
(OSD) (http://www.snia.org).
CONCLUSION
The largest benefit stemming from active disks comes
from the parallel processing capability provided by a
large number of disks, that the aggregate processing
power of disk controllers may exceed the computing
power of servers.
The filtering effect is another benefit of active disks.
Disk transfer rates are increasing rapidly, so that by
eliminating unnecessary I/O transfers, more disks can
be placed on I/O buses or storage area networks
(SANs).
ACKNOWLEDGMENT
Supported by NSF through Grant 0105485 in Computer
Systems Architecture.
REFERENCES
Acharya, A., Uysal, M., & Saltz, J. H. (1998). Active
disks: Programming model, algorithms, and evaluation.
Proceedings of the Eighth International Conference
on Architectural Support for Programming Languages
and Operating Systems (pp. 81-91), USA.
Agarwal, S., Agrawal, R., Deshpande, P., Gupta, A.,
Naughton, J., Ramakrishnan, R. et al. (1996). On the
computation of multidimensional aggregates. Proceed-
ings of the 22nd International Conference on Very
Large Data Bases (pp. 506-521), India.
Dunham, M. H. (2003). Data mining: Introductory and
advanced topics. Prentice-Hall.
TEAM LinG

Hennessey, J. L., & Patterson, D. A. (2003). Computer
architecture: A quantitative approach (3rd ed.). Morgan
Kaufmann.
Hsu, W. W., Smith, A. J., & Young, H. (2000). Projecting
the performance of decision support workloads with smart
storage (SmartSTOR). Proceedings of the Seventh Inter-
national Conference on Parallel and Distributed Sys-
tems (pp. 417-425), Japan.
Keeton, K., Patterson, D. A., & Hellerstein, J. M. (1998).
A case for intelligent disks (IDISKs). ACM SIGMOD
Record, 27(3), 42-52.
Lumb, C., Schindler, J., Ganger, G. R., Nagle, D. F., &
Riedel, E. (2000). Towards higher disk head utilization:
Extracting free bandwidth from busy disk drives. Pro-
ceedings of the Fourth Symposium on Operating Sys-
tems Design and Implementation (pp. 87-102), USA.
Patterson, D. A., & Keeton, K. (1998). Hardware technol-
ogy trends and database opportunities. Keynote address
of the ACM SIGMOD Conference on Management of
Data. Retrieved from http://www.cs.berkeley.edu/~pattr
sn/talks.html
Ramakrishnan, R., & Gehrke, J. (2003). Database manage-
ment systems (3rd ed.). McGraw-Hill.
Riedel, E. (1999). Active disk Remote execution for
network attached storage (Tech. Rep. No. CMU-CS-
99-177). CMU, Department of Computer Science.
Riedel, E., Faloutsos, C., Ganger, G. R., & Nagle, D. F.
(2000). Data mining in an OLTP system (nearly) for
free. Proceedings of the ACM SIGMOD International
Conference on Management of Data (pp. 13-21), USA.
Riedel, E., Gibson, G. A., & Faloutsos, C. (1998). Active
storage for large scale data mining and multimedia appli-
cations. Proceedings of the 24th International Very
Large Data Base Conference (pp. 62-73), USA.
Su, W.-Y.S. (1983). Advanced database machine archi-
tecture. Prentice-Hall.
Zaki, M. J. (1999). Parallel and distributed associative rule
mining: A survey. IEEE Concurrency, 7(4), 14-25.
KEY TERMS
Active Disk: A disk whose controller runs applica-
tion code, which can process data on disk.
Content-Addressable File Store (CAFS): Specialized
hardware from ICL (UKs International Computers Limited)
used as a filter for database applications.
10
Active Disks for Data Mining
Database Computer or Machine: A specialized com-
puter for database applications, which usually works in
conjunction with a host computer.
Database Gap: Processing demand for database appli-
cations is twofold in nine to 12 months, but it takes 18
months for the processor speed to increase that much
according to Moores law.
Disk Access Time: Sum of seek time (ST), rotational
latency (RL), and transfer time (TT). ST is the time to move
the read/write heads (attached to the disk arm) to the
appropriate concentric track on the disk. There is also a
head selection time to select the head on the appropriate
track on the disk. RL for small block transfers is half of the
disk rotation time. TT is the ratio of the block size and the
average disk transfer rate.
Freeblock Scheduling: A disk arm scheduling method
that uses opportunistic accesses to disk blocks required
for a low-priority activity.
Processor per Track/Head/Disk: The last organiza-
tion corresponds to active disks.
Shared Everything/Nothing/Disks System: The main
memory and disks are shared by the (multiple) processors
in the first case, nothing is shared in the second case (i.e.,
standalone computers connected via an interconnection
network), and disks are shared by processor memory
combinations in the third case.
SmartSTOR: A scheme where the disk array control-
ler for multiple disks assists the host in processing data-
base applications.
Table Scan: The sequential reading of all the blocks
of a relational table to select a subset of its attributes
based on a selection argument, which is either not
indexed (called a sargable argument) or the index is not
clustered.
Transaction Processing Council: This council
has published numerous benchmarks for transaction
processing (TPC-C), decision support (TPC-H and TPC-
R), transactional Web benchmark, also supporting brows-
ing (TPC-W).
ENDNOTES
1
There have been several concurrent activities in
the area of Active Disks: (a) Riedel, Gibson, and
Faloutsos at Carnegie-Mellon University (1998)
and Riedel (1999); (b) Acharya, Uysal, and Saltz
(1998) at the University of California at Santa
TEAM LinG
Active Disks for Data Mining
Barbara (UCSB) and University of Maryland; (c)
intelligent disks by Keeton, Patterson, and
Hellerstein (1998) at the University of California
(UC) at Berkeley; and (d) the SmartSTOR project at
UC Berkeley and the IBM Almaden Research Center
by Hsu, Smith, and Young (2000).
2
The SQL SELECT and GROUP BY statements are
easy to implement. The datacube operator com-
putes GROUP BYs for all possible combinations
of a list of attributes (Dunham, 2003). The
PipeHash algorithm represents the datacube as a
lattice of related GROUP BYs. A directed edge A
connects a GROUP BY i to a GROUP BY j, if j can
be generated by i and has one less attribute
(Agarwal, Agrawal, Deshpande, Gupta, Naughton,
Ramakrishnan, et al., 1996). Three other applica-
tions dealing with an external sort, image convolu-
tion, and generating composite satellite images
are beyond the scope of this discussion.
11
TEAM LinG
12
Active Learning with Multiple Views
Ion Muslea
SRI International, USA
INTRODUCTION
Inductive learning algorithms typically use a set of
labeled examples to learn class descriptions for a set of
user-specified concepts of interest. In practice, label-
ing the training examples is a tedious, time consuming,
error-prone process. Furthermore, in some applica-
tions, the labeling of each example also may be ex-
tremely expensive (e.g., it may require running costly
laboratory tests). In order to reduce the number of
labeled examples that are required for learning the
concepts of interest, researchers proposed a variety of
methods, such as active learning, semi-supervised learn-
ing, and meta-learning.
This article presents recent advances in reducing the
need for labeled data in multi-view learning tasks; that
is, in domains in which there are several disjoint subsets
of features (views), each of which is sufficient to learn
the target concepts. For instance, as described in Blum
and Mitchell (1998), one can classify segments of
televised broadcast based either on the video or on the
audio information; or one can classify Web pages based
on the words that appear either in the pages or in the
hyperlinks pointing to them. In summary, this article
focuses on using multiple views for active learning and
improving multi-view active learners by using semi-
supervised- and meta-learning.
BACKGROUND
Active, Semi-Supervised, and
Multi-view Learning
Most of the research on multi-view learning focuses on
semi-supervised learning techniques (Collins & Singer,
1999, Pierce & Cardie, 2001) (i.e., learning concepts
from a few labeled and many unlabeled examples). By
themselves, the unlabeled examples do not provide any
direct information about the concepts to be learned. How-
ever, as shown by Nigam, et al. (2000) and Raskutti, et al.
(2002), their distribution can be used to boost the accuracy
of a classifier learned from the few labeled examples.
Intuitively, semi-supervised, multi-view algorithms
proceed as follows: first, they use the small labeled
training set to learn one classifier in each view; then,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
they bootstrap the views from each other by augmenting
the training set with unlabeled examples on which the
other views make high-confidence predictions. Such
algorithms improve the classifiers learned from labeled
data by also exploiting the implicit information pro-
vided by the distribution of the unlabeled examples.
In contrast to semi-supervised learning, active learn-
ers (Tong & Koller, 2001) typically detect and ask the
user to label only the most informative examples in the
domain, thus reducing the users data-labeling burden.
Note that active and semi-supervised learners take dif-
ferent approaches to reducing the need for labeled data;
the former explicitly search for a minimal set of labeled
examples from which to perfectly learn the target con-
cept, while the latter aim to improve a classifier learned
from a (small) set of labeled examples by exploiting
some additional unlabeled data.
In keeping with the active learning approach, this
article focuses on minimizing the amount of labeled
data without sacrificing the accuracy of the learned
classifiers. We begin by analyzing co-testing (Muslea,
2002), which is a novel approach to active learning. Co-
testing is a multi-view active learner that maximizes the
benefits of labeled training data by providing a prin-
cipled way to detect the most informative examples in a
domain, thus allowing the user to label only these.
Then, we discuss two extensions of co-testing that
cope with its main limitationsthe inability to exploit
the unlabeled examples that were not queried and the
lack of a criterion for deciding whether a task is appro-
priate for multi-view learning. To address the former,
we present Co-EMT (Muslea et al., 2002a), which inter-
leaves co-testing with a semi-supervised, multi-view
learner. This hybrid algorithm combines the benefits of
active and semi-supervised learning by detecting the
most informative examples, while also exploiting the
remaining unlabeled examples. Second, we discuss Adap-
tive View Validation (Muslea et al., 2002b), which is a
meta-learner that uses the experience acquired while
solving past learning tasks to predict whether multi-
view learning is appropriate for a new, unseen task.
A Motivating Problem: Wrapper
Induction
Information agents such as Ariadne (Knoblock et al.,
2001) integrate data from pre-specified sets of Web
TEAM LinG
Active Learning with Multiple Views
sites so that they can be accessed and combined via
database-like queries. For example, consider the agent in
Figure 1, which answers queries such as the following:
Show me the locations of all Thai restaurants in L.A.
that are A-rated by the L.A. County Health Department.
To answer this query, the agent must combine data
from several Web sources:
from Zagats, it obtains the name and address of all
Thai restaurants in L.A.;
from the L.A. County Web site, it gets the health
rating of any restaurant of interest;
from the Geocoder, it obtains the latitude/longi-
tude of any physical address;
from Tiger Map, it obtains the plot of any location,
given its latitude and longitude.
Information agents typically rely on wrappers to
extract the useful information from the relevant Web
pages. Each wrapper consists of a set of extraction rules
and the code required to apply them. As manually writing
the extraction rules is a time-consuming task that re-
quires a high level of expertise, researchers designed
wrapper induction algorithms that learn the rules from
user-provided examples (Muslea et al., 2001).
In practice, information agents use hundreds of ex-
traction rules that have to be updated whenever the
format of the Web sites changes. As manually labeling
examples for each rule is a tedious, error-prone task,
one must learn high accuracy rules from just a few
labeled examples. Note that both the small training sets
and the high accuracy rules are crucial to the successful
deployment of an agent. The former minimizes the
amount of work required to create the agent, thus mak-
ing the task manageable. The latter is required in order
to ensure the quality of the agents answer to each query:
when the data from multiple sources is integrated, the
errors of the corresponding extraction rules get com-
pounded, thus affecting the quality of the final result;
for instance, if only 90% of the Thai restaurants and
90% of their health ratings are extracted correctly, the
result contains only 81% (90% x 90% = 81%) of the A-
rated Thai restaurants.
We use wrapper induction as the motivating problem
for this article because, despite the practical impor-
tance of learning accurate wrappers from just a few
labeled examples, there has been little work on active
learning for this task. Furthermore, as explained in
Muslea (2002), existing general-purpose active learn-
ers cannot be applied in a straightforward manner to
wrapper induction.
Figure 1. An information agent that combines data
from the Zagats restaurant guide, the L.A. County A
Health Department, the ETAK Geocoder, and the Tiger
Map service
Restaurant Guide
L.A. County Query:
Health Dept. A-rated Thai
restaurants
in L.A.
Agent
RESULTS:
Geocoder
Tiger Map Server
MAIN THRUST
In the context of wrapper induction, we intuitively de-
scribe three novel algorithms: Co-Testing, Co-EMT,
and Adaptive View Validation. Note that these algo-
rithms are not specific to wrapper induction, and they
have been applied to a variety of domains, such as text
classification, advertisement removal, and discourse
tree parsing (Muslea, 2002).
Co-Testing: Multi-View Active Learning
Co-Testing (Muslea, 2002, Muslea et al., 2000), which
is the first multi-view approach to active learning, works
as follows:
first, it uses a small set of labeled examples to
learn one classifier in each view;
then, it applies the learned classifiers to all unla-
beled examples and asks the user to label one of
the examples on which the views predict different
labels;
it adds the newly labeled example to the training
set and repeats the whole process.
Intuitively, Co-Testing relies on the following ob-
servation: if the classifiers learned in each view predict
a different label for an unlabeled example, at least one
of them makes a mistake on that prediction. By asking
the user to label such an example, Co-Testing is guaran-
teed to provide useful information for the view that
made the mistake.
13
TEAM LinG

To illustrate Co-Testing for wrapper induction, con-
sider the task of extracting restaurant phone numbers from
documents similar to the one shown in Figure 2. To extract
this information, the wrapper must detect both the begin-
ning and the end of the phone number. For instance, to find
where the phone number begins, one can use the following
rule:
R1 = SkipTo( Phone:<i> )
This rule is applied forward, from the beginning of
the page, and it ignores everything until it finds the string
Phone:<i>. Note that this is not the only way to detect
where the phone number begins. An alternative way to
perform this task is to use the following rule:
R2 = BackTo( Cuisine ) BackTo( ( Number ) )
which is applied backward, from the end of the document.
R2 ignores everything until it finds Cuisine and then,
again, skips to the first number between parentheses.
Note that R1 and R2 represent descriptions of the
same concept (i.e., beginning of phone number) that are
learned in two different views (see Muslea et al. [2001]
for details on learning forward and backward rules). That
is, views V1 and V2 consist of the sequences of charac-
ters that precede and follow the beginning of the item,
respectively. View V1 is called the forward view, while
V2 is the backward view. Based on V1 and V2, Co-Testing
can be applied in a straightforward manner to wrapper
induction. As shown in Muslea (2002), Co-Testing clearly
outperforms existing state-of-the-art algorithms, both
on wrapper induction and a variety of other real world
domains.
Co-EMT: Interleaving Active and
Semi-Supervised Learning
To further reduce the need for labeled data, Co-EMT
(Muslea et al., 2002a) combines active and semi-super-
vised learning by interleaving Co-Testing with Co-EM
(Nigam & Ghani, 2000). Co-EM, which is a semi-super-
vised, multi-view learner, can be seen as the following
Figure 2. The forward rule R1 and the backward rule
R2 detect the beginning of the phone number. Forward
and backward rules have the same semantics and differ
only in terms of from where they are applied (start/end
of the document) and in which direction
R1: SkipTo( Phone : <i> ) R2: BackTo(Cuisine) BackTo( (Number) )
Name: <i>Ginos </i> <p>Phone :<i> (800)111-1717 </i> <p> Cuisine :
14
Active Learning with Multiple Views
iterative, two-step process: first, it uses the hypotheses
learned in each view to probabilistically label all the
unlabeled examples; then it learns a new hypothesis in
each view by training on the probabilistically labeled
examples provided by the other view.
By interleaving active and semi-supervised learn-
ing, Co-EMT creates a powerful synergy. On one hand,
Co-Testing boosts Co-EMs performance by providing
it with highly informative labeled examples (instead of
random ones). On the other hand, Co-EM provides Co-
Testing with more accurate classifiers (learned from
both labeled and unlabeled data), thus allowing Co-
Testing to make more informative queries.
Co-EMT was not yet applied to wrapper induction,
because the existing algorithms are not probabilistic
learners; however, an algorithm similar to Co-EMT was
applied to information extraction from free text (Jones
et al., 2003). To illustrate how Co-EMT works, we
describe now the generic algorithm Co-EMTWI, which
combines Co-Testing with the semi-supervised wrap-
per induction algorithm described next.
In order to perform semi-supervised wrapper in-
duction, one can exploit a third view, which is used to
evaluate the confidence of each extraction. This new
content-based view (Muslea et al., 2003) describes the
actual item to be extracted. For example, in the phone
numbers extraction task, one can use the labeled ex-
amples to learn a simple grammar that describes the
field content: (Number) Number Number. Similarly,
when extracting URLs, one can learn that a typical URL
starts with the string http://www., ends with the string
.html, and contains no HTML tags.
Based on the forward, backward, and content-based
views, one can implement the following semi-super-
vised wrapper induction algorithm. First, the small set
of labeled examples is used to learn a hypothesis in
each view. Then, the forward and backward views feed
each other with unlabeled examples on which they
make high-confidence extractions (i.e., strings that are
extracted by either the forward or the backward rule and
are also compliant with the grammar learned in the
third, content-based view).
Given the previous Co-Testing and the semi-super-
vised learner, Co-EMTWI combines them as follows. First,
the sets of labeled and unlabeled examples are used for
semi-supervised learning. Second, the extraction rules
that are learned in the previous step are used for Co-
Testing. After making a query, the newly labeled example
is added to the training set, and the whole process is
repeated for a number of iterations. The empirical study in
Muslea, et al., (2002a) shows that, for a large variety of
text classification tasks, Co-EMT outperforms both Co-
Testing and the three state-of-the-art semi-supervised
learners considered in that comparison.
TEAM LinG
Active Learning with Multiple Views
View Validation: Are the Views
Adequate for Multi-View Learning?
The problem of view validation is defined as follows:
given a new unseen multi-view learning task, how does a
user choose between solving it with a multi- or a single-
view algorithm? In other words, how does one know
whether multi-view learning will outperform pooling all
features together and applying a single-view learner?
Note that this question must be answered while having
access to just a few labeled and many unlabeled ex-
amples: applying both the single- and multi-view active
learners and comparing their relative performances is a
self-defeating strategy, because it doubles the amount
of required labeled data (one must label the queries
made by both algorithms).
The need for view validation is motivated by the
following observation: while applying Co-Testing to
dozens of extraction tasks, Muslea et al. (2002b) no-
ticed that the forward and backward views are appropri-
ate for most, but not all, of these learning tasks. This
view adequacy issue is related tightly to the best extrac-
tion accuracy reachable in each view. Consider, for
example, an extraction task in which the forward and
backward rules lead to a high- and low-accuracy rule,
respectively. Note that Co-Testing is not appropriate
for solving such tasks; by definition, multi-view learn-
ing applies only to tasks in which each view is sufficient
for learning the target concept (obviously, the low-
accuracy view is insufficient for accurate extraction).
To cope with this problem, one can use Adaptive
View Validation (Muslea et al., 2002b), which is a meta-
learner that uses the experience acquired while solving
past learning tasks to predict whether the views of a new
unseen task are adequate for multi-view learning. The
view validation algorithm takes as input several solved
extraction tasks that are labeled by the user as having
views that are adequate or inadequate for multi-view
learning. Then, it uses these solved extraction tasks to
learn a classifier that, for new unseen tasks, predicts
whether the views are adequate for multi-view learning.
The (meta-) features used for view validation are
properties of the hypotheses that, for each solved task,
are learned in each view (i.e., the percentage of unla-
beled examples on which the rules extract the same
string, the difference in the complexity of the forward
and backward rules, the difference in the errors made on
the training set, etc.). For both wrapper induction and
text classification, Adaptive View Validation makes
accurate predictions based on a modest amount of train-
ing data (Muslea et al., 2002b).
FUTURE TRENDS
A
There are several major areas of future work in the field
of multi-view learning. First, there is a need for a view
detection algorithm that automatically partitions a
domains features in views that are adequate for multi-
view learning. Such an algorithm would remove the last
stumbling block against the wide applicability of multi-
view learning (i.e., the requirement that the user pro-
vides the views to be used). Second, in order to reduce
the computational costs of active learning (re-training
after each query is CPU-intensive), one must consider
look-ahead strategies that detect and propose (near)
optimal sets of queries. Finally, Adaptive View Valida-
tion has the limitation that it must be trained separately
for each application domain (e.g., once for wrapper
induction, once for text classification, etc.). A major
improvement would be a domain-independent view
validation algorithm that, once trained on a mixture of
tasks from various domains, can be applied to any new
learning task, independently of its application domain.
CONCLUSION
In this article, we focus on three recent developments
that, in the context of multi-view learning, reduce the
need for labeled training data.
Co-Testing: A general-purpose, multi-view ac-
tive learner that outperforms existing approaches
on a variety of real-world domains.
Co-EMT: A multi-view learner that obtains a ro-
bust behavior over a wide spectrum of learning
tasks by interleaving active and semi-supervised
multi-view learning.
Adaptive View Validation: A meta-learner that
uses past experiences to predict whether multi-
view learning is appropriate for a new unseen
learning task.
REFERENCES
Blum, A., & Mitchell, T. (1998). Combining labeled and
unlabeled data with co-training. Proceedings of the
Conference on Computational Learning Theory
(COLT-1998).
Collins, M., & Singer, Y. (1999). Unsupervised models
for named entity classification. Empirical Methods in
15
TEAM LinG

Natural Language Processing & Very Large Corpora
(pp. 100-110).
Jones, R., Ghani, R., Mitchell, T., & Riloff, E. (2003).
Active learning for information extraction with mul-
tiple view feature sets. Proceedings of the ECML-2003
Workshop on Adaptive Text Extraction and Mining.
Knoblock, C. et al. (2001). The Ariadne approach to Web-
based information integration. International Journal of
Cooperative Information Sources, 10, 145-169.
Muslea, I. (2002). Active learning with multiple views
[doctoral thesis]. Los Angeles: Department of Com-
puter Science, University of Southern California.
Muslea, I., Minton, S., & Knoblock, C. (2000). Selective
sampling with redundant views. Proceedings of the Na-
tional Conference on Artificial Intelligence (AAAI-2000).
Muslea, I., Minton, S., & Knoblock, C. (2001). Hierar-
chical wrapper induction for semi-structured sources.
Journal of Autonomous Agents & Multi-Agent Sys-
tems, 4, 93-114.
Muslea, I., Minton, S., & Knoblock, C. (2002a). Active
+ semi-supervised learning = robust multi-view learn-
ing. Proceedings of the International Conference on
Machine Learning (ICML-2002).
Muslea, I., Minton, S., & Knoblock, C. (2002b). Adap-
tive view validation: A first step towards automatic view
detection. Proceedings of the International Confer-
ence on Machine Learning (ICML-2002).
Muslea, I., Minton, S., & Knoblock, C. (2003). Active
learning with strong and weak views: A case study on
wrapper induction. Proceedings of the International
Joint Conference on Artificial Intelligence (IJCAI-2003).
Nigam, K., & Ghani, R. (2000). Analyzing the effective-
ness and applicability of co-training. Proceedings of
the Conference on Information and Knowledge Man-
agement (CIKM-2000).
16
Active Learning with Multiple Views
Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. (2000).
Text classification from labeled and unlabeled docu-
ments using EM. Machine Learning, 39(2-3), 103-134.
Pierce, D., & Cardie, C. (2001). Limitations of co-training
for natural language learning from large datasets. Empiri-
cal Methods in Natural Language Processing, 1-10.
Raskutti, B., Ferra, H., & Kowalczyk, A. (2002). Using
unlabeled data for text classification through addition
of cluster parameters. Proceedings of the Interna-
tional Conference on Machine Learning (ICML-2002).
Tong, S., & Koller, D. (2001). Support vector machine
active learning with applications to text classification.
Journal of Machine Learning Research, 2, 45-66.
KEY TERMS
Active Learning: Detecting and asking the user to
label only the most informative examples in the domain
(rather than randomly-chosen examples).
Inductive Learning: Acquiring concept descrip-
tions from labeled examples.
Meta-Learning: Learning to predict the most ap-
propriate algorithm for a particular task.
Multi-View Learning: Explicitly exploiting sev-
eral disjoint sets of features, each of which is sufficient
to learn the target concept.
Semi-Supervised Learning: Learning from both
labeled and unlabeled data.
View Validation: Deciding whether a set of views
is appropriate for multi-view learning.
Wrapper Induction: Learning (highly accurate)
rules that extract data from a collection of documents
that share a similar underlying structure.
TEAM LinG

Administering and Managing a Data
Warehouse
James E. Yao
Montclair State University, USA
Chang Liu
Northern Illinois University, USA
Qiyang Chen
Montclair State University, USA
June Lu
University of Houston-Victoria, USA
INTRODUCTION
As internal and external demands on information from
managers are increasing rapidly, especially the infor-
mation that is processed to serve managers specific
needs, regular databases and decision support systems
(DSS) cannot provide the information needed. Data ware-
houses came into existence to meet these needs, consoli-
dating and integrating information from many internal and
external sources and arranging it in a meaningful format
for making accurate business decisions (Martin, 1997). In
the past five years, there has been a significant growth in
data warehousing (Hoffer, Prescott, & McFadden, 2005).
Correspondingly, this occurrence has brought up the
issue of data warehouse administration and management.
Data warehousing has been increasingly recognized as an
effective tool for organizations to transform data into
useful information for strategic decision-making. To
achieve competitive advantages via data warehousing,
data warehouse management is crucial (Ma, Chou, & Yen,
2000).
BACKGROUND
Since the advent of computer storage technology and
higher level programming languages (Inmon, 2002),
organizations, especially larger organizations, have put
enormous amount of investment in their information
system infrastructures. In a 2003 IT spending survey,
45% of American company participants indicated that
their 2003 IT purchasing budgets had increased com-
pared with their budgets in 2002. Among the respon-
dents, database applications ranked top in areas of tech-
nology being implemented or had been implemented,
with 42% indicating a recent implementation (Informa-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
17
A
tion, 2004). The fast growth of databases enables compa-
nies to capture and store a great deal of business opera-
tion data and other business-related data. The data that are
stored in the databases, either historical or operational,
have been considered corporate resources and an asset
that must be managed and used effectively to serve the
corporate business for competitive advantages.
A database is a computer structure that houses a self-
describing collection of related data (Kroenke, 2004;
Rob & Coronel, 2004). This type of data is primitive,
detailed, and used for day-to-day operation. The data in
a warehouse is derived, meaning it is integrated, sub-
ject-oriented, time-variant, and nonvolatile (Inmon,
2002). A data warehouse is defined as an integrated
decision support database whose content is derived
from various operational databases (Hoffer, Prescott, &
McFadden, 2005; Sen & Jacob, 1998). Often a data ware-
house can be referred to as a multidimensional database
because each occurrence of the subject is referenced by
an occurrence of each of several dimensions or character-
istics of the subject (Gillenson, 2005). Some multidimen-
sional databases operate on a technological foundation
optimal for slicing and dicing the data, where data can
be thought of as existing in multidimensional cubes (Inmon,
2002). Regular databases load data in two-dimensional
tables. A data warehouse can use OLAP (online analytical
processing) to provide users with multidimensional views
of their data, which can be visually represented as a cube
for three dimensions (Senn, 2004).
With the host of differences between a database for
day-to-day operation and a data warehouse for support-
ing management decision-making process, the adminis-
tration and management of a data warehouse is of course
far from similar. For instance, a data warehouse team
requires someone who does routine data extraction,
transformation, and loading (ETL) from operational data-
TEAM LinG

bases into data warehouse databases. Thus the team
requires a technical role called ETL Specialist. On the
other hand, a data warehouse is intended to support the
business decision-making process. Someone like a busi-
ness analyst is also needed to ensure that business
information requirements are crossed to the data ware-
house development. Data in the data warehouse can be
very sensitive and cross functional areas, such as per-
sonal medical records and salary information. There-
fore, a higher level of security on the data is needed.
Encrypting the sensitive data in data warehouse is a
potential solution. Issues as such in data warehouse
administration and management need to be defined and
discussed.
MAIN THRUST
Data warehouse administration and management covers
a wide range of fields. This article focuses only on data
warehouse and business strategy, data warehouse devel-
opment life cycle, data warehouse team, process man-
agement, and security management to present the cur-
rent concerns and issues in data warehouse administra-
tion and management.
Data Warehouse and Business Strategy
Data is the blood of an organization. Without data, the
corporation has no idea where it stands and where it will
go (Ferdinandi, 1999, p. xi). With data warehousing,
todays corporations can collect and house large vol-
umes of data. Does the size of data volume simply
guarantee you a success in your business? Does it mean
that the more data you have the more strategic advan-
tages you have over your competitors? Not necessarily.
There is no predetermined formula that can turn your
information into competitive advantages (Inmon,
Terdeman, & Imhoff, 2000). Thus, top management and
data administration team are confronted with the ques-
tion of how to convert corporate information into com-
petitive advantages.
A well-managed data warehouse can assist a corpora-
tion in its strategy to gain competitive advantages. This
can be achieved by using an exploration warehouse,
which is a direct product of data warehouse, to identify
environmental factors, formulate strategic plans, and
determine business specific objectives:
Identifying Environmental Factors: Quantified
analysis can be used for identifying a corporations
products and services, market share of specific
products and services, financial management.
18
Administering and Managing a Data Warehouse
Formulating Strategic Plans: Environmental fac-
tors can be matched up against the strategic plan by
identifying current market positioning, financial
goals, and opportunities.
Determining Specific Objectives: Exploration ware-
house can be used to find patterns; if found, these
patterns are then compared with patterns discov-
ered previously to optimize corporate objectives
(Inmon, Terdeman, & Imhoff, 2000).
While managing a data warehouse for business strat-
egy, what needs to be taken into consideration is the
difference between companies. No one formula fits
every organization. Avoid using so called templates
from other companies. The data warehouse is used for
your companys competitive advantages. You need to
follow your companys user information requirements
for strategic advantages.
Data Warehouse Development Cycle
Data warehouse system development phases are similar
to the phases in the systems development life cycle
(SDLC) (Adelman & Rehm, 2003). However, Barker
(1998) thinks that there are some differences between
the two due to the unique functional and operational
features of a data warehouse. As business and informa-
tion requirements change, new corporate information
models evolve and are synthesized into the data ware-
house in the Synthesis of Model phase. These models
are then used to exploit the data warehouse in the
Exploit phase. The data warehouse is updated with new
data using appropriate updating strategies and linked to
various data sources.
Inmon (2002) sees system development for data
warehouse environment as almost exactly the opposite
of the traditional SDLC. He thinks that traditional SDLC
is concerned with and supports primarily the opera-
tional environment. The data warehouse operates under
a very different life cycle called CLDS (the reverse of
the SDLC). The CLDS is a classic data-driven develop-
ment life cycle, but the SDLC is a classic requirements-
driven development life cycle.
The Data Warehouse Team
Building a data warehouse is a large system develop-
ment process. Participants of data warehouse develop-
ment can range from a data warehouse administrator
(DWA) (Hoffer, Prescott, & McFadden, 2005) to a
business analyst (Ferdinandi, 1999). The data ware-
house team is supposed to lead the organization into
assuming their roles and thereby bringing about a part-
TEAM LinG
Administering and Managing a Data Warehouse
nership with the business (McKnight, 2000). A data ware-
house team may have the following roles (Barker, 1998;
Ferdinandi, 1999; Inmon, 2000, 2003; McKnight, 2000):
Data Warehouse Administrator (DWA): respon-
sible for integrating and coordinating of metadata
and data across many different data sources as well
as data source management, physical database de-
sign, operation, backup and recovery, security, and
performance and tuning.
Manager/Director: responsible for the overall
management of the entire team to ensure that the
team follows the guiding principles, business re-
quirements, and corporate strategic plans.
Project Manager: responsible for data warehouse
project development, including matching each team
members skills and aspirations to tasks on the
project plan.
Executive Sponsor: responsible for garnering and
retaining adequate resources for the construction
and maintenance of the data warehouse.
Business Analyst: responsible for determining
what information is required from a data warehouse
to manage the business competitively.
System Architect: responsible for developing and
implementing the overall technical architecture of
the data warehouse, from the backend hardware and
software to the client desktop configurations.
ETL Specialist: responsible for routine work on
data extraction, transformation, and loading for the
warehouse databases.
Front End Developer: responsible for develop-
ing the front-end, whether it is client-server or
over the Web.
OLAP Specialist: responsible for the develop-
ment of data cubes, a multidimensional view of data
in OLAP.
Data Modeler: responsible for modeling the ex-
isting data in an organization into a schema that is
appropriate for OLAP analysis.
Trainer: responsible for training the end-users to
use the system so that they can benefit from the
data warehouse system.
End User: responsible for providing feedback to
the data warehouse team.
In terms of the size of the data warehouse administra-
tor team, Inmon (2003) has several recommendations:
large warehouse requires more analysts;
every 100gbs of data in a data warehouse requires
another data warehouse administrator;
a new data warehouse administrator is required for
each year a data warehouse is up and running and A
is being used successfully;
if an ETL tool is being written manually, many data
warehouse administrators are needed; if automa-
tion tool is needed much fewer staffing is required;
automated data warehouse database management
system (DBMS) requires fewer data warehouse
administrators, otherwise more administrators
are needed;
fewer supporting staff is required if the corporate
information factory (CIF) architecture is fol-
lowed more closely; reversely, more staff is
needed.
McKnight (2000) suggests that all the technical
roles be performed full-time by dedicated personnel
and each responsible person receives specific data
warehouse training.
Data warehousing is growing rapidly. As the scope
and data storage size of the data warehouse change, the
roles and size of a data warehouse team should be
adjusted accordingly. In general, the extremes should
be avoided. Without sufficient professionals, job may
not be done satisfactorily. On the other hand, too many
people will certainly get the team overstuffed.
Process Management
Developing data warehouse has become a popular but
exceedingly demanding and costly activity in informa-
tion systems development and management. Data ware-
house vendors are competing intensively for their cus-
tomers because so much of their money and prestige
are at stake. Consulting vendors have redirected their
attention toward this rapidly expanding market seg-
ment. User companies are facing with a serious ques-
tion on which product they should buy. Sen & Jacobs
(1998) advice is to first understand the process of data
warehouse development before selecting the tools for
its implementation. A data warehouse development
process refers to the activities required to build a data
warehouse (Barquin, 1997). Sen & Jacob (1998) and
Ma, Chou, & Yen (2000) have identified some of these
activities, which need to be managed during the data
warehouse development cycle: initializing project, es-
tablishing the technical environment, tool integration,
determining scalability, developing an enterprise in-
formation architecture, designing the data warehouse
database, data extraction/transformation, managing
metadata, developing the end-user interface, managing
the production environment, managing decision sup-
port tools and applications, and developing warehouse
roll-out.
19
TEAM LinG

As mentioned before, data warehouse development is
a large system development process. Process manage-
ment is not required in every step of the development
processes. Devlin (1997) states that process management
is required in the following areas: process schedule,
which consists of a network of tasks and decision points;
process map definition, which defines and maintains the
network of tasks and decision points that make up a
process; task initiation, which supports to initiate tasks
on all of the hardware/software platforms in the entire data
warehouse environment; status information enquiry,
which enquires about the status of components that are
running on all platforms.
Security Management
In recent years, information technology (IT) security
has become one of the hottest and most important topics
facing both users and providers (Senn, 2005). The goal
of database security is the protection of data from
accidental or intentional threats to its integrity and
access (Hoffer, Prescott, & McFadden, 2005). The
same is true for a data warehouse. However, higher
security methods, in addition to the common practices
such as view-based control, integrity control, process-
ing rights, and DBMS security, need to be used for the
data warehouse due to the differences between a data-
base and data warehouse. One of the differences that
demand a higher level of security for a data warehouse
is the scope of and detail level of data in the data
warehouse, such as financial transactions, personal
medical records, and salary information. A method that
can be used to protect data that requires high level of
security in a data warehouse is by using encryption and
decryption.
Confidential and sensitive data can be stored in a
separate set of tables where only authorized users can
have access. These data can be encrypted while they are
being written into the data warehouse. In this way, the
data captured and stored in the data warehouse are
secure and can only be accessed on an authorized basis.
Three levels of security can be offered by using encryp-
tion and decryption. The first level is that only authorized
users can have access to the data in the data warehouse.
Each group of users, internal or external, ranging from
executives to information consumers should be granted
different rights for security reasons. Unauthorized users
are totally prevented from seeing the data in the data
warehouse. The second level is the protection from unau-
thorized dumping and interpretation of data. Without the
right key an unauthorized access will not be allowed to
write anything into the tables. On the other hand, the
existing data in the tables cannot be decrypted. The third
level is the protection from unauthorized access during the
20
Administering and Managing a Data Warehouse
transmission process. Even if unauthorized access occurs
during transmission, there is no harm to the encrypted data
unless the user has the decryption code (Ma, Chou, & Yen,
2000).
FUTURE TRENDS
Data warehousing administration and management is
facing several challenges, as data warehousing becomes
a mature part of the infrastructure of organizations.
More legislative work is necessary to protect individual
privacy from abuse by government or commercial enti-
ties that have large volumes of data concerning those
individuals. The protection also calls for tightened se-
curity through technology as well as user efforts for
workable rules and regulations while at the same time
still granting a data warehouse the ability to perform
large datasets for meaningful analyses (Marakas, 2003).
Todays data warehouse is limited to storage of
structured data in the form of records, fields, and data-
bases. Unstructured data, such as multimedia, maps,
graphs, pictures, sound, and video files are demanded
increasingly in organizations. How to manage the stor-
age and retrieval of unstructured data and how to search
for specific data items set a real challenge for data
warehouse administration and management. Alternative
storage, especially the near-line storage, which is one
of the two forms of alternative storage, is considered to
be one of the best future solutions for managing the
storage and retrieval of unstructured data in data ware-
houses (Marakas, 2003).
The past decade has seen a fast rise of the Internet
and World Wide Web. Today, Web-enabled versions of
all leading vendors warehouse tools are becoming avail-
able (Moeller, 2001). This recent growth in Web use
and advances in e-business applications have pushed the
data warehouse from the back office, where it is ac-
cessed by only a few business analysts, to the front lines
of the organization, where all employees and every
customer can use it.
To accommodate this move to the frontline of the
organization, the data warehouse demands massive
scalability for data volume as well as for performance.
As the number of and types of users increase rapidly,
enterprise data volume is doubling in size every 9 to 12
months. Around-the-clock access to the data warehouse
is becoming the norm. The data warehouse will require
fast implementation, continuous scalability, and ease of
management (Marakas, 2003).
Additionally, building distributed warehouses, which
are normally called data marts, will be on the rise. Other
technical advances in data warehousing will include an
increasing ability to exploit parallel processing, auto-
TEAM LinG
Administering and Managing a Data Warehouse
mated information delivery, greater support of object
extensions, very large database support, and user-friendly
Web-enabled analysis applications. These capabilities
should make data warehouses of the future more powerful
and easier to use, which will further increase the impor-
tance of data warehouse technology for business strate-
gic decision making and competitive advantages (Ma,
Chou, & Yen, 2000; Marakas, 2003; Pace University, 2004).
CONCLUSION
The data that organizations have captured and stored are
considered organizational assets. Yet the data themselves
cannot do anything until they are put into intelligent use.
One way to accomplish this goal is to use data warehouse
and data mining technology to transform corporate infor-
mation into business competitive advantages.
What impacts data warehouses the most is the
Internet and Web technology. Web browser will be-
come the universal interface for corporations, allowing
employees to browse their data warehouse worldwide
on public and private networks, eliminating the need to
replicate data across diverse geographic locations. Thus
strong data warehouse management sponsorship and an
effective administration team may become a crucial
factor to provide an organization with the information
service needed.
REFERENCES
Adelman, S., & Relm, C. (2003, November 5). What are
the various phases in implementing a data warehouse
solution? DMReview. Retrieved from http://www.
dmreview.com/article_sub.cfm?articleId=7660
Barker, R. (1998, February). Managing a data warehouse.
Chertsey, UK: Veritas Software Corporation.
Barquin, F. (1997). Building, using, and managing the data
warehouse. Upper Saddle River, NJ: Prentice Hall.
Devlin, B. (1997). Data warehouse: From architecture to
implementation. Reading, MA: Addison-Wesley.
Ferdinandi, P.L. (1999). Data warehouse advice for man-
agers. New York: AMACOM American Management
Association.
Gillenson, M.L. (2005). Fundamentals of database manage-
ment systems. New York: John Wiley & Sons Inc.
Hoffer, J.A., Prescott, M.B., & McFadden, F.R. (2005).
Modern database management (7th ed.) Upper Saddle
River, NJ: Prentice Hall.
Information Technology Toolbox. (2004). 2003 IToolbox
spending survey. Retrieved from http://datawarehouse. A
ittoolbox.com/research/survey.asp
Inmon, W.H. (2002). Building the data warehouse (3rd
ed.). New York: John Wiley & Sons Inc.
Inmon, W.H. (2000). Building the data warehouse: Get-
ting started. Retrieved from http://www.billinmon.com/
library/whiteprs/earlywp/ttbuild.pdf
Inmon, W.H. (2003). Data warehouse administration.
Retrieved from http://www.billinmon.com/library/other/
dwadmin.asp
Inmon, W.H., Terdeman, R.H., & Imhoff, C. (2000). Explo-
ration warehousing. New York: John Wiley & Sons Inc.
Kroenke, D.M. (2004). Database processing: Fundamen-
tals, design, and implementation (9th ed.). Upper Saddle
River, NJ: Prentice Hall.
Ma, C., Chou, D.V., & Yen, D.C. (2000). Data warehousing,
technology assessment and management. Industrial
Management + Data Systems, 100 (3), 125-137.
Marakas, G.M. (2003). Modern data warehousing, min-
ing, and visualization: Core concepts. Upper Saddle
River, NJ: Prentice Hall.
Martin, J. (1997, September). New tools for decision
making. DM Review, 7, 80.
McKnight Associates, Inc. (2000). Effective data ware-
house organizational roles and responsibilities.
Sunnyvale, CA.
Moeller, R.A. (2001). Distributed data warehousing us-
ing web technology: How to build a more cost-effective
and flexible warehouse. New York: AMACOM American
Management Association.
Pace University. (2004). Emerging technology. Retrieved
from http://webcomposer.pace.edu/ea10931w/Tappert/
Assignment2.htm
Post, G.V. (2005). Database management systems: design-
ing & building business applications (3 rd ed.). New York:
McGraw-Hill/Irwin.
Rob, P., & Coronel, C. (2004). Database systems: Design,
implementation, and management (6th ed.). Boston, MA:
Course Technology.
Sen, A., & Jacob, V.S. (1998). Industrial strength data
warehousing: Why process is so important and so often
ignored. Communication of the ACM, 41(9), 29-31.
Senn, J.A. (2004). Information technology: Principles,
practices, opportunities (3rd ed.). Upper Saddle River, NJ:
Prentice Hall.
21
TEAM LinG

KEY TERMS
Alternative Storage: An array of storage media that
consists of two forms of storage: near-line storage and/or
second storage.
CLDS: The facetiously named system develop-
ment life cycle (SDLC) for analytical, DSS systems.
CLDS is so named because in fact it is the reverse of the
classical SDLC.
Corporate Information Factory (CIF): The cor-
porate information factory is a logical architecture with
a purpose of delivering business intelligence and busi-
ness management capabilities driven by data provided
from business operations.
Data Mart: A data warehouse that is limited in scope
and facility, but for a restricted domain.
22
Administering and Managing a Data Warehouse
Database Management System (DBMS): A set of
programs used to define, administer, and process the
database and its applications.
Metadata: Data about data; data concerning the struc-
ture of data in a database stored in the data dictionary.
Near-line Storage: Near-line storage is siloed tape
storage where siloed cartridges of tape are archived,
accessed, and managed robotically.
Online Analytical Process (OLAP): Decision
Support System (DSS) tools that uses multidimensional
data analysis techniques to provide users with multidi-
mensional views of their data.
System Development Life Cycle (SDLC): The
methodology used by most organizations for develop-
ing large information systems.
TEAM LinG

Agent-Based Mining of User Profiles for
E-Services
Pasquale De Meo
Universit Mediterranea di Reggio Calabria, Italy
Giovanni Quattrone
Universit Mediterranea di Reggio Calabria, Italy
Giorgio Terracina
Universit della Calabria, Italy
Domenico Ursino
Universit Mediterranea di Reggio Calabria, Italy
INTRODUCTION
An electronic service (e-service) can be defined as a
collection of network-resident software programs that
collaborate for supporting users in both accessing and
selecting data and services of their interest present in a
provider site. Examples of e-services are e-commerce,
e-learning, and e-government applications. E-services
are undoubtedly one of the engines presently supporting
the Internet revolution (Hull, Benedikt, Christophides
& Su, 2003). Indeed, nowadays, a large number and a
great variety of providers offer their services also or
exclusively via the Internet.
BACKGROUND
In spite of their spectacular development and present
relevance, e-services are yet to be considered a stable
technology, and various improvements could be consid-
ered for them. Many of the present suggestions for
bettering them are based on the concept of adaptivity
(i.e., the capability to make them more flexible in such
a way so as to adapt their offers and behavior to the
environment in which they are operating. In this context,
systems capable of constructing, maintaining, and ex-
ploiting profiles of users accessing e-services appear to
be capable of playing a key role in the future.
Both in the past and in the present, various e-service
providers exploit (usually rough) user profiles for pro-
posing personalized offers. However, in most cases, the
profile construction methodology they adopt presents
some problems. Indeed, it often requires a user to spend
a certain amount of time for constructing and updating
the users profile; in addition, it stores only information
about the proposals that the user claims to be interested
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
23
A
in, without considering other ones somehow related to
those just provided, possibly interesting the user in the
future and what the user did not take into account in the past.
In spite of present user profile managers, generally
when accessing an e-service, a user must personally
search the proposals of the users interest through it. As
an example, consider the bookstore section of Amazon;
whenever a customer looks for a book of interest, the
customer must carry out an autonomous personal search
of it throughout the pages of the site. We argue that, for
improving the effectiveness of e-services, it is neces-
sary to increase the interaction between the provider
and the user on the one hand and to construct a rich
profile of the user, taking into account the users de-
sires, interests, and behavior, on the other hand.
In addition, it is necessary to take into account a
further important factor. Nowadays, electronic and tele-
communications technology is rapidly evolving in such
a way to allow cell phones, palmtops, and wireless PDAs
to navigate on the Web. These mobile devices do not
have the same display or bandwidth capabilities as their
desktop counterparts; nonetheless, present e-service
providers deliver the same content to all device
typologies (Communications of the ACM, 2002).
In the past, various approaches have been proposed
for handling e-service activities; many of them are
agent-based. For example:
In Terziyan and Vitko (2002), an agent-based
framework for managing commercial transactions
between a buyer and a seller is proposed. It ex-
ploits a user profile that is handled by means of a
content-based policy.
In Garcia, Patern, and Gil (2002), a multi-agent
system called e-CoUSAL, capable of supporting
Web-shop activities, is presented. Its activity is
TEAM LinG

based on the maintenance and the exploitation of
user profiles.
In Lau, Hofstede, and Bruza (2000), WEBS, an
agent-based approach for supporting e-commerce
activities, is proposed. It exploits probabilistic
logic rules for allowing the customer preferences
for other products to be deduced.
Ardissono, et al. (2001) describe SETA, a multi-
agent system conceived for developing adaptive
Web stores. SETA uses knowledge representation
techniques to construct, maintain, and exploit user
profiles.
In Bradley and Smyth (2003), the system CASPER,
for handling recruitment services, is proposed.
Given a user, CASPER first ranks job advertise-
ments according to an applicants desires and then
recommends job proposals to the applicant on the
basis of the applicants past behavior.
In Razek, Frasson, and Kaltenbach (2002), a multi-
agent prototype for e-learning called CITS (Con-
fidence Intelligent Tutoring Agent) is proposed.
The approach of CITS aims at being adaptive and
dynamic.
In Shang, Shi, and Chen (2001), IDEAL (Intelli-
gent Distributed Environment for Active Learn-
ing), a multi-agent system for active distance learn-
ing, is proposed. In IDEAL, course materials are
decomposed into small components called
lecturelets. These are XML documents containing
JAVA code; they are dynamically assembled to
cover course topics according to learner progress.
In Zaiane (2002), an approach for exploiting Web-
mining techniques to build a software agent sup-
porting e-learning activities is presented.
All these systems construct, maintain, and exploit a
user profile; therefore, we can consider them adaptive
w.r.t. the user; however, to the best of our knowledge,
none of them is adaptive w.r.t. the device.
On the other side, in various areas of computer
science research, a large variety of approaches adapting
their behavior to the device the user is exploiting has
been proposed. As an example:
In Anderson, Domingos, and Weld (2001), a frame-
work called MINPATH, capable of simplifying the
browsing activity of a mobile user and taking into
account the device the user is exploiting, is pre-
sented.
In Macskassy, Dayanik, and Hirsh (2000), a frame-
work named i-Valets is proposed for allowing a
user to visit an information source by using differ-
ent devices.
24
Agent-Based Mining of User Profiles for E-Services
Samaras and Panayiotou (2002) present a flexible
agent-based system for providing wireless users
with a personalized access to the Internet ser-
vices.
In Araniti, De Meo, Iera, and Ursino (2003), a
novel XML-based multi-agent system for QoS
management in wireless networks is presented.
These approaches are particularly general and inter-
esting; however, to the best of our knowledge, none of
them has been conceived for handling e-services.
MAIN THRUST
Challenges to Face
In order to overcome the problems outlined previously,
some challenges must be tackled.
First, a user can access many e-services, operating in
the same or in different application contexts; a faithful
and complete profile of the user can be constructed only
by taking into account the users behavior while access-
ing all the sites. In other words, it should be possible to
construct a unique structure on the user side, storing the
users profile and, therefore, representing the users
behavior while accessing all the sites.
Second, for a given user and e-service provider, it
should be possible to compare the profile of the user
with the offers of the provider for extracting those
proposals that probably will interest the user. Existing
techniques for satisfying such a requirement are based
mainly on the exploitation of either log files or cookies.
Techniques based on log files can register only some
information about the actions carried out by the user
upon accessing an e-service; however, they cannot match
user preferences and e-service proposals. Vice versa,
techniques based on cookies are able to carry out a
certain, even if primitive, match; however, they need to
know and exploit some personal information that a user
might consider private.
Third, it should be necessary to overcome the typical
one-size-fits-all philosophy of present e-service pro-
viders by developing systems capable of adapting their
behavior to both the profile of the user and to the
characteristics of the device the user is exploiting for
accessing them (Communications of the ACM, 2002).
System Description
The system we present in this article (called e-service
adaptive manager [ESA-Manager]) aims at solving all
TEAM LinG
Agent-Based Mining of User Profiles for E-Services
three problems mentioned previously. It is an XML-
based multi-agent system for handling user accesses to
e-services, capable of adapting its behavior to both user
and device profiles.
In ESA-Manager, a service provider agent is present
for each e-service provider, handling the proposals stored
therein as well as the interaction with the user. In addi-
tion, an agent is associated with each user, adapting its
behavior to the profiles of both the user and the device
the user is exploiting for visiting the sites. Actually,
since a user can access e-service providers by means of
different devices, the users profile cannot be stored in
only one of them; as a matter of fact, it is necessary to
have a unique copy of the user profile that registers the
users behavior in visiting the e-service providers during
the various sessions, possibly carried out by means of
different devices. For this reason, the profile of a user
must be handled and stored in a support different from
the devices generally exploited by the user for accessing
e-service providers. As a consequence, on the user side,
the exploitation of a profile agent appears compulsory,
storing the profiles of both involved users and devices,
and a user-device agent, associated with a specific user
operating by means of a specific device, supporting the
user in his or her activities.
As previously pointed out, for each user, a unique
profile is mined and maintained, storing information
about the users behavior in accessing all e-service pro-
viders1the techniques for mining, maintaining, and
exploiting user profiles are quite complex and slightly
differ in the various applications domains; the interested
reader can find examples of them, along with the corre-
sponding validation issues, in De Meo, Rosaci, Sarn,
Terracina, and Ursino (2003) for e-commerce and in De
Meo, Garro, Terracina, and Ursino (2003) for e-learn-
ing. In this way, ESA-Manager solves the first problem
mentioned previously.
Whenever a user accesses an e-service by means of a
certain device, the corresponding service provider agent
sends information about its proposals to the user device
agent associated with the service provider agent and the
device he or she is exploiting. The user device agent
determines similarities between the proposals presented
by the provider and the interests of the user. For each of
these similarities, both the service provider agent and
the user device agent cooperate for presenting to the
user a group of Web pages adapted to the exploited
device, illustrating the proposal.
We argue that this behavior provides ESA-Manager
with the capability of supporting the user in the search of
proposals of the users interest offered by the provider.
In addition, the algorithms underlying ESA-Manager al-
low it to identify not only the proposals probably inter-
esting for the user in the present, but also other ones
possibly interesting for the user in the future and that the
user disregarded to take into account in the past (see De A
Meo, Rosaci, Sarn, Terracina & Ursino [2003] for a
specialization of these algorithms to e-commerce). In
our opinion, this is a particularly interesting feature for
a novel approach devoted to deal with e-services.
Last, but not the least, it is worth observing that
since the user profile management is carried out at the
user side, no information about the user profile is sent
to the e-service providers. In this way, ESA-Manager
solves privacy problems left open by cookies.
All the reasonings presented show that ESA-Man-
ager is capable of solving also the second problem
mentioned previously.
In ESA-Manager, the device profile plays a central
role. Indeed, the proposals of a provider shown to a
user, as well as their presentation formats, depend on
the characteristics of the device the user is presently
exploiting. However, the ESA-Manager capability of
adapting its behavior to the device the user is exploiting
is not restricted to the presentation format of the
proposals; indeed, the exploited device can influence
also the computation of the interest degree shown by a
user for the proposals presented by each provider.
More specifically, one of the parameters that the
interest degree associated with a proposal is based on,
is the time the user spends visiting the corresponding
Web pages. This time is not to be considered as an
absolute measure, but it must be normalized w.r.t. both
the characteristics of the exploited device and the
navigation costs (Chan, 2000). The following example
allows this intuition to be clarified. Assume that a user
visits a Web page for two times and that each visit takes
n seconds. Suppose, also, that during the first access,
the user exploits a mobile phone having a low proces-
sor clock and supporting a connection characterized by
a low bandwidth and a high cost. During the second
visit, the user uses a personal computer having a high
processor clock and supporting a connection charac-
terized by a high bandwidth and a low cost. It is possible
to argue that the interest the user exhibited for the page
in the former access is greater than what the user
exhibited in the latter one. Also, other device param-
eters influence the behavior of ESA-Manager (see De
Meo, Rosaci, Sarn, Terracina & Ursino [2003] for a
detailed specification of the role of these parameters).
This reasoning allows us to argue that ESA-Manager
solves also the third problem mentioned previously.
As already pointed out, many agents are simulta-
neously active in ESA-Manager; they strongly interact
with each other and continuously exchange informa-
tion. In this scenario, an efficient management of in-
formation exchange appears crucial. One of the most
promising solutions to this problem has been the adop-
25
TEAM LinG

tion of XML. XML capabilities make it particularly suited
to be exploited in the agent research. In ESA-Manager, the
role of XML is central; indeed, (1) the agent ontologies are
stored as XML documents; (2) the agent communication
language is ACML; (3) the extraction of information from
the various data structures is carried out by means of
XQuery; and (4) the manipulation of agent ontologies is
performed by means of the Document Object Model
(DOM).
FUTURE TRENDS
The spectacular growth of the Internet during the last
decade has strongly conditioned the e-service land-
scape. Such a growth is particularly surprising in some
application domains, such as financial services or e-
government.
As an example, the Internet technology has enabled
the expansion of financial services by integrating the
already existing, quite variegate financial data and ser-
vices and by providing new channels for information
delivery. For instance, in 2004, the number of house-
holds in the U.S. that will use online banking is expected
to exceed approximately 24 million, nearly double the
number of households at the end of 2000.
Moreover, e-services are not a leading paradigm
only in business contexts, but they are an emerging
standard in several application domains. As an example,
they are applied vigorously by governmental units at
national, regional, and local levels around the world.
Moreover, e-service technology is currently success-
fully exploited in some metropolitan networks for pro-
viding mediation tools in a democratic system in order
to make citizen participation in rule- and decision-
making processes more feasible and direct. These are
only two examples of the role e-services can play in the
e-government context. Handling and managing this tech-
nology in all these environments is one of the most
challenging issues for present and future researchers.
CONCLUSION
In this article, we have proposed ESA-Manager, an XML-
based and adaptive multi-agent system for supporting a
user accessing an e-service provider in the search of
proposals present therein and appearing to be appealing
according to the users past interests and behavior.
We have shown that ESA-Manager is adaptive w.r.t.
the profile of both the user and the device the user is
exploiting for accessing the e-service provider. Finally,
we have seen that it is XML-based, since XML is ex-
26
Agent-Based Mining of User Profiles for E-Services
ploited for both storing the agent ontologies and for
handling the agent communication.
As for future work, we argue that various improve-
ments could be performed on ESA-Manager for better-
ing its effectiveness and completeness. As an example,
it might be interesting to categorize involved users on
the basis of their profiles, as well as involved providers
on the basis of their proposals. As a further example of
profitable features with which our system could be
enriched, we consider extremely promising the deriva-
tion of association rules representing and predicting the
user behavior on accessing one or more providers.
Finally, ESA-Manager could be made even more adap-
tive by considering the possibility to adapt its behavior
on the basis not only of the device a user is exploiting
during a certain access, but also of the context (e.g., job,
holidays) in which the user is currently operating.
REFERENCES
Adaptive Web. (2002). Communications of the ACM,
45(5).
Anderson, C.R., Domingos, P., & Weld, D.S. (2001).
Adaptive Web navigation for wireless devices. Pro-
ceedings of the Seventeenth International Joint Con-
ference on Artificial Intelligence (IJCAI 2001), Se-
attle, Washington.
Araniti, G., De Meo, P., Iera, A., & Ursino, D. (2003).
Adaptively controlling the QoS of multimedia wireless
applications through user-profiling techniques. Jour-
nal of Selected Areas in Communications, 21(10),
1546-1556.
Ardissono, L. et al. (2001). Agent technologies for the
development of adaptive Web stores. Agent Mediated
Electronic Commerce, The European AgentLink Per-
spective (pp. 194-213). Lecture Notes in Computer Sci-
ence, Springer.
Bradley, K., & Smyth, B. (2003). Personalized informa-
tion ordering: A case study in online recruitment. Knowl-
edge-Based Systems, 16(5-6), 269-275.
Chan, P.K. (2000). Constructing Web user profiles: A
non-invasive learning approach. Web Usage Analysis
and User Profiling, 39-55. Springer.
De Meo, P., Garro, A., Terracina, G., & Ursino, D.
(2003). X-Learn: An XML-based, multi-agent system
for supporting user-device adaptive e-learning. Pro-
ceedings of the International Conference on Ontolo-
gies, Databases and Applications of Semantics
(ODBASE 2003), Taormina, Italy.
TEAM LinG
Agent-Based Mining of User Profiles for E-Services
De Meo, P., Rosaci, D., Sarn G.M.L., Terracina, G., &
Ursino, D. (2003). An XML-based adaptive multi-agent
system for handling e-commerce activities. Proceed-
ings of the International Conference on Web Ser-
vicesEurope (ICWS-Europe 03), Erfurt, Germany.
Garcia, F.J., Patern, F., & Gil, A.B. (2002). An adaptive
e-commerce system definition. Proceedings of the
International Conference on Adaptive Hypermedia
and Adaptive Web-Based Systems (AH02), Malaga,
Spain.
Hull, R., Benedikt, M., Christophides, V., & Su, J. (2003).
E-services: A look behind the curtain. Proceedings of
the Symposium on Principles of Database Systems
(PODS 2003), San Diego, California.
Lau, R., Hofstede, A., & Bruza, P. (2000). Adaptive
profiling agents for electronic commerce. Proceed-
ings of the CollECTeR Conference on Electronic Com-
merce (CollECTeR 2000), Breckenridge, Colorado.
Macskassy, S.A., Dayanik, A.A., & Hirsh, H. (2000). Infor-
mation valets for intelligent information access. Proceed-
ings of the AAAI Spring Symposia Series on Adaptive
User Interfaces, (AUI-2000), Stanford, California.
Razek, M.A., Frasson, C., & Kaltenbach, M. (2002).
Toward more effective intelligent distance learning
environments. Proceedings of the International Con-
ference on Machine Learning and Applications
(ICMLA02), Las Vegas, Nevada.
Samaras, G., & Panayiotou, C. (2002). Personalized
portals for the wireless user based on mobile agents.
Proceedings of the International Workshop on Mo-
bile Commerce, Atlanta, Georgia.
Shang, Y., Shi, H., & Chen, S. (2001). An intelligent
distributed environment for active learning. Proceed-
ings of the ACM International Conference on World
Wide Web (WWW 2001), Hong Kong.
Terziyan, V., & Vitko, O. (2002). Intelligent informa-
tion management in mobile electronic commerce. Arti-
ficial Intelligence News, Journal of Russian Associa-
tion of Artificial Intelligence, 5.
Zaiane, O.R. (2002). Building a recommender agent for
e-learning systems. Proceedings of the International
Conference on Computers in Education (ICCE 2002),
Auckland, New Zealand.
KEY TERMS
ACML: The XML encoding of the Agent Communica-
tion Language defined by the Foundation for Intelligent
Physical Agent (FIPA). A
Adaptive System: A system adapting its behavior on
the basis of the environment it is operating in.
Agent: A computational entity capable of both per-
ceiving dynamic changes in the environment it is oper-
ating in and autonomously performing user delegated
tasks, possibly by communicating and cooperating with
other similar entities.
Agent Ontology: A description (like a formal speci-
fication of a program) of the concepts and relationships
that can exist for an agent or a community of agents.
Device Profile: A model of a device storing infor-
mation about both its costs and capabilities.
E-Service: A collection of network-resident soft-
ware programs that collaborate for supporting users in
both accessing and selecting data and services of their
interest handled by a provider site. Examples of e-
services are e-commerce, e-learning, and e-government
applications.
eXtensible Markup Language (XML): The novel
language, standardized by the World Wide Web Consor-
tium, for representing, handling, and exchanging infor-
mation on the Web.
Multi-Agent System (MAS): A loosely coupled
network of software agents that interact to solve prob-
lems that are beyond the individual capacities or knowl-
edge of each of them. An MAS distributes computa-
tional resources and capabilities across a network of
interconnected agents. The agent cooperation is handled
by means of an Agent Communication Language.
User Modeling: The process of gathering informa-
tion specific to each user either explicitly or implicitly.
This information is exploited in order to customize the
content and the structure of a service to the users
specific and individual needs.
User Profile: A model of a user representing both
the users preferences and behavior.
ENDNOTE
1
It is worth pointing out that providers could be
either homogeneous (i.e., all of them operate in
the same application context, such as e-commerce)
or heterogeneous (i.e., they operate in different
application contexts).
27
TEAM LinG
28
Aggregate Query Rewriting in Multidimensional
Databases
Leonardo Tininini
CNR - Istituto di Analisi dei Sistemi e Informatica Antonio Ruberti, Italy
INTRODUCTION
An efficient query engine is certainly one of the most
important components in data warehouses (also known as
OLAP systems or multidimensional databases) and its
efficiency is influenced by many other aspects, both
logical (data model, policy of view materialization, etc.)
and physical (multidimensional or relational storage, in-
dexes, etc). As is evident, OLAP queries are often based
on the usual metaphor of the data cube and the concepts
of facts, measures and dimensions and, in contrast to
conventional transactional environments, they require
the classification and aggregation of enormous quantities
of data. In spite of that, one of the fundamental require-
ments for these systems is the ability to perform multidi-
mensional analyses in online response times. Since the
evaluation from scratch of a typical OLAP aggregate
query may require several hours of computation, this can
only be achieved by pre-computing several queries, stor-
ing the answers permanently in the database and then
reusing them in the query evaluation process. These pre-
computed queries are commonly referred to as material-
ized views and the problem of evaluating a query by using
(possibly only) these precomputed results is known as
the problem of answering/rewriting queries using views.
In this paper we briefly analyze the difference between
query answering and query rewriting approach and why
query rewriting is preferable in a data warehouse context.
We also discuss the main techniques proposed in litera-
ture to rewrite aggregate multidimensional queries using
materialized views.
BACKGROUND
Multidimensional data are obtained by applying aggrega-
tions and statistical functions to elementary data, or more
precisely to data groups, each containing a subset of the
data and homogeneous with respect to a given set of
attributes. For example, the data Average duration of
calls in 2003 by region and call plan is obtained from the
so-called fact table, which is usually the product of
complex source integration activities (Lenzerini, 2002) on
the raw data corresponding to each phone call in that year.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Several groups are defined; each consisting of calls made
in the same region and with the same call plan, and finally
applying the average aggregation function on the dura-
tion attribute of the data in each group. The pair of values
(region, call plan) is used to identify each group and is
associated with the corresponding average duration value.
In multidimensional databases, the attributes used to
group data define the dimensions, whereas the aggregate
values define the measures.
The term multidimensional data comes from the well-
known metaphor of the data cube (Gray, Bosworth, Lay-
man, & Pirahesh, 1996). For each of n attributes, used to
identify a single measure, a dimension of an n-dimensional
space is considered. The possible values of the identify-
ing attributes are mapped to points on the dimensions
axis, and each point of this n-dimensional space is thus
mapped to a single combination of the identifying at-
tribute values and hence to a single aggregate value. The
collection of all these points, along with all possible
projections in lower dimensional spaces, constitutes the
so-called data cube. In most cases, dimensions are struc-
tured in hierarchies, representing several granularity lev-
els of the corresponding measures (Jagadish, Lakshmanan,
& Srivastava, 1999). Hence a time dimension can be
organized into days, months and years; a territorial dimen-
sion into towns, regions and countries; a product dimen-
sion into brands, families and types. When querying
multidimensional data, the user specifies the measures of
interest and the level of detail required by indicating the
desired hierarchy level for each dimension. In a multidi-
mensional environment querying is often an exploratory
process, where the user moves along the dimension
hierarchies by increasing or reducing the granularity of
displayed data. The drill-down operation corresponds to
an increase in detail, for example, by requesting the
number of calls by region and month, starting from data
on the number of calls by region or by region and year.
Conversely, roll-up allows the user to view data at a
coarser level of granularity (Agrawal, Gupta, & Sarawagi,
1997; Cabibbo & Torlone, 1997).
Multidimensional querying systems are commonly
known as OLAP (Online Analytical Processing) Systems,
in contrast to conventional OLTP (Online Transactional
Processing) Systems. The two types have several con-
TEAM LinG
Aggregate Query Rewriting in Multidimensional Databases
trasting features, although they share the same require-
ment of fast online response times. In particular, one of
the key differences between OLTP and OLAP queries is
the number of records required to calculate the answer.
OLTP queries typically involve a rather limited number of
records, accessed through primary key or other specific
indexes, which need to be processed for short, isolated
transactions or to be issued on a user interface. In con-
trast, multidimensional queries usually require the classi-
fication and aggregation of a huge amount of data (Gupta,
Harinarayan, & Quass, 1995) and fast response times are
made possible by the extensive use of pre-computed
queries, called materialized views (whose answers are
stored permanently in the database), and by sophisti-
cated techniques enabling the query engine to exploit
these pre-computed results.
MAIN THRUST
The problem of evaluating the answer to a query by using
pre-computed (materialized) views has been extensively
studied in literature and generically denoted as answering
queries using views (Levy, Mendelzon, Sagiv, &
Srivastava, 1995; Halevy, 2001). The problem can be
informally stated as follows: given a query Q and a
collection of views V over the same schema s, is it possible
to evaluate the answer to Q by using (only) the informa-
tion provided by V? A more rigorous distinction has also
been made between view-based query rewriting and query
answering, corresponding to two distinct approaches to
the general problem (Calvanese, De Giacomo, Lenzerini, &
Vardi, 2000; Halevy, 2001). This is strictly related to the
distinction between view definition and view extension,
which is analogous to the standard distinction between
schema and instance in database literature. Broadly speak-
ing, view definition corresponds to the way the query is
syntactically defined, for example to the corresponding
SQL expression, while its extension corresponds to the
set of returned tuples, that is, the result obtained by
evaluating the view on a specific database instance.
Query Answering vs. Query Rewriting
Query rewriting is based on the use of view definitions to
produce a new rewritten query, expressed in terms of
available view names and equivalent to the original. The
answer can then be obtained by using the rewritten query
and the view extensions (instances). Query answering, in
contrast, is based on the exploitation of both view defini-
tions and extensions and attempts to determine the best
possible answer, possibly a subset of the exact answer,
which can be extracted from the view extensions
(Abiteboul & Duschka, 1998; Grahne & Mendelzon, 1999).
In general, query answering techniques are preferable
in contexts where exact answers are unlikely to be ob- A
tained (e.g., integration of heterogeneous data sources,
like Web sites), and response time requirements are not
very stringent. However, as noted in Grahne & Mendelzon
(1999), query answering methods can be extremely ineffi-
cient, as it is difficult or even impossible to process only
the useful views and apply optimization techniques
such as pushing selections and joins. As a consequence,
the rewriting approach is more appropriate in contexts
such as OLAP systems, where there is a very large amount
of data and fast response times are required (Goldstein &
Larson, 2001), and for query optimization, where different
query plans need to be maintained in the main memory and
efficiently compared (Afrati, Li, & Ullman, 2001).
Rewriting and Answering: An Example
Consider a fact table Cens, of elementary census data on
the simplified schema: (Census_tract_ID, Sex,
Empl_status, Educ_status, Marital_status) and a collec-
tion of aggregate data representing the resident popula-
tion by sex and marital status, stored in a materialized view
on the schema V: (Sex, Marital_status, Pop_res). For
simplicity, it is assumed that the dimensional tables are
collapsed in the fact table Cens. A typical multidimen-
sional query will be shown in the next section. The view
V is computed by a simple count(*)-group-by query on
the table Cens.
CREATE VIEW V AS
SELECT Sex, Marital_status, COUNT(*) AS Pop_res
FROM Cens
GROUP BY Sex, Marital_status
The query Q expressed by
SELECT Marital_status, COUNT(*)
FROM Cens
GROUP BY Marital_status
corresponding to the resident population by marital sta-
tus can be computed without accessing the data in Cens,
and be rewritten as follows:
SELECT Marital_status, SUM(Pop_res)
FROM V
GROUP BY Marital_status
Note that the rewritten query can be obtained very
efficiently by simple syntactic manipulations on Q and V
and its applicability does not depend on the records in V.
Suppose now some subsets of (views on) Cens are avail-
able, corresponding to the employment statuses stu-
29
TEAM LinG
 Aggregate Query Rewriting in Multidimensional Databases
dents, employed and retired, called V_ST, V_EMP and
V_RET respectively. For example V_RET may be defined
by:
CREATE VIEW V_RET AS
SELECT *
FROM Cens
WHERE Empl_status = retired
It is evident that no rewriting can be obtained by using
only the specified views, both because some individuals
are not present in any of the views (e.g., young children,
unemployed, housewives, etc.) and because some may be
present in two views (a student may also be employed).
However, a query answering technique tries to collect
each useful accessible record and build the best pos-
sible answer, possibly by introducing approximations.
By using the information on the census tract and a match-
ing algorithm most overlapping records may be determined
and an estimate (lower bound) of the result obtained by
summing the non-replicated contributions from the views.
Obviously, this would require a considerable computation
time, but it might be able to produce an approximated
answer, in a situation where rewriting techniques would
produce no answer at all.
Rewriting Aggregate Queries
A typical elementary multidimensional query is de-
scribed by the join of the fact table with two or more
dimension tables to which is applied an aggregate group
by query (see the example query Q1 below). As a conse-
quence, the rewriting of this form of query and view has
been studied by many researchers.
SELECT D1.dim1, D2.dim2, AGG(F.measure)
FROM fact_table F, dim_table1 D1, dim_table2 D2
WHERE F.dimKey1 = D1.dimKey1
AND F.dimKey2 = D2.dimKey2
GROUP BY D1.dim1, D2.dim2 (Q1)
In Gupta, Harinarayan, & Quass (1995), an algorithm is
proposed to rewrite conjunctive queries with aggrega-
tions using views of the same form. The technique is based
on the concept of generalized projection (GP) and some
transformation rules utilizable by an optimizer, which en-
ables the query and views to be put in a particular normal
form, based on GPSJ (Generalized Projection/Selection/
Join) expressions. The query and views are analyzed in
terms of their query tree, that is, the tree representing how
to calculate them by applying selections, joins and gener-
alized projections on the base relations. By using the
transformation rules, the algorithm tries to produce a
match between one or more view trees and subtrees (and
30
consequently to replace the calculations with access to
the corresponding materialized views). The results are
extended to NGPSJ (Nested GPSJ) expressions in
Golfarelli & Rizzi (2000).
In Srivastava, Dar, Jagadish, & Levy (1996) an algo-
rithm is proposed to rewrite a single block (conjunctive)
SQL query with GROUP BY and aggregations using
various views of the same form. The aggregate functions
considered are MIN, MAX, COUNT and SUM. The
algorithm is based on the detection of homomorphisms
from view to query, as in the non-aggregate context
(Levy, Mendelzon, Sagiv, & Srivastava, 1995). However,
it is shown that more restrictive conditions must be
considered when dealing with aggregates, as the view
has to produce not only the right tuples, but also their
correct multiplicities.
In Cohen, Nutt, & Serebrenik (1999, 2000) a somewhat
different approach is proposed: the original query, us-
able views and rewritten query are all expressed by an
extension of Datalog with aggregate functions (again
COUNT, SUM, MIN and MAX) as query language.
Queries and views are assumed to be conjunctive. Sev-
eral candidates for rewriting of particular forms are con-
sidered and for each candidate, the views in its body are
unfolded (i.e., replaced by their body in the view defini-
tion). Finally, the unfolded candidate is compared with
the original query to verify equivalence by using known
equivalence criteria for aggregate queries, particularly
those proposed in Nutt, Sagiv, & Shurin (1998) for COUNT,
SUM, MIN and MAX queries. The technique can be
extended by using the equivalence criteria for AVG
queries presented in Grumbach, Rafanelli, & Tininini
(1999), based on the syntactic notion of isomorphism
modulo a product.
In query rewriting it is important to identify the views
that may be actually useful in the rewriting process: this
is often referred to as the view usability problem. In the
non-aggregate context, it is shown (Levy, Mendelzon,
Sagiv, & Srivastava, 1995) that a conjunctive view can be
used to produce a conjunctive rewritten query if a homo-
morphism exists from the body of the view to that of the
query. Grumbach, Rafanelli, & Tininini (1999) demon-
strate that more restrictive (necessary and sufficient)
conditions are needed for the usability of conjunctive
count views for rewriting of conjunctive count queries,
based on the concept of sound homomorphisms. It is
also shown that in the presence of aggregations, it is not
sufficient only to consider rewritten queries of conjunc-
tive form: more complex forms may be required, particu-
larly those based on the concept of isomorphism modulo
a product.
All rewriting algorithms proposed in the literature are
based on trying to obtain a rewritten query with a particu-
lar form by using (possibly only) the available views. An
TEAM LinG
Aggregate Query Rewriting in Multidimensional Databases
interesting question is: Can I rewrite more by consider-
ing rewritten queries of more complex form?, and the
even more ambitious one, Given a collection of views,
is the information they provide sufficient to rewrite a
query? In Grumbach & Tininini (2003) the problem is
investigated in a general framework based on the con-
cept of query subsumption. Basically, the information
content of a query is characterized by its distinguishing
power, that is, by its ability to determine that two
database instances are different. Hence a collection of
views subsumes a query if it is able to distinguish any
pair of instances also distinguishable by the query, and
it is shown that a query rewriting using various views
exists if the views subsume the query. In the particular
case of count and sum queries defined over the same fact
table, an algorithm is proposed which is demonstrated to
be complete. In other words, even if the algorithm (as
with any algorithm of practical use) considers rewritten
queries of particular forms, it is shown that no improve-
ment could be obtained by considering rewritten que-
ries of more complex forms.
Finally, in Grumbach & Tininini (2000) a completely
different approach to the problem of aggregate rewriting
is proposed. The technique is based on the idea of
formally expressing the relationships (metadata) between
raw and aggregate data and also among aggregate data of
different types and/or levels of detail. Data is stored in
standard relations, while the metadata are represented by
numerical dependencies, namely Horn clauses formally
expressing the semantics of the aggregate attributes. The
mechanism is tested by transforming the numerical de-
pendencies into Prolog rules and then exploiting the
Prolog inference engine to produce the rewriting.
FUTURE TRENDS
Although query rewriting techniques are currently con-
sidered to be preferable to query answering in OLAP
systems, the always increasing processing capabilities of
modern computers may change the relevance of query
answering techniques in the near future. Meanwhile, the
limitations in the applicability of several rewriting algo-
rithms shows that a substantial effort is still needed and
important contributions may stem from results in other
research areas like logic programming and automated
reasoning. Particularly, aggregate query rewriting is strictly
related to the problem of query equivalence for aggregate
queries and current equivalence criteria only apply to
rather simple forms of query, and dont consider, for
example, the combination of conjunctive formulas with
nested aggregations.
Also the results on view usability and query
subsumption can be considered only preliminary and it
would be interesting to study the property of complete-
ness of known rewriting algorithms and to provide neces- A
sary and sufficient conditions for the usability of a view
to rewrite a query, even when both the query and the view
are aggregate and of non-trivial form (e.g., allowing dis-
junction and some limited form of negation).
CONCLUSION
This paper has discussed a fundamental issue related to
multidimensional query evaluation, that is, how a multidi-
mensional query expressed in a given language can be
translated, using some available materialized views, into
an (efficient) evaluation plan which retrieves the neces-
sary information and calculates the required results. We
have analyzed the difference between query answering
and query rewriting approach and discussed the main
techniques proposed in literature to rewrite aggregate
multidimensional queries using materialized views.
REFERENCES
Abiteboul, S., & Duschka, O.M. (1998). Complexity of
answering queries using materialized views. In ACM Sym-
posium on Principles of Database Systems (PODS98)
(pp. 254-263).
Afrati, F.N., Li, C., & Ullman, J.D. (2001). Generating
efficient plans for queries using views. In ACM Interna-
tional Conference on Management of Data (SIGMOD01)
(pp. 319-330).
Agrawal, R., Gupta, A., & Sarawagi, S. (1997). Modeling
multidimensional databases. In International Confer-
ence on Data Engineering (ICDE97) (pp. 232-243).
Cabibbo, L., & Torlone, R. (1997). Querying multidimen-
sional databases. In International Workshop on Data-
base Programming Languages (DBPL97) (pp. 319-335).
Calvanese, D., De Giacomo, G., Lenzerini, M., & Vardi,
M.Y. (2000). What is view-based query rewriting? In
International Workshop on Knowledge Representation
meets Databases (KRDB00) (pp. 17-27).
Cohen, S., Nutt, W., & Serebrenik, A. (1999). Rewriting
aggregate queries using views. In ACM Symposium on
Principles of Database Systems (PODS99) (pp. 155-166).
Cohen, S., Nutt, W., & Serebrenik, A. (2000). Algorithms
for rewriting aggregate queries using views. In ABDIS-
DASFAA Conference 2000 (pp. 65-78).
Goldstein, J., & Larson, P. (2001). Optimizing queries
using materialized views: A practical, scalable solution. In
31
TEAM LinG
 Aggregate Query Rewriting in Multidimensional Databases
ACM International Conference on Management of Data
(SIGMOD01) (pp. 331-342).
Golfarelli, M., & Rizzi, S. (2000). Comparing nested GPSJ
queries in multidimensional databases. In Workshop on
Data Warehousing and OLAP (DOLAP 2000) (pp. 65-71).
Grahne, G., & Mendelzon, A.O. (1999). Tableau tech-
niques for querying information sources through global
schemas. In International Conference on Database
Theory (ICDT99) (pp. 332-347).
Gray, J., Bosworth, A., Layman, A., & Pirahesh, H. (1996).
Data cube: A relational aggregation operator generalizing
group-by, cross-tab, and sub-total. In International Con-
ference on Data Engineering (ICDE96) (pp. 152-159).
Grumbach, S., Rafanelli, M., & Tininini, L. (1999). Query-
ing aggregate data. In ACM Symposium on Principles of
Database Systems (PODS99) (pp. 174-184).
Grumbach, S., & Tininini, L. (2000). Automatic aggrega-
tion using explicit metadata. In International Conference
on Scientific and Statistical Database Management
(SSDBM00) (pp. 85-94).
Grumbach, S., & Tininini, L. (2003). On the content of
materialized aggregate views. Journal of Computer and
System Sciences, 66(1), 133-168.
Gupta, A., Harinarayan, V., & Quass, D. (1995). Aggre-
gate-query processing in data warehousing environments.
In International Conference on Very Large Data Bases
(VLDB95) (pp. 358-369).
Halevy, A.Y. (2001). Answering queries using views.
VLDB Journal, 10(4), 270-294.
Jagadish, H.V., Lakshmanan, L.V.S., & Srivastava, D.
(1999). What can hierarchies do for data warehouses? In
International Conference on Very Large Data Bases
(VLDB99) (pp. 530-541).
Lenzerini, M. (2002). Data integration: A theoretical per-
spective. In ACM Symposium on Principles of Database
Systems (PODS02) (pp. 233-246).
Levy, A.Y., Mendelzon, A.O., Sagiv, Y., & Srivastava, D.
(1995). Answering queries using views. In ACM Sympo-
sium on Principles of Database Systems (PODS95) (pp.
95-104).
Nutt, W., Sagiv, Y., & Shurin, S. (1998). Deciding equiva-
lences among aggregate queries. In ACM Symposium on
32
Principles of Database Systems (PODS98) (pp. 214-
223).
Srivastava, D., Dar, S., Jagadish, H.V., & Levy, A.Y.
(1996). Answering queries with aggregation using views.
In International Conference on Very Large Data Bases
(VLDB96) (pp. 318-329).
KEY TERMS
Data Cube: A collection of aggregate values classified
according to several properties of interest (dimensions).
Combinations of dimension values are used to identify the
single aggregate values in the cube.
Dimension: A property of the data used to classify it
and navigate the corresponding data cube. In multidimen-
sional databases dimensions are often organized into
several hierarchical levels, for example, a time dimension
may be organized into days, months and years.
Drill-Down (Roll-Up): Typical OLAP operation, by
which aggregate data are visualized at a finer (coarser)
level of detail along one or more analysis dimensions.
Fact: A single elementary datum in an OLAP system,
the properties of which correspond to dimensions and
measures.
Fact Table: A table of (integrated) elementary data
grouped and aggregated in the multidimensional query-
ing process.
Materialized View: A particular form of query whose
answer is stored in the database to accelerate the evalu-
ation of further queries.
Measure: A numeric value obtained by applying an
aggregate function (such as count, sum, min, max or
average) to groups of data in a fact table.
Query Answering: Process by which the (possibly
approximate) answer to a given query is obtained by
exploiting the stored answers and definitions of a collec-
tion of materialized views.
Query Rewriting: Process by which a source query is
transformed into an equivalent one referring (almost ex-
clusively) to a collection of materialized views. In multidi-
mensional databases, query rewriting is fundamental in
achieving acceptable (online) response times.
TEAM LinG

Aggregation for Predictive Modeling with
Relational Data
Claudia Perlich
IBM Research, USA
Foster Provost
New York University, USA
INTRODUCTION
Most data mining and modeling techniques have been
developed for data represented as a single table, where
every row is a feature vector that captures the character-
istics of an observation. However, data in most domains
are not of this form and consist of multiple tables with
several types of entities. Such relational data are ubiq-
uitous; both because of the large number of multi-table
relational databases kept by businesses and government
organizations, and because of the natural, linked nature
of people, organizations, computers, and etc. Relational
data pose new challenges for modeling and data mining,
including the exploration of related entities and the aggre-
gation of information from multi-sets (bags) of related
entities.
BACKGROUND
Relational learning differs from traditional feature-
vector learning both in the complexity of the data repre-
sentation and in the complexity of the models. The
relational nature of a domain manifests itself in two
ways: (1) entities are not limited to a single type, and (2)
entities are related to other entities. Relational learning
allows the incorporation of knowledge from entities in
multiple tables, including relationships between ob-
jects of varying cardinality. Thus, in order to succeed,
relational learners have to be able to identify related
objects and to aggregate information from bags of re-
lated objects into a final prediction.
Traditionally, the analysis of relational data has in-
volved the manual construction by a human expert of
attributes (e.g., the number of purchases of a customer
during the last three months) that together will form a
feature vector. Automated analysis of relational data is
becoming increasingly important as the number and
complexity of databases increases. Early research on
automated relational learning was dominated by Induc-
tive Logic Programming (Muggleton, 1992), where the
classification model is a set of first-order-logic clauses
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
33
A
and the information aggregation is based on existential
unification. More recent relational learning approaches
include distance-based methods (Kirsten et al., 2001),
propositionalization (Kramer et al., 2001; Knobbe et
al., 2001; Krogel et al., 2003), and upgrades of proposi-
tional learners such as Nave Bayes (Neville et al.,
2003), Logistic Regression (Popescul et al., 2002),
Decision Trees (Jensen & Neville, 2002) and Bayesian
Networks (Koller & Pfeffer, 1998). Similar to manual
feature construction, both upgrades and
propositionalization use Boolean conditions and com-
mon aggregates like min, max, or sum to transform
either explicitly (propositionalization) or implicitly
(upgrades) the original relational domain into a tradi-
tional feature-vector representation.
Recent work by Knobbe et al. (2001) and Wrobel &
Krogel (2001) recognizes the essential role of aggrega-
tion in all relational modeling and focuses specifically
on the effect of aggregation choices and parameters.
Wrobel & Krogel (2003) present one of the few empiri-
cal comparisons of aggregation in propositionalization
approaches (however with inconclusive results). Perlich
& Provost (2003) show that the choice of aggregation
operator can have a much stronger impact on the result-
ant models generalization performance than the choice
of the model induction method (decision trees or logis-
tic regression, in their study).
MAIN THRUST
For illustration, imagine a direct marketing task where
the objective is to identify customers who would re-
spond to a special offer. Available are demographic
information and all previous purchase transactions, which
include PRODUCT, TYPE and PRICE. In order to take
advantage of these transactions, information has to be
aggregated. The choice of the aggregation operator is
crucial, since aggregation invariably involves loss of
(potentially discriminative) information.
Typical aggregation operators like min, max and sum
can only be applied to sets of numeric values, not to
TEAM LinG
 Aggregation for Predictive Modeling with Relational Data
objects (an exception being count). It is therefore neces-
sary to assume class-conditional independence and ag-
gregate the attributes independently, which limits the
expressive power of the model. Perlich & Provost (2003)
discuss in detail the implications of various assump-
tions and aggregation choices on the expressive power
of resulting classification models. For example, cus-
tomers who buy mostly expensive books cannot be
identified if price and type are aggregated separately. In
contrast, ILP methods do not assume independence and
can express an expensive book (TYPE=BOOK and
PRICE>20); however aggregation through existential
unification can only capture whether a customer bought
at least one expensive book, not whether he has bought
primarily expensive books. Only two systems, POLKA
(Knobbe et al., 2001) and REGLAGGS (Wrobel &
Krogel, 2001) combine Boolean conditions and nu-
meric aggregates to increase the expressive power of
the model.
Another challenge is posed by categorical attributes
with many possible values, such as ISBN numbers of
books. Categorical attributes are commonly aggregated
using mode (the most common value) or the count for
all values if the number of different values is small.
These approaches would be ineffective for ISBN: it has
many possible values and the mode is not meaningful
since customers usually buy only one copy of each
book. Many relational domains include categorical at-
tributes of this type. One common class of such do-
mains involves networked data, where most of the infor-
mation is captured by the relationships between objects,
possibly without any further attributes. The identity of
an entity (e.g., Bill Gates) in social, scientific, and
economic networks may play a much more important
role than any of its attributes (e.g., age or gender).
Identifiers such as name, ISBN, or SSN are categorical
attributes with excessively many possible values that
cannot be accounted for by either mode or count.
Perlich and Provost (2003) present a new multi-step
aggregation methodology based on class-conditional
distributions that shows promising performance on net-
Figure 1. Example domain with two tables that are
linked through Paper ID
Document Table Author Table
Paper ID Class Paper ID Author Name
P1 0 P1 A
P2 1 P2 B
P3 1 P2 A
P4 0 P3 B table with the additional cosine features is shown in
P3 A Figure 2. In this simple example, the distance from DClass
P3 C 1
P4 C 0
34
worked data with identifier attributes. As Knobbe et al.
(1999) point out, traditional aggregation operators like
min, max, and count are based on histograms. A histo-
gram itself is a crude approximation of the underlying
distribution. Rather than estimating one distribution for
every bag of attributes, as done by traditional aggrega-
tion operators, this new aggregation approach estimates
in a first step only one distribution for each class, by
combining all bags of objects for the same class. The
combination of bags of related objects results in much
better estimates of the distribution, since it uses many
more observations. The number of parameters differs
across distributions: for a normal distribution only two
parameters are required, mean and variance, whereas
distributions of categorical attributes have as many
parameters as possible attribute values. In a second step,
the bags of attributes of related objects are aggregated
through vector distances (e.g., Euclidean, Cosine, Like-
lihood) between a normalized vector-representation of
the bag and the two class-conditional distributions.
Imagine the following example of a document clas-
sification domain with two tables (Document and Au-
thor) shown in Figure 1.
The first aggregation step estimates the class-
conditional distributions DClass n of authors from the
Author table. Under the alphabetical ordering of
position:value pairs, 1:A, 2:B, and 3:C, the value for
DClass n at position k is defined as:
Number of occurrences of author k in the set of authors related to documents of class n
DClass n[k] =
Number of authors related to documents of class n
The resulting estimates of the class-conditional dis-
tributions for our example are given by:
DClass 0 = [0.5 0 0.5] and DClass 1 = [0.4 0.4 0.2]
The second aggregation step is the representation
of every document as a vector:
Number of occurrences of author k related to the document Pn
DPn[k] =
Number of authors related to document Pn
The vector-representation for the above examples
are D P1 = [1 0 0], D P2 = [0.5 0.5 0], D P3 = [0.33 0.33
0.33], and DP4 = [0 0 1].
The third aggregation step calculates vector dis-
tances (e.g., cosine) between the class-conditional distri-
bution and the documents DP1,...,DP4. The new Document
separates the examples perfectly; the distance from DClass
does not.
TEAM LinG
Aggregation for Predictive Modeling with Relational Data

Figure 2. Extended document table with new cosine high-dimensional categorical fields (author names and
features added document identifiers) are not applicable. A
The generalization performance of the new aggrega-
Document Table tion approach is related to a number of properties that are
Paper ID Class Cosine(Pn, DClass 1) Cosine(Pn, DClass 0)
of particular relevance and advantage for predictive
P1 0 0.667 0.707
P2 1 0.707 0.5 modeling:
P3 1 0.962 0.816
P4 0 0.333 0.707 Dimensionality Reduction: The use of distances
compresses the high-dimensional space of pos-
sible categorical values into a small set of dimen-
sions one for each class and distance metric. In
By taking advantage of DClass 1 and D Class 0 another new
particular, this allows the aggregation of object
aggregation approach becomes possible. Rather than
identifiers.
constructing counts for all distinct values (impossible
Preservation of Discriminative Information:
for high-dimensional categorical attributes) one can se-
Changing the class labels of the target objects
lect a small subset of values where the absolute differ-
will change the values of the aggregates. The loss
ence between entries in DClass 0 and D Class 1 is maximal. This
of discriminative information is lower since the
method would identify author B as the most discriminative.
class-conditional distributions capture signifi-
These new features, constructed from class-condi-
cant differences.
tional distributions, show superior classification per-
Domain Independence: The density estimation
formance on a variety of relational domains (Perlich &
does not require any prior knowledge about the
Provost, 2003, 2004). Table 1 summarizes the relative
application domain and therefore is suitable for a
out-of-sample performances (averaged over 10 experi-
variety of domains.
ments with standard deviations in parentheses) as pre-
Applicability to Numeric Attributes: The ap-
sented in Perlich (2003) on the CORA document classi-
proach is not limited to categorical values but can
fication task (McCallum et al., 2000) for 400 training
also be applied to numeric attributes after
examples. The data set includes information about the
discretization. Note that using traditional aggre-
authorship, citations, and the full text. This example also
gation through mean and variance assumes im-
demonstrates the opportunities arising from the ability
plicitly a normal distribution; whereas this aggre-
of relational models to take advantage of additional
gation makes no prior distributional assumptions
background information such as citations and authorship
and can capture arbitrary numeric distributions.
over simple text classification. The comparison includes
Monotonic Relationship: The use of distances to
in addition to two distribution-based feature construc-
class-conditional densities constructs numerical
tion approaches (1 and 2) using logistic regression for
features that are monotonic in the probability of
model induction: 3) a Nave Bayes classifier using the
class membership. This makes logistic regression
full text learned by the Rainbow (McCallum, 1996)
a natural choice for the model induction step.
system, 4) a Probabilistic Relational Model (Koller &
Aggregation of Identifiers: By using object iden-
Pfeffer, 1998) using traditional aggregates on both text
tifiers such as names it can overcome some of the
and citation/authorship with the results reported by Taskar
limitations of the independence assumptions and
et al. (2001), and 5) a Simple Relational Classifier (Macskassy
even allow the learning from unobserved object
& Provost, 2003) that uses only the known class labels of
properties (Perlich & Provost, 2004). The identifier
related (e.g., cited) documents. It is important to observe
represents the full information of the object and in
that traditional aggregation operators such as mode for

Table 1. Comparative classification performance

Method Used Information Accuracy

1) Class-Conditional Distributions (Authorship & Citations) 0.78 (0.01)

2) Class-Conditional Distributions and Most Discriminative Counts (Authorship & Citations) 0.81 (0.01)
3) Nave Bayes Classifier using Rainbow (Text) 0.74 (0.03)
4) Probabilistic Relational Model (Text, Authorship & Citiations) 0.74 (0.01)
5) Simple Relational Model (Related Class Labels) 0.68 (0.01)

35

TEAM LinG
 Aggregation for Predictive Modeling with Relational Data
particular the joint distribution of all other attributes
and even further unknown properties.
Task-Specific Feature Construction: The advan-
tages outlined above are possible through the use
the target value during feature construction. This
practice requires the splitting of the training set into
two separate portions for 1) the class-conditional
density estimation and feature construction and 2)
the estimation of the classification model.
To summarize, most relational modeling has limited
itself to a small set of existing aggregation operators.
The recognition of the limited expressive power moti-
vated the combination of Boolean conditioning and
aggregation, and the development of new aggregation
methodologies that are specifically designed for pre-
dictive relational modeling.
FUTURE TRENDS
Computer-based analysis of relational data is becoming
increasingly necessary as the size and complexity of
databases grow. Many important tasks, including
counter-terrorism (Tang et al., 2003), social and eco-
nomic network analysis (Jensen & Neville, 2002), docu-
ment classification (Perlich, 2003), customer relation-
ship management, personalization, fraud detection
(Fawcett & Provost, 1997), and genetics [e.g., see the
overview by Deroski (2001)], used to be approached with
special-purpose algorithms, but now are recognized as
inherently relational. These application domains both
profit from and contribute to research in relational model-
ing in general and aggregation for feature construction in
particular.
In order to accommodate such a variety of domains,
new aggregators must be developed. In particular, it is
necessary to account for domain-specific dependencies
between attributes and entities that currently are ig-
nored. One common type of such dependency is the
temporal order of events which is important for the
discovery of causal relationships.
Aggregation as a research topic poses the opportu-
nity for significant theoretical contributions. There is
little theoretical work on relational model estimation
outside of first-order logic. In contrast to a large body
of work in mathematics and the estimation of functional
dependencies that map well-defined input spaces to
output spaces, aggregation operators have not been in-
vestigated nearly as thoroughly. Model estimation tasks
are usually framed as search over a structured (either in
terms of parameters or increasing complexity) space of
possible solutions. But the structuring of a search space
of aggregation operators remains an open question.
36
The potential complexity of relational models and the
resulting computational complexity of relational modeling
remains an obstacle to real-time applications. This limita-
tion has spawned work in efficiency improvements (Yin et
al., 2003; Tang et al., 2003) and will remain an important
task.
CONCLUSION
Relational modeling is a burgeoning topic within ma-
chine learning research, and is applicable commonly in
real-world domains. Many domains collect large
amounts of transaction and interaction data, but so far
lack a reliable and automated mechanism for model
estimation to support decision-making. Relational
modeling with appropriate aggregation methods has the
potential to fill this gap and allow the seamless integration
of model estimation on top of existing relational data-
bases, relieving the analyst from the manual, time-con-
suming, and omission-prone task of feature construction.
REFERENCES
Deroski, S. (2001). Relational data mining applications:
An overview, In S. D eroski & N. Lavra (Eds.), Rela-
tional data mining (pp. 339-364). Berlin: Springer Verlag.
Fawcett, T., & Provost, F. (1997). Adaptive fraud detec-
tion. Data Mining and Knowledge Discovery, (1).
Jensen, D., & Neville, J. (2002). Data mining in social
networks. In R. Breiger, K. Carley, & P. Pattison (Eds.),
Dynamic social networks modeling and analysis (pp.
287-302). The National Academies Press.
Kirsten, M., Wrobel, S., & Horvath, T. (2001). Distance
based approaches to relational learning and clustering.
In S. Deroski & N. Lavra (Eds.), Relational data mining
(pp. 213-234). Berlin: Springer Verlag.
Knobbe A.J., de Haas, M., & Siebes, A. (2001).
Propositionalisation and aggregates. In L. DeRaedt & A.
Siebes (Eds.), Proceedings of the Fifth European Confer-
ence on Principles of Data Mining and Knowledge
Discovery (LNAI 2168) (pp. 277-288). Berlin: Springer
Verlag.
Koller, D., & Pfeffer, A. (1998). Probabilistic frame-
based systems. In Proceedings of Fifteenth/Tenth Con-
ference on Artificial Intelligence/Innovative Applica-
tion of Artificial Intelligence (pp. 580-587). American
Association for Artificial Intelligence.
TEAM LinG
Aggregation for Predictive Modeling with Relational Data
Kramer, S., Lavra , N., & Flach, P. (2001). Proposition-
alization approaches to relational data mining. In S.
Deroski & N. Lavra (Eds.), Relational data mining (pp.
262-291). Berlin: Springer Verlag.
Krogel, M.A., Rawles, S., Zelezny, F., Flach, P.A., Lavrac,
N., & Wrobel, S. (2003). Comparative evaluation of ap-
proaches to propositionalization. In T. Horvth & A.
Yamamoto (Eds.), Proceedings of the 13th International
Conference on Inductive Logic Programming (LNAI
2835) (pp. 197-214). Berlin: Springer-Verlag.
Krogel, M.A., & Wrobel, S. (2001). Transformation-based
learning using multirelational aggregation. In C. Rouveirol
& M. Sebag (Eds.), Proceedings of the Eleventh Interna-
tional Conference on Inductive Logic Programming (ILP)
(LNAI 2157) (pp. 142-155). Berlin: Springer Verlag.
Krogel M.A., & Wrobel, S. (2003). Facets of aggregation
approaches to propositionalization. In T. Horvth & A.
Yamamoto (Eds.), Proceedings of the Work-in-Progress
Track at the 13th International Conference on Inductive
Logic Programming (pp. 30-39).
Macskassy, S.A., & Provost, F. (2003). A simple relational
classifier. In Proceedings of the Workshop of Multi-
Relational Data Mining at SIGKDD-2003.
McCallum, A.K. (1996). Bow: A toolkit for statistical
language modeling, text retrieval, classification and
clustering. Retrieved from http://www.cs.cmu.edu/
~mccallum/bow
McCallum, A.K., Nigam, K., Rennie, J., & Seymore, K.
(2000). Automating the construction of Internet portals
with machine learning. Information Retrieval, 3(2), 127-
163.
Muggleton, S. (Ed.). (1992). Inductive logic program-
ming. London: Academic Press.
Neville J., Jensen, D., & Gallagher, B. (2003). Simple
estimators for relational bayesian classifers. In Pro-
ceedings of the Third IEEE International Conference
on Data Mining (pp. 609-612).
Perlich, C. (2003). Citation-based document classifi-
cation. In Proceedings of the Workshop on Informa-
tion Technology and Systems (WITS).
Perlich, C., & Provost, F. (2003). Aggregation-based
feature invention and relational concept classes. In Pro-
ceedings of the Ninth ACM SIGKDD International Con-
ference on Knowledge Discovery and Data Mining.
Perlich, C., & Provost, F. (2004). ACORA: Distribution-
based aggregation for relational learning from identi-
fier attributes. Working Paper CeDER-04-04. Stern School
of Business. A
Popescul, L., Ungar, H., Lawrence, S., & Pennock, D.M.
(2002). Structural logistic regression: Combining rela-
tional and statistical learning. In Proceedings of the
Workshop on Multi-Relational Data Mining.
Tang, L.R., Mooney, R.J., & Melville, P. (2003). Scaling up
ILP to large examples: Results on link discovery for
counter-terrorism. In Proceedings of the Workshop on
Multi-Relational Data Mining (pp. 107-121).
Taskar, B., Segal, E., & Koller, D. (2001). Probabilistic
classification and clustering in relational data. In Pro-
ceedings of the 17th International Joint Conference
on Artificial Intelligence (pp. 870-878).
Yin, X., Han, J., & Yang, J. (2003). Efficient multi-
relational classification by tuple ID propagation. In
Proceedings of the Workshop on Multi-Relational
Data Mining.
KEY TERMS
Aggregation: Also commonly called a summary, an
aggregation is the calculation of a value from a bag or
(multi)set of entities. Typical aggregations are sum,
count, and average.
Discretization: Conversion of a numeric variable
into a categorical variable, usually though binning. The
entire range of the numeric values is split into a number
of bins. The numeric value of the attributes is replaced by
the identifier of the bin into which it falls.
Class-Conditional Independence: Property of a multi-
variate distribution with a categorical class variable c and
a set of other variables (e.g., x and y). The probability of
observing a combination of variable values given the
class label is equal to the product of the probabilities of
each variable value given the class: P(x,y|c) = P(x|c)*P(y|c).
Inductive Logic Programming: A field of research at
the intersection of logic programming and inductive ma-
chine learning, drawing ideas and methods from both
disciplines. The objective of ILP methods is the inductive
construction of first-order Horn clauses from a set of
examples and background knowledge in relational form.
Propositionalization: The process of transforming a
multi-relational dataset, containing structured examples,
into a propositional data set (one table) with derived
attribute-value features, describing the structural proper-
ties of the example.
37
TEAM LinG
 Aggregation for Predictive Modeling with Relational Data
Relational Data: Data where the original information
cannot be represented in a single table but requires two
or more tables in a relational database. Every table can
either capture the characteristics of entities of a particular
type (e.g., person or product) or relationships between
entities (e.g., person bought product).
38
Relational Learning: Learning in relational domains
that include information from multiple tables, not based
on manual feature construction.
Target Objects: Objects in a particular target tables
for which a prediction is to be made. Other objects reside
in additional background tables, but are not the focus
of the prediction task.
TEAM LinG

API Standardization Efforts for Data Mining
Jaroslav Zendulka
Brno University of Technology, Czech Republic
INTRODUCTION
Data mining technology just recently became actually
usable in real-world scenarios. At present, the data mining
models generated by commercial data mining and statis-
tical applications are often used as components in other
systems in such fields as customer relationship manage-
ment, risk management or processing scientific data.
Therefore, it seems to be natural that most data mining
products concentrate on data mining technology rather
than on the easy-to-use, scalability, or portability. It is
evident that employing common standards greatly simpli-
fies the integration, updating, and maintenance of appli-
cations and systems containing components provided by
other producers (Grossman, Hornick, & Meyer, 2002).
Data mining models generated by data mining algorithms
are good examples of such components.
Currently, established and emerging standards ad-
dress especially the following aspects of data mining:
Metadata: for representing data mining metadata
that specify a data mining model and results of
model operations (CWM, 2001).
Application Programming Interfaces (APIs): for
employing data mining components in applications.
Process: for capturing the whole knowledge dis-
covery process (CRISP-DM, 2000).
In this paper, we focus on standard APIs. The objec-
tive of these standards is to facilitate integration of data
mining technology with application software. Probably
the best-known initiatives in this field are OLE DB for Data
Mining (OLE DB for DM), SQL/MM Data Mining (SQL/
MM DM), and Java Data Mining (JDM).
Another standard, which is not an API but is important
for integration and interoperability of data mining prod-
ucts and applications, is a Predictive Model Markup
Language (PMML). It is a standard format for data mining
model exchange developed by Data Mining Group (DMG)
(PMML, 2003). It is supported by all the standard APIs
presented in this paper.
BACKGROUND
The goal of data mining API standards is to make it
possible for different data mining algorithms from various
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
39
A
software vendors to be easily plugged into applications.
A software package that provides data mining services is
called data mining provider and an application that
employs these services is called data mining consumer.
The data mining provider itself includes three basic archi-
tectural components (Hornick et al., 2002):
API the End User Visible Component: An appli-
cation developer using a data mining provider has
to know only its API.
Data Mining Engine (or Server): the core com-
ponent of a data mining provider. It provides an
infrastructure that offers a set of data mining
services to data mining consumers.
Metadata Repository: a repository that serves to
store data mining metadata.
The standard APIs presented in this paper are not
designed to support the entire knowledge discovery
process but the data mining step only (Han & Kamber,
2001). They do not provide all necessary facilities for data
cleaning, transformations, aggregations, and other data
preparation operations. It is assumed that data prepara-
tion is done before an appropriate data mining algorithm
offered by the API is applied.
There are four key concepts that are supported by the
APIs: a data mining model, data mining task, data mining
technique, and data mining algorithm. The data mining
model is a representation of a given set of data. It is the
result of one of the data mining tasks, during which a data
mining algorithm for a given data mining technique
builds the model. For example, a decision tree as one of the
classification models is the result of a run of a decision
tree-based algorithm.
The basic data mining tasks that the standard APIs
support enable users to:
1. Build a data mining model. This task consists of two
steps. First the data model is defined, that is, the
source data that will be mined is specified, the
source data structure (referred to as physical schema)
is mapped on inputs of a data mining algorithm
(referred to as logical schema), and the algorithm
used to build the data mining model is specified.
Then, the data mining model is built from training
data.
TEAM LinG
 API Standardization Efforts for Data Mining

2. Test the quality of a mining model by applying
testing data.
3. Apply a data mining model to new data.
4. Browse a data mining model for reporting and visu-
alization applications.
The APIs support several commonly accepted and
widely used techniques both for predictive and descrip-
tive data mining (see Table 1). Not all techniques need all
the tasks listed above. For example, association rule
mining does not require testing and application to new
data, whereas classification does.
The goals of the APIs are very similar but the approach
of each of them is different. OLE DB for DM is a language-
based interface, SQL/MM DM is based on user-defined
data types in SQL:1999, and JDM contains packages of
data mining oriented Java interfaces and classes.
In the next section, each of the APIs is briefly charac-
terized. An example showing their application in predic-
tion is presented in another article in this encyclopedia.
oriented specification for a set of data access interfaces
designed for record-oriented data stores. It employs SQL
commands as arguments of interface operations. The
approach in defining OLE DB for DM was not to extend
OLE DB interfaces but to expose data mining interfaces in
a language-based API.
OLE DB for DM treats a data mining model as if it were
a special type of table: (a) Input data in the form of a set
of cases is associated with a data mining model and
additional meta-information while defining the data min-
ing model. (b) When input data is inserted into the data
mining model (it is populated), a mining algorithm
builds an abstraction of the data and stores it into this
special table. For example, if the data model represents a
decision tree, the table contains a row for each leaf node
of the tree (Netz et al., 2001). Once the data mining model
is populated, it can be used for prediction, or it can be
browsed for visualization.
OLE DB for DM extends syntax of several SQL state-
ments for defining, populating, and using a data mining
model see Figure 1.

MAIN THRUST SQL/MM Data Mining

OLE DB for Data Mining SQL/MM DM is an international ISO/IEC standard (SQL,

2002), which is part of the SQL Multimedia and Applica-
OLE DB for DM (OLE DB, 2000) is Microsofts API that tion Packages (SQL/MM) (Melton & Eisenberg, 2001). It
aims to become the industry standard. It provides a set of is based on SQL:1999 and its structured user-defined data
extensions to OLE DB, which is a Microsofts object- types (UDT). The structured UDT is the fundamental

Table 1. Supported data mining techniques

Technique OLE DB for DM SQL/MM DM JDM

Association rules X X X
Clustering (segmentation) X X X
Classification X X X
Sequence and deviation analysis X
Density estimation X
Regression X
Approximation X
Attribute importance X

Figure 1. Extended SQL statements in OLE DB for DM

source data columns

INSERT model columns SELECT

mining algorithm
algorithm settings

CREATE

Populating the data Defining a data Testing, applying, browsing

2 1 mining model 3
mining model the data mining model

40

TEAM LinG
API Standardization Efforts for Data Mining
facility in SQL:1999 that supports object orientation (Melton
& Simon, 2001). The idea of SQL/MM DM is to provide
UDTs and associated methods for defining input data, a
data mining model, data mining task and its settings, and for
results of testing or applying the data mining model. Train-
ing, testing, and application data must be stored in a table.
Relations of the UDTs are shown in Figure 2.
Some of the UDTs are related to mining techniques.
Their names contain XX in the figure, which should be
Clas, Rule, Clus, and Reg for classification, asso- systems is essential for implementations of data mining
ciation rules, clustering and regression, respectively.
Java Data Mining
Java Data Mining (JDM) known as a Java Specification
Request 73 (JSR-73) (Hornick et al., 2004) is a Java
standard being developed under SUNs Java Community
Process. The standard is based on a generalized, object-
oriented, data mining conceptual model. JDM supports
common data mining operations, as well as the creation,
persistence, access, and maintenance of metadata sup-
porting mining activities.
Compared with OLE DB for DM and SQL/MM DM,
JDM is more complex because it does not rely on any other
built-in support, such as OLE DB or SQL. It is a pure Java
API that specifies a set of Java interfaces and classes,
which must be implemented in a data mining provider.
Some of JDM concepts are close to those in SQL/MM DM
but the number of Java interfaces and classes in JDM is
higher than the number of UDTs in SQL/MM DM. JDM
specifies interfaces for objects which provide an abstrac-
tion of the metadata needed to execute data mining tasks.
Once a task is executed, another object that represents the
result of the task is created.
Figure 2. Relations of data mining UDTs introduced in SQL/MM DM
DM_MiningData
DM_LogicalDataSpec
Training data
DM_XXSettings
DM_XXBldTask
FUTURE TRENDS
A
OLE DB for DM is a Microsofts standard which aims to
be an industry standard. A reference implementation of
a data mining provider based on this standard is available
in Microsoft SQL Server 2000 (Netz et al., 2001). SQL/MM
DM was adopted as an international ISO/IEC standard.
As it is based on a user-defined data type feature of
SQL:1999, support of UDTs in database management
providers based on this standard.
JDM must still go through several steps before being
accepted as an official Java standard. At the time of
writing this paper, it was in the stage of final draft public
review. The Oracle9i Data Mining (Oracle9i, 2002) API
provides an early look at concepts and approaches
proposed for JDM. It is assumed to comply with the JDM
standard when the standard is published.
All the standards support PMML as a format for data
mining model exchange. They enable a data mining model
to be imported and exported in this format. In OLE DB for
DM, a new model can be created from a PMML document.
SQL/MM DM provides methods of the DM_XXModel
UDT to import and export a PMML document. Similarly,
JDM specifies interfaces for import and export tasks.
CONCLUSION
Three standard APIs for data mining were presented in
this paper. However, their implementation is not avail-
able yet or is only a reference one. Schwenkreis (2001)
commented on this situation, as, the fact that implemen-
tations come after standards is a general trend in todays
standardization efforts. It seems that in case of data
Testing data
DM_MinigData
DM_XXTestResult
DM_XXModel
DM_ApplicationData DM_XXResult
Application data
41
TEAM LinG

mining, standards are not only intended to unify existing
products with well-known functionality but to (partially)
design the functionality such that future products match
real world requirements. A simple example of using the
APIs in prediction is presented in another article of this
book (Zendulka, 2005).
REFERENCES
Common Warehouse Metamodel Specification: Data
Mining. Version 1.0. (2001). Retrieved from http://
www.omg.org/docs/ad/01-02-01.pdf
Cross Industry Standard Process for Data Mining
(CRISP-DM). Version 1.0. (2000). Retrieved from http://
www.crisp-dm.org/
Grossman, R.L., Hornick, M.F., & Meyer, G. (2002). Data
mining standards initiatives. Communications of the ACM,
45 (8), 59-61.
Han, J., & Kamber, M. (2001). Data mining: concepts and
techniques. Morgan Kaufmann Publishers.
Hornick, M. et al. (2004). JavaSpecification Request
73: JavaData Mining (JDM). Version 0.96. Retrieved
from http://jcp.org/aboutJava/communityprocess/first/
jsr073/
Melton, J., & Eisenberg, A. (2001). SQL Multimedia and
Application Packages (SQL/MM). SIGMOD Record, 30
(4), 97-102.
Melton, J., & Simon, A. (2001). SQL: 1999. Understanding
relational language components. Morgan Kaufmann
Publishers.
Microsoft Corporation. (2000). OLE DB for Data Mining
Specification Version 1.0.
Netz, A. et al. (2001, April). Integrating data mining with
SQL Databases: OLE DB for data mining. In Proceedings
of the 17 th International Conference on Data Engineer-
ing (ICDE 01) (pp. 379-387). Heidelberg, Germany.
Oracle9i Data Mining. Concepts. Release 9.2.0.2. (2002).
Viewable CD Release 2 (9.2.0.2.0).
PMML Version 2.1. (2003). Retrieved from http://
www.dmg.org/pmml-v2-1.html
Saarenvirta, G. (2001, Summer). Operation Data Mining.
DB2 Magazine, 6(2). International Business Machines
Corporation. Retrieved from http://www.db2mag.com/
db_area/archives/2001/q2/saarenvirta.shtml
42
API Standardization Efforts for Data Mining
SAS Enterprise Miner to support PMML. (September
17, 2002). Retrieved from http://www.sas.com/news/
preleases/091702/news1.html
Schwenkreis, F. (2001). Data mining Technology driven
by standards? Retrieved from http://www.research.
microsoft.com/~jamesrh/hpts2001/submissions/
FriedemannSchwenkreis.htm
SQL Multimedia and Application Packages. Part 6: Data
Mining. ISO/IEC 13249-6. (2002).
Zendulka, J. (2005). Using standard APIs for data mining
in prediction. In J. Wang (Ed.) Encyclopedia of data
warehousing and mining. Hershey, PA: Idea Group Ref-
erence.
KEY TERMS
API: Application programming interface (API) is a
description of the way one piece of software asks another
program to perform a service. A standard API for data
mining enables for different data mining algorithms from
various vendors to be easily plugged into application
programs.
Data Mining Model: A high-level global description
of a given set of data which is the result of a data mining
technique over the set of data. It can be descriptive or
predictive.
DMG: Data Mining Group (DMG) is a consortium of
data mining vendors for developing data mining stan-
dards. They have developed a Predictive Model Markup
language (PMML).
JDM: Java Data Mining (JDM) is an emerging stan-
dard API for the programming language Java. It is an
object-oriented interface that specifies a set of Java classes
and interfaces supporting data mining operations for
building, testing, and applying a data mining model.
OLE DB for DM: OLE DB for Data Mining (OLE DB for
DM) is a Microsofts language-based standard API that
introduces several SQL-like statements supporing data
mining operations for building, testing, and applying a
data mining model.
PMML: Predictive Model Markup Language (PMML)
is an XML-based language which provides a quick and
easy way for applications to produce data mining models
in a vendor-independent format and to share them be-
tween compliant applications.
TEAM LinG
API Standardization Efforts for Data Mining
SQL1999: Structured Query Language (SQL): 1999.
The version of the standard database language SQL
adapted in 1999, which introduced object-oriented fea-
tures.
SQL/MM DM: SQL Multimedia and Application Pack-
ages Part 6: Data Mining (SQL/MM DM) is an interna- A
tional standard the purpose of which is to define data
mining user-defined types and associated routines for
building, testing, and applying data mining models. It is
based on structured user-defined types of SQL:1999.
43
TEAM LinG
44
The Application of Data Mining to
Recommender Systems
J. Ben Schafer
University of Northern Iowa, USA
INTRODUCTION
In a world where the number of choices can be overwhelm-
ing, recommender systems help users find and evaluate
items of interest. They connect users with items to con-
sume (purchase, view, listen to, etc.) by associating the
content of recommended items or the opinions of other
individuals with the consuming users actions or opin-
ions. Such systems have become powerful tools in do-
mains from electronic commerce to digital libraries and
knowledge management. For example, a consumer of just
about any major online retailer who expresses an interest
in an item either through viewing a product description
or by placing the item in his shopping cart will likely
receive recommendations for additional products. These
products can be recommended based on the top overall
sellers on a site, on the demographics of the consumer, or
on an analysis of the past buying behavior of the con-
sumer as a prediction for future buying behavior. This
paper will address the technology used to generate rec-
ommendations, focusing on the application of data min-
ing techniques.
BACKGROUND
Many different algorithmic approaches have been ap-
plied to the basic problem of making accurate and efficient
recommender systems. The earliest recommender sys-
tems were content filtering systems designed to fight
information overload in textual domains. These were often
based on traditional information filtering and information
retrieval systems. Recommender systems that incorpo-
rate information retrieval methods are frequently used to
satisfy ephemeral needs (short-lived, often one-time
needs) from relatively static databases. For example, re-
questing a recommendation for a book preparing a sibling
for a new child in the family. Conversely, recommender
systems that incorporate information-filtering methods
are frequently used to satisfy persistent information (long-
lived, often frequent, and specific) needs from relatively
stable databases in domains with a rapid turnover or
frequent additions. For example, recommending AP sto-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
ries to a user concerning the latest news regarding a
senators re-election campaign.
Without computers, a person often receives recom-
mendations by listening to what people around him have
to say. If many people in the office state that they enjoyed
a particular movie, or if someone he tends to agree with
suggests a given book, then he may treat these as recom-
mendations. Collaborative filtering (CF) is an attempt to
facilitate this process of word of mouth. The simplest of
CF systems provide generalized recommendations by
aggregating the evaluations of the community at large.
More personalized systems (Resnick & Varian, 1997)
employ techniques such as user-to-user correlations or a
nearest-neighbor algorithm.
The application of user-to-user correlations derives
from statistics, where correlations between variables are
used to measure the usefulness of a model. In recommender
systems correlations are used to measure the extent of
agreement between two users (Breese, Heckerman, &
Kadie, 1998) and used to identify users whose ratings will
contain high predictive value for a given user. Care must
be taken, however, to identify correlations that are actu-
ally helpful. Users who have only one or two rated items
in common should not be treated as strongly correlated.
Herlocker et al. (1999) improved system accuracy by
applying a significance weight to the correlation based on
the number of co-rated items.
Nearest-neighbor algorithms compute the distance
between users based on their preference history. Dis-
tances vary greatly based on domain, number of users,
number of recommended items, and degree of co-rating
between users. Predictions of how much a user will like an
item are computed by taking the weighted average of the
opinions of a set of neighbors for that item. As applied in
recommender systems, neighbors are often generated
online on a query-by-query basis rather than through the
off-line construction of a more thorough model. As such,
they have the advantage of being able to rapidly incorpo-
rate the most up-to-date information, but the search for
neighbors is slow in large databases. Practical algorithms
use heuristics to search for good neighbors and may use
opportunistic sampling when faced with large popula-
tions.
TEAM LinG
The Application of Data Mining to Recommender Systems
Both nearest-neighbor and correlation-based
recommenders provide a high level of personalization in
their recommendations, and most early systems using
these techniques showed promising accuracy rates. As
such, CF-based systems have continued to be popular in
recommender applications and have provided the bench-
marks upon which more recent applications have been
compared.
DATA MINING IN RECOMMENDER
APPLICATIONS
The term data mining refers to a broad spectrum of math-
ematical modeling techniques and software tools that are
used to find patterns in data and user these to build
models. In this context of recommender applications, the
term data mining is used to describe the collection of
analysis techniques used to infer recommendation rules
or build recommendation models from large data sets.
Recommender systems that incorporate data mining tech-
niques make their recommendations using knowledge
learned from the actions and attributes of users. These
systems are often based on the development of user
profiles that can be persistent (based on demographic or
item consumption history data), ephemeral (based on
the actions during the current session), or both. These
algorithms include clustering, classification techniques,
the generation of association rules, and the production of
similarity graphs through techniques such as Horting.
Clustering techniques work by identifying groups of
consumers who appear to have similar preferences. Once
the clusters are created, averaging the opinions of the
other consumers in her cluster can be used to make
predictions for an individual. Some clustering techniques
represent each user with partial participation in several
clusters. The prediction is then an average across the
clusters, weighted by degree of participation. Clustering
techniques usually produce less-personal recommenda-
tions than other methods, and in some cases, the clusters
have worse accuracy than CF-based algorithms (Breese,
Heckerman, & Kadie, 1998). Once the clustering is com-
plete, however, performance can be very good, since the
size of the group that must be analyzed is much smaller.
Clustering techniques can also be applied as a first step
for shrinking the candidate set in a CF-based algorithm or
for distributing neighbor computations across several
recommender engines. While dividing the population into
clusters may hurt the accuracy of recommendations to
users near the fringes of their assigned cluster, pre-
clustering may be a worthwhile trade-off between accu-
racy and throughput.
Classifiers are general computational models for as-
signing a category to an input. The inputs may be vectors
of features for the items being classified or data about
relationships among the items. The category is a domain- A
specific classification such as malignant/benign for tumor
classification, approve/reject for credit requests, or in-
truder/authorized for security checks. One way to build a
recommender system using a classifier is to use informa-
tion about a product and a customer as the input, and to
have the output category represent how strongly to
recommend the product to the customer. Classifiers may
be implemented using many different machine-learning
strategies including rule induction, neural networks, and
Bayesian networks. In each case, the classifier is trained
using a training set in which ground truth classifications
are available. It can then be applied to classify new items
for which the ground truths are not available. If subse-
quent ground truths become available, the classifier may
be retrained over time.
For example, Bayesian networks create a model based
on a training set with a decision tree at each node and
edges representing user information. The model can be
built off-line over a matter of hours or days. The resulting
model is very small, very fast, and essentially as accurate
as CF methods (Breese, Heckerman, & Kadie, 1998). Baye-
sian networks may prove practical for environments in
which knowledge of consumer preferences changes slowly
with respect to the time needed to build the model but are
not suitable for environments in which consumer prefer-
ence models must be updated rapidly or frequently.
Classifiers have been quite successful in a variety of
domains ranging from the identification of fraud and
credit risks in financial transactions to medical diagnosis
to intrusion detection. Good et al. (1999) implemented
induction-learned feature-vector classification of movies
and compared the classification with CF recommenda-
tions; this study found that the classifiers did not perform
as well as CF, but that combining the two added value over
CF alone.
One of the best-known examples of data mining in
recommender systems is the discovery of association
rules, or item-to-item correlations (Sarwar et. al., 2001).
These techniques identify items frequently found in as-
sociation with items in which a user has expressed
interest. Association may be based on co-purchase data,
preference by common users, or other measures. In its
simplest implementation, item-to-item correlation can be
used to identify matching items for a single item, such
as other clothing items that are commonly purchased with
a pair of pants. More powerful systems match an entire set
of items, such as those in a customers shopping cart, to
identify appropriate items to recommend. These rules can
also help a merchandiser arrange products so that, for
example, a consumer purchasing a childs handheld video
game sees batteries nearby. More sophisticated temporal
data mining may suggest that a consumer who buys the
45
TEAM LinG
 The Application of Data Mining to Recommender Systems
video game today is likely to buy a pair of earplugs in the
next month.
Item-to-item correlation recommender applications
usually use current interest rather than long-term cus-
tomer history, which makes them particularly well suited
for ephemeral needs such as recommending gifts or locat-
ing documents on a topic of short lived interest. A user
merely needs to identify one or more starter items to elicit
recommendations tailored to the present rather than the
past.
Association rules have been used for many years in
merchandising, both to analyze patterns of preference
across products, and to recommend products to consum-
ers based on other products they have selected. An asso-
ciation rule expresses the relationship that one product is
often purchased along with other products. The number of
possible association rules grows exponentially with the
number of products in a rule, but constraints on confi-
dence and support, combined with algorithms that build
association rules with itemsets of n items from rules with
n-1 item itemsets, reduce the effective search space. As-
sociation rules can form a very compact representation of
preference data that may improve efficiency of storage as
well as performance. They are more commonly used for
larger populations rather than for individual consumers,
and they, like other learning methods that first build and
then apply models, are less suitable for applications where
knowledge of preferences changes rapidly. Association
rules have been particularly successfully in broad applica-
tions such as shelf layout in retail stores. By contrast,
recommender systems based on CF techniques are easier
to implement for personal recommendation in a domain
where consumer opinions are frequently added, such as
online retail.
In addition to use in commerce, association rules have
become powerful tools in recommendation applications in
the domain of knowledge management. Such systems
attempt to predict which Web page or document can be
most useful to a user. As Gry (2003) writes, The problem
of finding Web pages visited together is similar to finding
associations among itemsets in transaction databases.
Once transactions have been identified, each of them
could represent a basket, and each web resource an item.
Systems built on this approach have been demonstrated to
produce both high accuracy and precision in the coverage
of documents recommended (Geyer-Schultz et al., 2002).
Horting is a graph-based technique in which nodes are
users, and edges between nodes indicate degree of simi-
larity between two users (Wolf et al., 1999). Predictions are
produced by walking the graph to nearby nodes and
combining the opinions of the nearby users. Horting dif-
fers from collaborative filtering as the graph may be walked
through other consumers who have not rated the product
in question, thus exploring transitive relationships that
46
traditional CF algorithms do not consider. In one study
using synthetic data, Horting produced better predic-
tions than a CF-based algorithm (Wolf et al., 1999).
FUTURE TRENDS
As data mining algorithms have been tested and vali-
dated in their application to recommender systems, a
variety of promising applications have evolved. In this
section we will consider three of these applications
meta-recommenders, social data mining systems, and
temporal systems that recommend when rather than what.
Meta-recommenders are systems that allow users to
personalize the merging of recommendations from a va-
riety of recommendation sources employing any number
of recommendation techniques. In doing so, these sys-
tems let users take advantage of the strengths of each
different recommendation method. The SmartPad super-
market product recommender system (Lawrence et al.,
2001) suggests new or previously unpurchased prod-
ucts to shoppers creating shopping lists on a personal
digital assistant (PDA). The SmartPad system considers
a consumers purchases across a stores product tax-
onomy. Recommendations of product subclasses are
based upon a combination of class and subclass associa-
tions drawn from information filtering and co-purchase
rules drawn from data mining. Product rankings within a
product subclass are based upon the products sales
rankings within the users consumer cluster, a less per-
sonalized variation of collaborative filtering. MetaLens
(Schafer et al., 2002) allows users to blend content re-
quirements with personality profiles to allow users to
determine which movie they should see. It does so by
merging more persistent and personalized recommenda-
tions, with ephemeral content needs such as the lack of
offensive content or the need to be home by a certain
time. More importantly, it allows the user to customize
the process by weighting the importance of each indi-
vidual recommendation.
While a traditional CF-based recommender typically
requires users to provide explicit feedback, a social data
mining system attempts to mine the social activity records
of a community of users to implicitly extract the impor-
tance of individuals and documents. Such activity may
include Usenet messages, system usage history, cita-
tions, or hyperlinks. TopicShop (Amento et al., 2003) is
an information workspace which allows groups of com-
mon Web sites to be explored, organized into user de-
fined collections, manipulated to extract and order com-
mon features, and annotated by one or more users. These
actions on their own may not be of large interest, but the
collection of these actions can be mined by TopicShop
and redistributed to other users to suggest sites of
TEAM LinG
The Application of Data Mining to Recommender Systems
general and personal interest. Agrawal et al. (2003) ex-
plored the threads of newsgroups to identify the relation-
ships between community members. Interestingly, they
concluded that due to the nature of newsgroup postings
users are more likely to respond to those with whom they
disagree links between users are more likely to sug-
gest that users should be placed in differing partitions
rather than the same partition. Although this technique
has not been directly applied to the construction of
recommendations, such an application seems a logical
field of future study.
Although traditional recommenders suggest what item
a user should consume they have tended to ignore changes
over time. Temporal recommenders apply data mining
techniques to suggest when a recommendation should be
made or when a user should consume an item. Adomavicius
and Tuzhilin (2001) suggest the construction of a recom-
mendation warehouse, which stores ratings in a hypercube.
This multidimensional structure can store data on not
only the traditional user and item axes, but also for
additional profile dimensions such as time. Through this
approach, queries can be expanded from the traditional
what items should we suggest to user X to at what
times would user X be most receptive to recommendations
for product Y. Hamlet (Etzioni et al., 2003) is designed
to minimize the purchase price of airplane tickets. Hamlet
combines the results from time series analysis, Q-learn-
ing, and the Ripper algorithm to create a multi-strategy
data-mining algorithm. By watching for trends in airline
pricing and suggesting when a ticket should be pur-
chased, Hamlet was able to save the average user 23.8%
when savings was possible.
CONCLUSION
Recommender systems have emerged as powerful tools
for helping users find and evaluate items of interest.
These systems use a variety of techniques to help users
identify the items that best fit their tastes or needs. While
popular CF-based algorithms continue to produce mean-
ingful, personalized results in a variety of domains, data
mining techniques are increasingly being used in both
hybrid systems, to improve recommendations in previ-
ously successful applications, and in stand-alone
recommenders, to produce accurate recommendations in
previously challenging domains. The use of data mining
algorithms has also changed the types of recommenda-
tions as applications move from recommending what to
consume to also recommending when to consume. While
recommender systems may have started as largely a
passing novelty, they clearly appear to have moved into
a real and powerful tool in a variety of applications, and
that data mining algorithms can be and will continue to be
an important part of the recommendation process. A
REFERENCES
Adomavicius, G., & Tuzhilin, A. (2001). Extending
recommender systems: A multidimensional approach.
IJCAI-01 Workshop on Intelligent Techniques for Web
Personalization (ITWP2001), Seattle, Washington.
Agrawal, R., Rajagopalan, S., Srikant, R., & Xu, Y. (2003).
Mining newsgroups using networks arising from social
behavior. In Proceedings of the Twelfth World Wide
Web Conference (WWW12) (pp. 529-535), Budapest, Hun-
gary.
Amento, B., Terveen, L., Hill, W., Hix, D., & Schulman, R.
(2003). Experiments in social data mining: The TopicShop
System. ACM Transactions on Computer-Human Inter-
action, 10 (1), 54-85.
Breese, J., Heckerman, D., & Kadie, C. (1998). Empirical
analysis of predictive algorithms for collaborative filter-
ing. In Proceedings of the 14th Conference on Uncer-
tainty in Artificial Intelligence (UAI-98) (pp. 43-52),
Madison, Wisconsin.
Etzioni, O., Knoblock, C.A., Tuchinda, R., & Yates, A.
(2003). To buy or not to buy: Mining airfare data to
minimize ticket purchase price. In Proceedings of the
Ninth ACM SIGKDD International Conference on Knowl-
edge Discovery and Data Mining (pp. 119-128), Wash-
ington. D.C.
Gry, M., & Haddad, H. (2003). Evaluation of Web usage
mining approaches for users next request prediction. In
Fifth International Workshop on Web Information and
Data Management (pp. 74-81), Madison, Wisconsin.
Geyer-Schulz, A., & Hahsler, M. (2002). Evaluation of
recommender algorithms for an Internet information bro-
ker based on simple association rules and on the repeat-
buying theory. In Fourth WEBKDD Workshop: Web Min-
ing for Usage Patterns & User Profiles (pp. 100-114),
Edmonton, Alberta, Canada.
Good, N. et al. (1999). Combining collaborative filtering
with personal agents for better recommendations. In Pro-
ceedings of Sixteenth National Conference on Artificial
Intelligence (AAAI-99) (pp. 439-446), Orlando, Florida.
Herlocker, J., Konstan, J.A., Borchers, A., & Riedl, J.
(1999). An algorithmic framework for performing collabo-
rative filtering. In Proceedings of the 1999 Conference on
Research and Development in Information Retrieval,
(pp. 230-237), Berkeley, California.
47
TEAM LinG
 The Application of Data Mining to Recommender Systems
Lawrence, R.D. et al. (2001). Personalization of supermar-
ket product recommendations. Data Mining and Knowl-
edge Discovery, 5(1/2), 11-32.
Lin, W., Alvarez, S.A., & Ruiz, C. (2002). Efficient adap-
tive-support association rule mining for recommender
systems. Data Mining and Knowledge Discovery, 6(1)
83-105.
Resnick, P., & Varian, H.R. (1997). Communications of the
Association of Computing Machinery Special issue on
Recommender Systems, 40(3), 56-89.
Sarwar, B., Karypis, G., Konstan, J.A., & Reidl, J. (2001).
Item-based collaborative filtering recommendation algo-
rithms. In Proceedings of the Tenth International Con-
ference on World Wide Web (pp. 285-295), Hong Kong.
Schafer, J.B., Konstan, J.A., & Riedl, J. (2001). E-Com-
merce Recommendation Applications. Data Mining and
Knowledge Discovery, 5(1/2), 115-153.
Schafer, J.B., Konstan, J.A., & Riedl, J. (2002). Meta-
recommendation systems: User-controlled integration of
diverse recommendations. In Proceedings of the Elev-
enth Conference on Information and Knowledge (CIKM-
02) (pp. 196-203), McLean, Virginia.
Shoemaker, C., & Ruiz, C. (2003). Association rule mining
algorithms for set-valued data. Lecture Notes in Com-
puter Science, 2690, 669-676.
Wolf, J., Aggarwal, C., Wu, K-L., & Yu, P. (1999). Horting
hatches an egg: A new graph-theoretic approach to col-
laborative filtering. In Proceedings of ACM SIGKDD
International Conference on Knowledge Discovery &
Data Mining (pp. 201-212), San Diego, CA.
48
KEY TERMS
Association Rules: Used to associate items in a data-
base sharing some relationship (e.g., co-purchase infor-
mation). Often takes the for if this, then that, such as,
If the customer buys a handheld videogame then the
customer is likely to purchase batteries.
Collaborative Filtering: Selecting content based on
the preferences of people with similar interests.
Meta-Recommenders: Provide users with personal-
ized control over the generation of a single recommenda-
tion list formed from the combination of rich recommenda-
tion data from multiple information sources and recom-
mendation techniques.
Nearest-Neighbor Algorithm: A recommendation
algorithm that calculates the distance between users
based on the degree of correlations between scores in the
users preference histories. Predictions of how much a
user will like an item are computed by taking the weighted
average of the opinions of a set of nearest neighbors for
that item.
Recommender Systems: Any system that provides a
recommendation, prediction, opinion, or user-configured
list of items that assists the user in evaluating items.
Social Data-Mining: Analysis and redistribution of
information from records of social activity such as
newsgroup postings, hyperlinks, or system usage his-
tory.
Temporal Recommenders: Recommenders that incor-
porate time into the recommendation process. Time can be
either an input to the recommendation function, or the
output of the function.
TEAM LinG

Approximate Range Queries by Histograms
in OLAP
Francesco Buccafurri
University Mediterranea of Reggio Calabria, Italy
Gianluca Lax
University Mediterranea of Reggio Calabria, Italy
INTRODUCTION
Online analytical processing applications typically ana-
lyze a large amount of data by means of repetitive
queries involving aggregate measures on such data. In
fast OLAP applications, it is often advantageous to
provide approximate answers to queries in order to
achieve very high performances. A way to obtain this
goal is by submitting queries on compressed data in
place of the original ones. Histograms, initially intro-
duced in the field of query optimization, represent one
of the most important techniques used in the context of
OLAP for producing approximate query answers.
BACKGROUND
Computing aggregate information is a widely exploited
task in many OLAP applications. Every time it is neces-
sary to produce fast query answers and a certain estima-
tion error can be accepted, it is possible to inquire
summary data rather than the original ones and to per-
form suitable interpolations. The typical OLAP query is
the range query.
The range query estimation problem in the one-
dimensional case can be stated as follows: given an
attribute X of a relation R, and a range I belonging to the
domain of X, estimate the number of records of R with
value of X lying in I. The challenge is finding methods
for achieving a small estimation error by consuming a
fixed amount of storage space.
A possible solution to this problem is using sampling
methods; only a small number of suitably selected records
of R, representing R well, are stored. The range query is
then evaluated by exploiting this sample instead of the
full relation R. Recently, Wu, Agrawal, and Abbadi (2002)
have shown that in terms of accuracy, sampling tech-
niques based on the cumulative distribution function are
definitely better than the methods based on tuple sam-
pling (Chaudhuri, Das & Narasayya, 2001; Ganti, Lee &
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
49
A
Ramakrishnan, 2000). The main advantage of sampling
techniques is that they are very easy to implement.
Besides sampling, regression techniques try to model
data as a function in such a way that only a small set of
coefficients representing such a function is stored,
rather than the original data. The simplest regression
technique is the linear one, which models a data distri-
bution as a linear function. Despite its simplicity, not
allowing the capture of complex relationships among
data, this technique often produces acceptable results.
There are also non linear regressions, significantly
more complex than the linear one from the computa-
tional point of view, but applicable to a much larger set
of cases.
Another possibility for facing the range query esti-
mation problem consists of using wavelets-based tech-
niques (Chakrabarti, Garofalakis, Rastogi & Shim, 2001;
Garofalakis & Gibbons, 2002; Garofalakis & Kumar,
2004). Wavelets are mathematical transformations stor-
ing data in a compact and hierarchical fashion used in
many application contexts, like image and signal pro-
cessing (Kacha, Grenez, De Doncker & Benmahammed,
2003; Khalifa, 2003). There are several types of trans-
formations, each belonging to a family of wavelets. The
result of each transformation is a set of values, called
wavelet coefficients. The advantage of this technique is
that, typically, the value of a (possibly large) number of
wavelet coefficients results to be below a fixed thresh-
old, so that such coefficients can be approximated by 0.
Clearly, the overall approximation of the technique as
well as the compression ratio depends on the value of
such a threshold. In the last years, wavelets have been
exploited in data mining and knowledge discovery in
databases, thanks to time and space efficiency and data
hierarchical decomposition characterizing them. For a
deeper treatment about wavelets, see Li, Li, Zhu, and
Ogihara (2002).
Besides sampling and wavelets, histograms are used
widely for estimating range queries. Although some-
times wavelets are viewed as a particular class of histo-
grams, we prefer to describe histograms separately.
TEAM LinG

MAIN THRUST
Histograms are a lossy compression technique widely
applied in various application contexts, like query opti-
mization, statistical and temporal databases, and OLAP
applications. In OLAP, compression allows us to obtain
fast approximate answers by evaluating queries on re-
duced data in place of the original ones. Histograms are
well suited to this purpose, especially in the case of
range queries.
A histogram is a compact representation of a rela-
tion R. It is obtained by partitioning an attribute X of the
relation R into k subranges, called buckets, and by
maintaining for each of them a few pieces of informa-
tion, typically corresponding to the bucket boundaries,
the number of tuples with value of X belonging to the
subrange associated to the bucket (often called sum of
the bucket), and the number of distinct values of X of
such a subrange occurring in some tuple of R (i.e., the
number of non-null frequencies of the subrange).
Recall that a range query, defined on an interval I of
X, evaluates the number of occurrences in R with value
of X in I. Thus, buckets embed a set of precomputed
disjoint range queries capable of covering the whole
active domain of X in R (here, active means attribute
values actually appearing in R). As a consequence, the
histogram, in general, does not give the possibility of
evaluating exactly a range query not corresponding to
one of the precomputed embedded queries. In other
words, while the contribution to the answer coming
from the subranges coinciding with entire buckets can
be returned exactly, the contribution coming from the
subranges that partially overlap buckets can only be
estimated, since the actual data distribution is not avail-
able.
Constructing the best histogram thus may mean de-
fining the boundaries of buckets in such a way that the
estimation of the non-precomputed range queries be-
comes more effective (e.g., by avoiding that large fre-
quency differences arise inside a bucket). This approach
corresponds to finding, among all possible sets of pre-
computed range queries, the set that guarantees the best
estimation of the other (non-precomputed) queries,
once a technique for estimating such queries is defined.
Besides this problem, which we call the partition
problem, there is another relevant issue to investigate:
how to improve the estimation inside the buckets. We
discuss both of these issues in the following two sections.
The Partition Problem
This issue has been analyzed widely in the past, and a
number of techniques has been proposed. Among these,
50
Approximate Range Queries by Histograms in OLAP
we first consider the Max-Diff histogram and the V-
Optimal histogram. Even though they are not the most
recent techniques, we cite them, since they are still
considered points of reference.
We start by describing the Max-Diff histogram.
Let V={v1, ... , v n}be the set of values of the attribute
X actually appearing in the relation R and f(v i) be the
number of tuples of R having value v i in X. A MaxDiff
histogram with h buckets is obtained by putting a bound-
ary between two adjacent attribute values v i and vi+1 of V
if the difference between f(vi+1) si+1 and f(vi) s i is one
of the h-1 largest such differences (where s i denotes the
spread of vi, that is the distance from vi to the next non-
null value).
A V-Optimal histogram, which is the other classical
histogram we describe, produces more precise results
than the Max-Diff histogram. It is obtained by selecting
the boundaries for each bucket i so that i SSEi is
minimal, where SSE i is the standard squared error of the
bucket i-th.
V-Optimal histogram uses a dynamic programming
technique in order to find the optimal partitioning w.r.t.
a given error metrics. Even though the V-Optimal histo-
gram results more accurate than Max-Diff, its high
space and time complexities make it rarely used in
practice.
In order to overcome such a drawback, an approxi-
mate version of the V-Optimal histogram has been pro-
posed. The basic idea is quite simple. First, data are
partitioned into l disjoint chunks, and then the V-Opti-
mal algorithm is used in order to compute a histogram
within each chunk. The consequent problem is how to
allocate buckets to the chunks such that exactly B buck-
ets are used. This is solved by implementing a dynamic
programming scheme. It is shown that an approximate
V-Optimal histogram with B + l buckets has the same
accuracy as the non-approximate V-Optimal with B buck-
ets. Moreover, the time required for executing the
approximate algorithm is reduced by multiplicative fac-
tor equal to1/l.
We call the histograms so far described classical
histograms.
Besides accuracy, new histograms tend to satisfy
other properties in order to allow their application to
new environments (e.g., knowledge discovery). In par-
ticular, (1) the histogram should maintain in a certain
measure the semantic nature of original data, in such a
way that meaningful queries for mining activities can be
submitted to reduced data in place of original ones.
Then, (2) for a given kind of query, the accuracy of the
reduced structure should be guaranteed. In addition, (3)
the histogram should efficiently support hierarchical
range queries in order not to limit too much the capabil-
ity of drilling down and rolling up over data.
TEAM LinG
Approximate Range Queries by Histograms in OLAP
Classical histograms lack the last point, since they
are flat structures. Many proposals have been presented
in order to guarantee the three properties previously
described, and we report some of them in the following.
Requirement (3) was introduced by Koudas,
Muthukrishnan, and Srivastava (2000), where the authors
have shown the insufficient accuracy of classical histo-
grams in evaluating hierarchical range queries. Therein, a
polynomial-time algorithm for constructing optimal his-
tograms with respect to hierarchical queries is proposed.
The selectivity estimation problem for non-hierarchi-
cal range queries was studied by Gilbert, Kotidis,
Muthukrishnan, and Strauss (2001), and, according to
property (2), optimal and approximate polynomial (in the
database size) algorithms with a provable approximation
guarantee for constructing histograms are also presented.
Guha, Koudas, and Srivastava (2002) have proposed
efficient algorithms for the problem of approximating
the distribution of measure attributes organized into
hierarchies. Such algorithms are based on dynamic pro-
gramming and on a notion of sparse intervals.
Algorithms returning both optimal and suboptimal
solutions for approximating range queries by histograms
and their dynamic maintenance by additive changes are
provided by Muthukrishnan and Strauss (2003). The best
algorithm, with respect to the construction time, return-
ing an optimal solution takes polynomial time.
Buccafurri and Lax (2003) have presented a histo-
gram based on a hierarchical decomposition of the data
distribution kept in a full binary tree. Such a tree, con-
taining a set of precomputed hierarchical queries, is
encoded by using bit saving for obtaining a smaller
structure and, thus, for efficiently supporting hierarchi-
cal range queries.
Besides bucket-based histograms, there are other
kinds of histograms whose construction is not driven by
the search of a suitable partition of the attribute domain,
and, further, their structure is more complex than simply
a set of buckets. This class of histograms is called non-
bucket based histograms. Wavelets are an example of
such kind of histograms.
In the next section, we deal with the second problem
introduced earlier concerning the estimation of range
queries partially involving buckets.
Estimation Inside a Bucket
While finding the optimal bucket partition has been
widely investigated in past years, the problem of estimat-
ing queries partially involving a bucket has received a
little attention.
Histograms are well suited to range query evaluation,
since buckets basically correspond to a set of precom-
puted range queries. A range query that involves entirely
one or more buckets can be computed exactly, while if it
partially overlaps a bucket, then the result only can be A
estimated.
The simplest adopted estimation technique is the
Continuous Value Assumption (CVA). Given a bucket
of size s and sum c, a range query overlapping the bucket
in i points is estimated as (i / s ) c . This corresponds to
estimating the partial contribution of the bucket to the
range query result by linear interpolation.
Another possibility is to use the Uniform Spread
Assumption (USA). It assumes that values are distrib-
uted at equal distance from each other and that the
overall frequency sum is equally distributed among
them. In this case, it is necessary to know the number of
non-null frequencies belonging to the bucket. Denoting
by t such a value, the range query is estimated by
(s 1) + (i 1) (t 1) c
.
( s 1) t
An interesting problem is understanding whether,
by exploiting information typically contained in histo-
gram buckets and possibly by adding some concise
summary information, the frequency estimation inside
buckets and, then, the histogram accuracy can be im-
proved. To this aim, starting from a theoretical analysis
about limits of CVA and USA, Buccafurri, Pontieri,
Rosaci, and Sacc (2002) have proposed to use an addi-
tional storage space of 32 bits, called 4LT, in each
bucket in order to store the approximate representation
of the data distribution inside the bucket. In particular, 4LT
is used to save approximate cumulative frequencies at
seven equidistant intervals internal to the bucket.
Clearly, approaches similar to that followed in
Buccafurri, Pontieri, Rosaci, and Sacc (2002) have to
deal with the trade-off between the extra storage space
required for each bucket and the number of total buck-
ets the allowed total storage space consents.
FUTURE TRENDS
Data streams is an emergent issue that in the last two
years has captured the interest of many scientific com-
munities. The crucial problem arising in several appli-
cation contexts like network monitoring, sensor net-
works, financial applications, security, telecommuni-
cation data management, Web applications, and so on is
dealing with continuous data flows (i.e., data streams)
having the following characteristics: (1) they are time
dependent; (2) their size is very large, so that they
cannot be stored totally due to the actual memory
51
TEAM LinG

limitation; and (3) data arrival is very fast and unpredict-
able, so that each data management operation should be
very efficient.
Since a data stream consists of a large amount of
data, it is usually managed on the basis of a sliding
window, including only the most recent data (Babcock,
Babu, Datar, Motwani & Widom, 2002). Thus, any tech-
nique capable of compressing sliding windows by main-
taining a good approximate representation of data dis-
tribution is certainly relevant in this field. Typical que-
ries performed on sliding windows are similarity que-
ries and other analyses, like change mining queries
(Dong, Han, Lakshmanan, Pei, Wang & Yu, 2003) useful
for trend analysis and, in general, for understanding the
dynamics of data. Also in this field, histograms may
become an important analysis tool. The challenge is
finding new histograms that (1) are fast to construct and
to maintain; that is, the required updating operations
(performed at each data arrival) are very efficient; (2)
maintain a good accuracy in approximating data distri-
bution; and (3) support continuous querying on data.
An example of the above emerging approaches is
reported in Buccafurri and Lax (2004), where a tree-like
histogram with cyclic updating is proposed. By using
such a compact structure, many mining techniques, which
would take computational cost very high if used on real
data streams, can be implemented effectively.
CONCLUSION
Data reduction represents an important task both in data
mining task and in OLAP, since it allows us to represent
very large amounts of data in a compact structure, which
efficiently perform on mining techniques or OLAP
queries. Time and memory cost advantages arisen from
data compression, provided that a sufficient degree of
accuracy is guaranteed, may improve considerably the
capabilities of mining and OLAP tools.
This opportunity (added to the necessity, coming
from emergent research fields such as data streams) of
producing more and more compact representations of
data explains the attention that the research community
is giving toward techniques like histograms and wave-
lets, which provide a concrete answer to the previous
requirements.
REFERENCES
Babcock, B., Babu, S., Datar, M., Motwani, R., & Widom,
J. (2002). Models and issues in data stream system.
Proceedings of the ACM SIGMOD-SIGACT-SIGART
Symposium on Principles of Database Systems.
52
Approximate Range Queries by Histograms in OLAP
Buccafurri, F., & Lax, G. (2003). Pre-computing approxi-
mate hierarchical range queries in a tree-like histogram.
Proceedings. of the International Conference on Data
Warehousing and Knowledge Discovery.
Buccafurri, F., & Lax, G. (2004). Reducing data stream
sliding windows by cyclic tree-like histograms. Pro-
ceedings of the 8th European Conference on Principles
and Practice of Knowledge Discovery in Databases.
Buccafurri, F., Pontieri, L., Rosaci, D., & Sacc, D.
(2002). Improving range query estimation on histo-
grams. Proceedings of the International Conference
on Data Engineering.
Chakrabarti, K., Garofalakis, M., Rastogi, R., & Shim,
K. (2001). Approximate query processing using wave-
lets. VLDB Journal, The International Journal on
Very Large Data Bases, 10(2-3), 199-223.
Chaudhuri, S., Das, G., & Narasayya, V. (2001). A ro-
bust, optimization-based approach for approximate an-
swering of aggregate queries. Proceedings of the 2001
ACM SIGMOD International Conference on Manage-
ment of Data.
Dong, G. et al. (2003). Online mining of changes from data
streams: Research problems and preliminary results. Pro-
ceedings of the ACM SIGMOD Workshop on Manage-
ment and Processing of Data Streams.
Ganti, V., Lee, M. L., & Ramakrishnan, R. (2000).
Icicles: Self-tuning samples for approximate query an-
swering. Proceedings of 26th International Confer-
ence on Very Large Data Bases.
Garofalakis, M., & Gibbons, P.B. (2002). Wavelet syn-
opses with error guarantees. Proceedings of the ACM
SIGMOD International Conference on Management
of Data.
Garofalakis, M., & Kumar, A. (2004). Deterministic wave-
let thresholding for maximum error metrics. Proceedings
of the Twenty-third ACM SIGMOD-SIGACT-SIGART Sym-
posium on Principles of Database Systems.
Gilbert, A. C., Kotidis, Y., Muthukrishnan, S., & Strauss,
M.J. (2001). Optimal and approximate computation of
summary statistics for range aggregates. Proceedings
of the Twentieth ACM SIGMOD-SIGACT-SIGART Sym-
posium on Principles of Database Systems.
Guha, S., Koudas, N., & Srivastava, D. (2002). Fast
algorithms for hierarchical range histogram construc-
tion. Proceedings of the Twenty-Ffirst ACM SIGMOD-
SIGACT-SIGART Symposium on Principles of Data-
base Systems.
TEAM LinG
Approximate Range Queries by Histograms in OLAP
Kacha, A., Grenez, F., De Doncker, P., & Benmahammed, K.
(2003). A wavelet-based approach for frequency estima-
tion of interference signals in printed circuit boards. Pro-
ceedings of the 1st International Symposium on Informa-
tion and Communication Technologies.
Khalifa, O. (2003). Image data compression in wavelet
transform domain using modified LBG algorithm. Pro-
ceedings of the 1st International Symposium on Infor-
mation and Communication Technologies.
Koudas, N., Muthukrishnan, S., & Srivastava, D. (2000).
Optimal histograms for hierarchical range queries (ex-
tended abstract). Proceedings of the Nineteenth ACM
SIGMOD-SIGACT-SIGART Symposium on Principles
of Database Systems.
Li, T., Li, Q., Zhu, S., & Ogihara, M. (2002). Survey on
wavelet applications in data mining. ACM SIGKDD Ex-
plorations, 4(2), 49-68.
Muthukrishnan, S., & Strauss, M. (2003). Rangesum
histograms. Proceedings of the Fourteenth Annual
ACM-SIAM Symposium on Discrete Algorithms.
Wu, Y., Agrawal, D., & Abbadi, A.E. (2002). Query
estimation by adaptive sampling. Proceedings of the
International Conference on Data Engineering.
KEY TERMS
Bucket: An element obtained by partitioning the
domain of an attribute X of a relation into non-overlap-
ping intervals. Each bucket consists of a tuple <inf, sup,
val>, where val is an aggregate information (i.e., sum,
average, count, etc.) about tuples with value of X be-
longing to the interval (inf, sup).
Bucket-Based Histogram: A type of histogram whose
construction is driven by the search of a suitable partition A
of the attribute domain into buckets.
Continuous Value Assumption (CVA): A tech-
nique allowing us to estimate values inside a bucket by
linear interpolation.
Data Preprocessing: The application of several
methods preceding the mining phase, done for improv-
ing the overall data mining results. Usually, it consists
of (1) data cleaning, a method for fixing missing values,
outliers, and possible inconsistent data; (2) data inte-
gration, the union of (possibly heterogeneous) data
coming from different sources into a unique data store;
and (3) data reduction, the application of any technique
working on data representation capable of saving stor-
age space without compromising the possibility of in-
quiring them.
Histogram: A set of buckets implementing a parti-
tion of the overall domain of a relation attribute.
Range Query: A query returning an aggregate infor-
mation (i.e., sum, average) about data belonging to a
given interval of the domain.
Uniform Spread Assumption (USA): A technique
for estimating values inside a bucket by assuming that
values are distributed at an equal distance from each
other and that the overall frequency sum is distributed
equally among them.
Wavelets: Mathematical transformations imple-
menting hierarchical decomposition of functions lead-
ing to the representation of functions through sets of
wavelet coefficients.
53
TEAM LinG
54
Artificial Neural Networks for Prediction
Rafael Mart
Universitat de Valncia, Spain
INTRODUCTION
The design and implementation of intelligent systems
with human capabilities is the starting point to design
Artificial Neural Networks (ANNs). The original idea
takes after neuroscience theory on how neurons in the
human brain cooperate to learn from a set of input
signals to produce an answer. Because the power of the
brain comes from the number of neurons and the mul-
tiple connections between them, the basic idea is that
connecting a large number of simple elements in a
specific way can form an intelligent system.
Generally speaking, an ANN is a network of many
simple processors called units, linked to certain neigh-
bors with varying coefficients of connectivity (called
weights) that represent the strength of these connec-
tions. The basic unit of ANNs, called an artificial neu-
ron, simulates the basic functions of natural neurons: it
receives inputs, processes them by simple combination
and threshold operations, and outputs a final result.
ANNs often employ supervised learning in which
training data (including both the input and the desired
output) is provided. Learning basically refers to the
process of adjusting the weights to optimize the net-
work performance. ANNs belongs to machine-learning
algorithms because the changing of a networks connec-
tion weights causes it to gain knowledge in order to
solve the problem at hand.
Neural networks have been widely used for both
classification and prediction. In this article, I focus on
the prediction or estimation problem (although with
some few changes, my comments and descriptions also
apply to classification). Estimating and forecasting fu-
ture conditions are involved in different business activi-
ties. Some examples include cost estimation, predic-
tion of product demand, and financial planning. More-
over, the field of prediction also covers other activities,
such as medical diagnosis or industrial process model-
ing.
In this short article I focus on the multilayer neural
networks because they are the most common. I describe
their architecture and some of the most popular training
methods. Then I finish with some associated conclu-
sions and the appropriate list of references to provide
some pointers for further study.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
From a technical point of view, ANNs offer a general
framework for representing nonlinear mappings from
several input variables to several output variables. They
are built by tuning a set of parameters known as weights
and can be considered as an extension of the many
conventional mapping techniques. In classification or
recognition problems, the nets outputs are categories,
while in prediction or approximation problems, they are
continuous variables. Although this article focuses on
the prediction problem, most of the key issues in the net
functionality are common to both.
In the process of training the net (supervised learn-
ing), the problem is to find the values of the weights w
that minimize the error across a set of input/output pairs
(patterns) called the training set E. For a single output
and input vector x, the error measure is typically the
root mean squared difference between the predicted
output p(x,w) and the actual output value f(x) for all the
elements x in E (RMSE); therefore, the training is an
unconstrained nonlinear optimization problem, where
the decision variables are the weights, and the objective
is to reduce the training error. Ideally, the set E is a
representative sample of points in the domain of the
function f that you are approximating; however, in prac-
tice it is usually a set of points for which you know the
f-value.
( f ( x) p( x, w)) 2
Min error ( E , w) = xE
(1)
w E
The main goal in the design of an ANN is to obtain a
model that makes good predictions for new inputs (i.e.,
to provide good generalization). Therefore, the net must
represent the systematic aspects of the training data
rather than their specific details. The standard way to
measure the generalization provided by the net consists
of introducing a second set of points in the domain of f
called the testing set, T. Assume that no point in T
belongs to E and f(x) is known for all x in T. After the
optimization has been performed and the weights have
been set to minimize the error in E (w=w*), the error
across the testing set T is computed (error(T,w*)). The
TEAM LinG
Artificial Neural Networks for Prediction
net must exhibit a good fit between the target f-values and
the output (prediction) in the training set and also in the
testing set. If the RMSE in T is significantly higher than
that one in E, you say that the net has memorized the data
instead of learning them (i.e., the net has overfitted the
training data).
The optimization of the function given in (1) is a hard
problem by itself. Moreover, keep in mind that the final
objective is to obtain a set of weights that provides low
values of error(T,w*) for any set T. In the following
sections I summarize some of the most popular and
other not so popular but more efficient methods to train
the net (i.e., to compute appropriate weight values).
MAIN THRUST
Several models inspired by biological neural networks
have been proposed throughout the years, beginning
with the perceptron introduced by Rosenblatt (1962).
He studied a simple architecture where the output of the
net is a transformation of a linear combination of the
input variables and the weights. Minskey and Papert
(1969) showed that the perceptron can only solve lin-
early separable classification problems and is therefore
of limited interest. A natural extension to overcome its
limitations is given by the so-called multilayer-
perceptron, or, simply, multilayer neural networks. I
have considered this architecture with a single hidden
layer. A schematic representation of the network ap-
pears in Figure 1.
Neural Network Architecture
Let NN=(N, A) be an ANN where N is the set of nodes and
A is the set of arcs. N is partitioned into three subsets:
NI, input nodes, NH, hidden nodes, and NO, output nodes.
I assume that n variables exist in the function that I want
to predict or approximate, therefore |N I|= n. The neural
network has m hidden neurons (|N H|= m) with a bias term
in each hidden neuron and a single output neuron (we
Figure 1. Neural network diagram
inp u ts w 1, n+ 1
x1 1 n+1
w n+ 1 ,s
ou tp ut
x2 2 n+2 s
xn n n+m
restrict our attention to real functions f: n ). Figure
1 shows a net where NI = 1, 2, ..., n, NH = n+1, n+2,..., A
n+m and N O = s.
Given an input pattern x=(x1,...,xn), the neural net-
work provides the user with an associated output NN(x,w),
which is a function of the weights w. Each node i in the
input layer receives a signal of amount xi that it sends
through all its incident arcs to the nodes in the hidden
layer. Each node n+j in the hidden layer receives a signal
input(n+j) according to the expression
n
Input(n+j)=wn+j + x w
i =1
i i ,n + j
where wn+j is the bias value for node n+j, and wi,n+j is the
weight value on the arc from node i in the input layer to
node n+j in the hidden layer. Each hidden node trans-
forms its input by means of a nonlinear activation func-
tion: output(j)=sig(input(j)). The most popular choice
for the activation function is the sigmoid function
sig(x)= 1/(1+e-x). Laguna and Mart (2002) test two
activation functions for the hidden neurons and con-
clude that the sigmoid presents superior performance.
Each hidden node n+j sends the amount of signal
output(n+j) through the arc (n+j,s). The node s in the
output layer receives the weighted sum of the values
coming from the hidden nodes. This sum, NN(x,w), is the
nets output according to the expression:
m
NN(x,w) = ws + output (n + j) w
j =1
n+ j ,s
In the process of training the net (supervised learn-
ing), the problem is to find the values of the weights
(including the bias factors) that minimize the error
(RMSE) across the training set E. After the optimization
has been performed and the weights have been set
(w=w*),the net is ready to produce the output for any
input value. The testing error Error(T,w*) computes the
Root Mean Squared Error across the elements in the
testing set T={y1, y2,..,ys}, where no one belongs to the
training set E:
s
error ( y , w ) i
Error(T,w*) = i =1 .
s
Training Methods
Considering the supervised learning described in the
previous section, many different training methods have
been proposed. Here, I summarize some of the most
relevant, starting with the well-known backpropagation
55
TEAM LinG

method. For a deeper understanding of them, see the
excellent book by Bishop (1995).
Backpropagation (BP) was the first method for neu-
ral network training and is still the most widely used
algorithm in practical applications. It is a gradient de-
scent method that searches for the global optimum of the
network weights. Each iteration consists of two steps.
First, partial derivatives Error/ w are computed for
each weight in the net. Then weights are modified to
reduce the RMSE according to the direction given by the
gradient. There have been different modifications to this
basic procedure; the most significant is the addition of a
momentum term to prevent zigzagging in the search.
Because the neural network training problem can be
expressed as a nonlinear unconstrained optimization
problem, I might use more elaborated nonlinear methods
than the gradient descent to solve it. A selection of the
best established algorithms in unconstrained nonlinear
optimization has also been used in this context. Specifi-
cally, the nonlinear simplex method, the direction set
method, the conjugate gradient method, the Levenberg-
Marquardt algorithm (More, 1978), and the GRG2
(Smith and Lasdon, 1992).
Recently, metaheuristic methods have also been
adapted to this problem. Specifically, on one hand you
can find those methods based on local search proce-
dures, and on the other, those methods based on popula-
tion of solutions known as evolutionary methods. In the
first category, two methods have been applied, simulated
annealing and tabu search, while in the second you can
find the so-called genetic algorithms, the scatter search,
and, more recently, a path relinking implementation.
Several studies (Sexton, 1998) have shown that tabu
search outperforms the simulated annealing implemen-
tation; therefore, I first focus on the different tabu
search implementations for ANN training.
Tabu Search
Tabu search (TS) is based on the premise that in order to
qualify as intelligent, problem solving must incorporate
adaptive memory and responsive exploration. The adap-
tive memory feature of TS allows the implementation of
procedures that are capable of searching the solution
space economically and effectively. Because local
choices are guided by information collected during the
search, TS contrasts with memoryless designs that heavily
rely on semirandom processes that implement a form of
sampling. The emphasis on responsive exploration in
tabu search, whether in a deterministic or probabilistic
implementation, derives from the supposition that a bad
strategic choice can yield more information than a good
random choice. In a system that uses memory, a bad choice
56
Artificial Neural Networks for Prediction
based on strategy can provide useful clues about how
the strategy may profitably be changed.
As far as I know, the first tabu search approach for
neural network training is due to Sexton et al. (1998).
A short description follows. An initial solution x0 is
randomly drawn from a uniform distribution in the
range [-10,10]. Solutions are randomly generated in
this range for a given number of iterations. When
generating a new point xnew, aspiration level and tabu
conditions are checked. If f(xnew)<f(xbest), then the point
is automatically accepted and both xbest and f(xbest) are
updated; otherwise, the tabu conditions are tested. If
there is one solution xi in the tabu list (TL) such as
f(xnew) [f(xi)-0.01*f(x i), f(xi)+0.01*f(xi)], then the com-
plete test is applied to xnew and xi; otherwise, the point
is accepted. The test checks whether all the weights in
x new are within 0.01 from xi in this case the point is
rejected; otherwise, the point is accepted, and xnew and
f(x new) are entered into TL. This process continues for
1,000 iterations of accepted solutions. Then another
cycle of 1,000 iterations of random sampling begins.
These cycles will continuously repeat while f(x best)
improves.
Mart and El-Fallahi (2004) propose an improved
tabu search method that consists of three phases:
MultiRSimplex, TSProb, and TSFreq. After the initial-
ization with the MultiRSimplex phase, the procedure
performs iterations in a loop consisting in alternating
both phases, TSProb and TSFreq, to intensify and diver-
sify the search respectively. In this work, a computa-
tional study of 12 methods for neural network training
is presented, including nonlinear and local-search-based
optimizers. Overall, experiments with 45 functions
from the literature were performed to compare the
procedures. The experiments show that some functions
cannot be approximated with a reasonable accuracy
level when training the net for a limited number of
iterations. The experimentation also shows that the
proposed TS provides, on average, the best solutions
(best approximations).
Evolutionary Methods
The idea of applying the biological principle of natural
evolution to artificial systems, introduced more than
three decades ago, has seen impressive growth in the
past few years. Evolutionary algorithms have been suc-
cessfully applied to numerous problems from different
domains, including optimization, automatic program-
ming, machine learning, economics, ecology, popula-
tion genetics, studies of evolution and learning, and
social systems.
A genetic algorithm is an iterative procedure that
consists of a constant-size population of individuals,
TEAM LinG
Artificial Neural Networks for Prediction
each represented by a finite string of symbols, known as
the genome, encoding a possible solution in a given
problem space. This space, referred to as the search
space, comprises all possible solutions to the problem
at hand. Solutions to a problem were originally encoded
as binary strings due to certain computational advan-
tages associated with such encoding. Also, the theory
about the behavior of algorithms was based on binary
strings. Because in many instances it is impractical to
represent solutions by using binary strings, the solution
representation has been extended in recent years to
include character-based encoding, real-valued encod-
ing, and tree representations.
The standard genetic algorithm proceeds as follows.
An initial population of individuals is generated at ran-
dom or heuristically. Every evolutionary step, known as
a generation, the individuals in the current population
are decoded and evaluated according to some predefined
quality criterion, referred to as the fitness, or fitness
function. To form a new population (the next genera-
tion), individuals are selected according to their fitness.
Many selection procedures are currently in use, one of
the simplest being Hollands original fitness-propor-
tionate selection, where individuals are selected with a
probability proportional to their relative fitness. This
ensures that the expected number of times an individual
is chosen is approximately proportional to its relative
performance in the population. Thus, high-fitness
(good) individuals stand a better chance of reproduc-
ing, while low-fitness ones are more likely to disappear.
In terms of ANN training, a solution (or individual)
consists of an array with the nets weights and its asso-
ciated fitness is usually the RMSE obtained with this
solution in the training set. You can find a lot of research
in GA implementations to ANNs. Consider, for in-
stance, the recent work by Alba and Chicano (2004), in
which a hybrid GA is proposed. Here, the hybridization
refers to the inclusion of problem-dependent knowl-
edge in a general search template. The hybrid algorithms
used in this work are combinations of two algorithms
(weak hybridization), where one of them acts as an
operator in the other. This kind of combinations has
produced the most successful training methods in the
last few years. The authors proposed here the combina-
tion of GA with the BP algorithm as well as GA with the
Levenberg-Marquardt for training ANNs.
Scatter search (SS) was first introduced in Glover
(1977) as a heuristic for integer programming. The
following template is a standard for implementing scat-
ter search that consists of five methods. A diversifica-
tion generation method to generate a collection of
diverse trial solutions, an improvement method to trans-
form a trial solution into one or more enhanced trial
solutions, a reference set update method to build and
maintain a reference set consisting of the b best solu- A
tions found, a subset generation method to operate on the
reference set in order to produce a subset of its solutions
as a basis for creating combined solutions, and a solution
combination method to transform a given subset of
solutions produced by the subset generation method into
one or more combined solution vectors. An exhaustive
description of these methods and how they operate can
be found in Laguna and Mart (2003).
Laguna and Mart (2002) proposed a three-step Scat-
ter Search algorithm for ANNs. El-Fallahi, Mart, and
Lasdon (in press) propose a new training method based
on the path relinking methodology. Path relinking starts
from a given set of elite solutions obtained during a
previous search process. Path relinking and its cousin,
Scatter Search, are mainly based on two elements: com-
binations and local search. Path relinking generalizes
the concept of combination beyond its usual application
to consider paths between solutions. Local search, per-
formed now with the GRG2 optimizer, intensifies the
search by seeking local optima. The paper shows an
empirical comparison of the proposed method with the
best previous evolutionary approaches, and the associ-
ated experiments show the superiority of the new method
in terms of solution quality (prediction accuracy). On
the other hand, these experiments confirm again that a
few functions cannot be approximated with any of the
current training methods.
FUTURE TRENDS
An open problem in the context of prediction is to
compare ANNs with some modern approximation tech-
niques developed in statistics. Specifically, the non-
parametric additive models and local regression can
also offer good solutions to the general approximation
or prediction problem. The development of hybrid sys-
tems from both technologies could give the starting
point for a new generation of prediction systems.
CONCLUSION
In this work I revise the most representative methods for
neural network training. Several computational studies
with some of these methods reveal that the best results
are achieved with a combination of a metaheuristic
procedure with a nonlinear optimizer. These experi-
ments also show that from a practical point of view,
some functions cannot be approximated.
57
TEAM LinG

REFERENCES
Alba, E., & Chicano, J. F. (2004). Training neural networks
with GA Hybrid algorithms. In K. Deb (Ed.), Proceedings
of the Genetic and Evolutionary Computation Confer-
ence, USA.
Bishop, C. M. (1995). Neural networks for pattern
recognition. New York: Oxford University Press.
El-Fallahi, A., Mart, R., & Lasdon, L. (in press). Path
relinking and GRG for artificial neural networks. Euro-
pean Journal of Operational Research.
Glover, F. (1977). Heuristics for integer programming
using surrogate constraints. Decision Sciences, 8, 156-166.
Glover, F., & Laguna, M. (1993). Tabu search. In C. Reeves
(Ed.), Heuristic techniques for combinatorial problems
(pp. 70-150).
Laguna, M., & Mart, R. (2002). Neural network predic-
tion in a system for optimizing simulations. IIE Trans-
actions, 34(3), 273-282.
Laguna, M., & Mart, R. (2003). Scatter search: Meth-
odology and implementations in C. Kluwer Academic.
Mart, R., & El-Fallahi, A. (2004). Multilayer neural
networks: An experimental evaluation of on-line train-
ing methods. Computers and Operations Research,
31, 1491-1513.
Mart, R., Laguna, M., & Glover, F. (in press). Principles
of scatter search. European Journal of Operational
Research.
Minsky, M. L., & Papert, S.A. (1969). Perceptrons
(Expanded ed.). Cambridge, MA: MIT Press.
More, J. J. (1978). The Levenberg-Marquardt algo-
rithm: Implementation and theory. In G. Watson (Ed.),
Lecture Notes in Mathematics: Vol. 630.
Press, W. H., Teukolsky, S. A., Vetterling, W. T., &
Flannery, B. P. (1992). Numerical recipes: The art of
scientific computing. Cambridge, MA: Cambridge Uni-
versity Press.
Rosemblatt, F. (1962). Principles of neurodynamics:
Perceptrons and theory of brain mechanisms. Washing-
ton, DC: Spartan.
58
Artificial Neural Networks for Prediction
Sexton. (1998). Global optimization for artificial neural
networks: A tabu search application. European Journal
of Operational Research, 106, 570-584.
Sexton, R. S., Dorsey, R. E., & Johnson, J. D. (1999).
Optimization of neural networks: A comparative analy-
sis of the genetic algorithm and simulated annealing.
European Journal of Operational Research, 114, 589-601.
Smith, S., & Lasdon, L. (1992). Solving large nonlinear
programs using GRG. ORSA Journal on Computing,
4(1), 2-15.
KEY TERMS
Classification: Also known as a recognition prob-
lem; the identification of the class to which a given
object belongs.
Genetic Algorithm: An iterative procedure that
consists of a constant-size population of individuals,
each represented by a finite string of symbols, known as
the genome, encoding a possible solution in a given
problem space.
Metaheuristic: A master strategy that guides and
modifies other heuristics to produce solutions beyond
those that are normally generated in a quest for local
optimality.
Network Training: The process of finding the val-
ues of the network weights that minimize the error
across a set of input/output pairs (patterns) called the
training set.
Optimization: The quantitative study of optima and
the methods for finding them.
Prediction: Consists of approximating unknown
functions. The nets input is the values of the function
variables, and the output is the estimation of the func-
tion image.
Scatter Search: A metaheuristic that belongs to the
evolutionary methods.
Tabu Search: A metaheuristic procedure based on
principles of intelligent search. Its premise is that prob-
lem solving, in order to qualify as intelligent, must incor-
porate adaptive memory and responsive exploration.
TEAM LinG

Association Rule Mining
Yew-Kwong Woon
Nanyang Technological University, Singapore
Wee-Keong Ng
Nanyang Technological University, Singapore
Ee-Peng Lim
Nanyang Technological University, Singapore
INTRODUCTION
Association Rule Mining (ARM) is concerned with how
items in a transactional database are grouped together. It
is commonly known as market basket analysis, because
it can be likened to the analysis of items that are fre-
quently put together in a basket by shoppers in a market.
From a statistical point of view, it is a semiautomatic
technique to discover correlations among a set of vari-
ables.
ARM is widely used in myriad applications, includ-
ing recommender systems (Lawrence, Almasi, Kotlyar,
Viveros, & Duri, 2001), promotional bundling (Wang,
Zhou, & Han, 2002), Customer Relationship Manage-
ment (CRM) (Elliott, Scionti, & Page, 2003), and cross-
selling (Brijs, Swinnen, Vanhoof, & Wets, 1999). In
addition, its concepts have also been integrated into
other mining tasks, such as Web usage mining (Woon,
Ng, & Lim, 2002), clustering (Yiu & Mamoulis, 2003),
outlier detection (Woon, Li, Ng, & Lu, 2003), and
classification (Dong & Li, 1999), for improved effi-
ciency and effectiveness.
CRM benefits greatly from ARM as it helps in the
understanding of customer behavior (Elliott et al., 2003).
Marketing managers can use association rules of prod-
ucts to develop joint marketing campaigns to acquire
new customers. The application of ARM for the cross-
selling of supermarket products has been successfully
attempted in many cases (Brijs et al., 1999). In one
particular study involving the personalization of super-
market product recommendations, ARM has been ap-
plied with much success (Lawrence et al., 2001). To-
gether with customer segmentation, ARM helped to
increase revenue by 1.8%.
In the biology domain, ARM is used to extract novel
knowledge on protein-protein interactions (Oyama,
Kitano, Satou, & Ito, 2002). It is also successfully
applied in gene expression analysis to discover biologi-
cally relevant associations between different genes or
between different environment conditions (Creighton
& Hanash, 2003).
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
59
A
BACKGROUND
Recently, a new class of problems emerged to challenge
ARM researchers: Incoming data is streaming in too
fast and changing too rapidly in an unordered and un-
bounded manner. This new phenomenon is termed data
stream (Babcock, Babu, Datar, Motwani, & Widom,
2002).
One major area where the data stream phenomenon
is prevalent is the World Wide Web (Web). A good
example is an online bookstore, where customers can
purchase books from all over the world at any time. As
a result, its transactional database grows at a fast rate
and presents a scalability problem for ARM. Traditional
ARM algorithms, such as Apriori, were not designed to
handle large databases that change frequently (Agrawal
& Srikant, 1994). Each time a new transaction arrives,
Apriori needs to be restarted from scratch to perform
ARM. Hence, it is clear that in order to conduct ARM on
the latest state of the database in a timely manner, an
incremental mechanism to take into consideration the
latest transaction must be in place.
In fact, a host of incremental algorithms have already
been introduced to mine association rules incremen-
tally (Sarda & Srinivas, 1998). However, they are only
incremental to a certain extent; the moment the univer-
sal itemset (the number of unique items in a database)
(Woon, Ng, & Das, 2001) is changed, they have to be
restarted from scratch. The universal itemset of any
online store would certainly be changed frequently,
because the store needs to introduce new products and
retire old ones for competitiveness. Moreover, such
incremental ARM algorithms are efficient only when
the database has not changed much since the last mining.
The use of data structures in ARM, particularly the
trie, is one viable way to address the data stream phe-
nomenon. Data structures first appeared when program-
ming became increasingly complex during the 1960s. In
his classic book, The Art of Computer Programming
Knuth (1968) reviewed and analyzed algorithms and data
structures that are necessary for program efficiency.
TEAM LinG

Since then, the traditional data structures have been
extended, and new algorithms have been introduced for
them. Though computing power has increased tremen-
dously over the years, efficient algorithms with custom-
ized data structures are still necessary to obtain timely
and accurate results. This fact is especially true for ARM,
which is a computationally intensive process.
The trie is a multiway tree structure that allows fast
searches over string data. In addition, as strings with
common prefixes share the same nodes, storage space
is better utilized. This makes the trie very useful for
storing large dictionaries of English words. Figure 1
shows a trie storing four English words (ape, apple,
base, and ball). Several novel trielike data structures
have been introduced to improve the efficiency of ARM,
and we discuss them in this section.
Amir, Feldman, & Kashi (1999) presented a new way
of mining association rules by using a trie to preprocess
the database. In this approach, all transactions are mapped
onto a trie structure. This mapping involves the extrac-
tion of the powerset of the transaction items and the
updating of the trie structure. Once built, there is no
longer a need to scan the database to obtain support
counts of itemsets, because the trie structure contains
all their support counts. To find frequent itemsets, the
structure is traversed by using depth-first search, and
itemsets with support counts satisfying the minimum
support threshold are added to the set of frequent
itemsets.
Drawing upon that work, Yang, Johar, Grama, &
Szpankowski (2000) introduced a binary Patricia trie to
reduce the heavy memory requirements of the prepro-
cessing trie. To support faster support queries, the
authors added a set of horizontal pointers to index
nodes. They also advocated the use of some form of
primary threshold to further prune the structure. How-
ever, the compression achieved by the compact Patricia
trie comes at a hefty price: It greatly complicates the
horizontal pointer index, which is a severe overhead. In
addition, after compression, it will be difficult for the
Patricia trie to be updated whenever the database is
altered.
Figure 1. An example of a trie for storing English
words
ROOT
A B
P A
E P S L
L E L
E B
60
Association Rule Mining
The Frequent Pattern-growth (FP-growth) algo-
rithm is a recent association rule mining algorithm that
achieves impressive results (Han, Pei, Yin, & Mao,
2004). It uses a compact tree structure called a Fre-
quent Pattern-tree (FP-tree) to store information about
frequent 1-itemsets. This compact structure removes
the need for multiple database scans and is constructed
with only 2 scans. In the first database scan, frequent 1-
itemsets are obtained and sorted in support descending
order. In the second scan, items in the transactions are
first sorted according to the order of the frequent 1-
itemsets. These sorted items are used to construct the
FP-tree. Figure 2 shows an FP-tree constructed from
the database in Table 1.
FP-growth then proceeds to recursively mine FP-
trees of decreasing size to generate frequent itemsets
without candidate generation and database scans. It does
so by examining all the conditional pattern bases of
the FP-tree, which consists of the set of frequent itemsets
occurring with the suffix pattern. Conditional FP-trees
are constructed from these conditional pattern bases,
and mining is carried out recursively with such trees to
discover frequent itemsets of various sizes. However,
because both the construction and the use of the FP-
trees are complex, the performance of FP-growth is
reduced to be on par with Apriori at support thresholds
of 3% and above. It only achieves significant speed-ups
at support thresholds of 1.5% and below. Moreover, it is
only incremental to a certain extent, depending on the
FP-tree watermark (validity support threshold). As new
transactions arrive, the support counts of items in-
crease, but their relative support frequency may de-
crease, too. Suppose, however, that the new transactions
cause too many previously infrequent itemsets to become
Table 1. A sample transactional database
TID Items
100 AC
200 BC
300 ABC
400 ABCD
Figure 2. An FP-tree constructed from the database in
Table 1 at a support threshold of 50%
ROOT
C
A B
TEAM LinG
Association Rule Mining
frequent that is, the watermark is raised too high (in
order to make such itemsets infrequent) according to a
user-defined level then the FP-tree must be recon-
structed.
The use of lattice theory in ARM was pioneered by
Zaki (2000). Lattice theory allows the vast search space
to be decomposed into smaller segments that can be
tackled independently in memory or even in other ma-
chines, thus promoting parallelism. However, they re-
quire additional storage space as well as different tra-
versal and construction techniques. To complement the
use of lattices, Zaki uses a vertical database format,
where each itemset is associated with a list of transac-
tions known as a tid-list (transaction identifierlist).
This format is useful for fast frequency counting of
itemsets but generates additional overheads because most
databases have a horizontal format and would need to be
converted first.
The Continuous Association Rule Mining Algorithm
(CARMA), together with the support lattice, allows the
user to change the support threshold and continuously
displays the resulting association rules with support and
confidence bounds during its first scan/phase (Hidber,
1999). During the second phase, it determines the pre-
cise support of each itemset and extracts all the frequent
itemsets. CARMA can readily compute frequent itemsets
for varying support thresholds. However, experiments
reveal that CARMA only performs faster than Apriori at
support thresholds of 0.25% and below, because of the
tremendous overheads involved in constructing the sup-
port lattice.
The adjacency lattice, introduced by Aggarwal & Yu
(2001), is similar to Zakis boolean powerset lattice,
except the authors introduced the notion of adjacency
among itemsets, and it does not rely on a vertical data-
base format. Two itemsets are said to be adjacent to each
other if one of them can be transformed to the other with
the addition of a single item. To address the problem of
heavy memory requirements, a primary threshold is de-
fined. This term signifies the minimum support thresh-
old possible to fit all the qualified itemsets into the
adjacency lattice in main memory. However, this ap-
proach disallows the mining of frequent itemsets at
support thresholds lower than the primary threshold.
MAIN THRUST
As shown in our previous discussion, none of the exist-
ing data structures can effectively address the issues
induced by the data stream phenomenon. Here are the
desirable characteristics of an ideal data structure that
can help ARM cope with data streams:
It is highly scalable with respect to the size of both
the database and the universal itemset. A
It is incrementally updated as transactions are
added or deleted.
It is constructed independent of the support
threshold and thus can be used for various support
thresholds.
It helps to speed up ARM algorithms to a certain
extent that allows results to be obtained in real-
time.
We shall now discuss our novel trie data structure
that not only satisfies the above requirements but also
outperforms the discussed existing structures in terms
of efficiency, effectiveness, and practicality. Our struc-
ture is termed Support-Ordered Trie Itemset (SOTrieIT
pronounced so-try-it). It is a dual-level support-
ordered trie data structure used to store pertinent
itemset information to speed up the discovery of fre-
quent itemsets.
As its construction is carried out before actual
mining, it can be viewed as a preprocessing step. For
every transaction that arrives, 1-itemsets and 2-itemsets
are first extracted from it. For each itemset, the
SOTrieIT will be traversed in order to locate the node
that stores its support count. Support counts of 1-
itemsets and 2-itemsets are stored in first-level and
second-level nodes, respectively. The traversal of the
SOTrieIT thus requires at most two redirections, which
makes it very fast. At any point in time, the SOTrieIT
contains the support counts of all 1-itemsets and 2-
itemsets that appear in all the transactions. It will then
be sorted level-wise from left to right according to the
support counts of the nodes in descending order.
Figure 3 shows a SOTrieIT constructed from the
database in Table 1. The bracketed number beside an item
is its support count. Hence, the support count of itemset
{AB} is 2. Notice that the nodes are ordered by support
counts in a level-wise descending order.
In algorithms such as FP-growth that use a similar
data structure to store itemset information, the structure
must be rebuilt to accommodate updates to the universal
Figure 3. A SOTrieIT structure
ROOT
C(4) A(3) B(3) D(1)
D(1) C(3) B(2) D(1) C(3) D(1)
61
TEAM LinG

itemset. The SOTrieIT can be easily updated to accommo-
date the new changes. If a node for a new item in the
universal itemset does not exist, it will be created and
inserted into the SOTrieIT accordingly. If an item is
removed from the universal itemset, all nodes contain-
ing that item need only be removed, and the rest of the
nodes would still be valid.
Unlike the trie structure of Amir et al. (1999), the
SOTrieIT is ordered by support count (which speeds up
mining) and does not require the powersets of transac-
tions (which reduces construction time). The main weak-
ness of the SOTrieIT is that it can only discover frequent
1-itemsets and 2-itemsets; its main strength is its speed
in discovering them. They can be found promptly be-
cause there is no need to scan the database. In addition,
the search (depth first) can be stopped at a particular
level the moment a node representing a nonfrequent
itemset is found, because the nodes are all support
ordered.
Another advantage of the SOTrieIT, compared with
all previously discussed structures, is that it can be
constructed online, meaning that each time a new trans-
action arrives, the SOTrieIT can be incrementally up-
dated. This feature is possible because the SOTrieIT is
constructed without the need to know the support thresh-
old; it is support independent. All 1-itemsets and 2-
itemsets in the database are used to update the SOTrieIT
regardless of their support counts. To conserve storage
space, existing trie structures such as the FP-tree have
to use thresholds to keep their sizes manageable; thus,
when new transactions arrive, they have to be recon-
structed, because the support counts of itemsets will
have changed.
Finally, the SOTrieIT requires far less storage space
than a trie or Patricia trie because it is only two levels
deep and can be easily stored in both memory and files.
Although this causes some input/output (I/O) overheads,
it is insignificant as shown in our extensive experiments.
We have designed several algorithms to work synergis-
tically with the SOTrieIT and, through experiments with
existing prominent algorithms and a variety of databases,
we have proven the practicality and superiority of our
approach (Das, Ng, & Woon, 2001; Woon et al., 2001). In
fact, our latest algorithm, FOLD-growth, is shown to
outperform FP-growth by more than 100 times (Woon,
Ng, & Lim, 2004).
FUTURE TRENDS
The data stream phenomenon will eventually become
ubiquitous as Internet access and bandwidth become
increasingly affordable. With keen competition, prod-
ucts will become more complex with customization and
62
Association Rule Mining
more varied to cater to a broad customer base; transac-
tion databases will grow in both size and complexity.
Hence, association rule mining research will certainly
continue to receive much attention in the quest for
faster, more scalable and more configurable algorithms.
CONCLUSION
Association rule mining is an important data mining task
with several applications. However, to cope with the
current explosion of raw data, data structures must be
utilized to enhance its efficiency. We have analyzed
several existing trie data structures used in association
rule mining and presented our novel trie structure, which
has been proven to be most useful and practical. What
lies ahead is the parallelization of our structure to
further accommodate the ever-increasing demands of
todays need for speed and scalability to obtain associa-
tion rules in a timely manner. Another challenge is to
design new data structures that facilitate the discovery
of trends as association rules evolve over time. Differ-
ent association rules may be mined at different time
points and, by understanding the patterns of changing rules,
additional interesting knowledge may be discovered.
REFERENCES
Aggarwal, C. C., & Yu, P. S. (2001). A new approach to
online generation of association rules. IEEE Transac-
tions on Knowledge and Data Engineering, 13(4), 527-
540.
Agrawal, R., & Srikant, R. (1994). Fast algorithms for
mining association rules. Proceedings of the 20th In-
ternational Conference on Very Large Databases (pp.
487-499), Chile.
Amir, A., Feldman, R., & Kashi, R. (1999). A new and
versatile method for association generation. Informa-
tion Systems, 22(6), 333-347.
Babcock, B., Babu, S., Datar, M., Motwani, R., & Widom,
J. (2002). Models and issues in data stream systems.
Proceedings of the ACM SIGMOD/PODS Conference
(pp. 1-16), USA.
Brijs, T., Swinnen, G., Vanhoof, K., & Wets, G. (1999).
Using association rules for product assortment deci-
sions: A case study. Proceedings of the Fifth ACM
SIGKDD Conference (pp. 254-260), USA.
Creighton, C., & Hanash, S. (2003). Mining gene expres-
sion databases for association rules. Bioinformatics,
19(1), 79-86.
TEAM LinG
Association Rule Mining
Das, A., Ng, W. K., & Woon, Y. K. (2001). Rapid associa-
tion rule mining. Proceedings of the 10th International
Conference on Information and Knowledge Manage-
ment (pp. 474-481), USA.
Dong, G., & Li, J. (1999). Efficient mining of emerging
patterns: Discovering trends and differences. Proceed-
ings of the Fifth International Conference on Knowl-
edge Discovery and Data Mining (pp. 43-52), USA.
Elliott, K., Scionti, R., & Page, M. (2003). The confluence
of data mining and market research for smarter CRM.
Retrieved from http://www.spss.com/home_page/
wp133.htm
Han, J., Pei, J., Yin Y., & Mao, R. (2004). Mining frequent
patterns without candidate generation: A frequent-pat-
tern tree approach. Data Mining and Knowledge Discov-
ery, 8(1), 53-97.
Hidber, C. (1999). Online association rule mining. Pro-
ceedings of the ACM SIGMOD Conference (pp. 145-154),
USA.
Knuth, D.E. (1968). The art of computer programming, Vol.
1. Fundamental Algorithms. Addison-Wesley Publish-
ing Company.
Lawrence, R. D., Almasi, G. S., Kotlyar, V., Viveros, M. S.,
& Duri, S. (2001). Personalization of supermarket product
recommendations. Data Mining and Knowledge Discov-
ery, 5(1/2), 11-32.
Oyama, T., Kitano, K., Satou, K., & Ito, T. (2002). Extrac-
tion of knowledge on protein-protein interaction by asso-
ciation rule discovery. Bioinformatics, 18(5), 705-714.
Sarda, N. L., & Srinivas, N. V. (1998). An adaptive algo-
rithm for incremental mining of association rules. Pro-
ceedings of the Ninth International Conference on Da-
tabase and Expert Systems (pp. 240-245), Austria.
Wang, K., Zhou, S., & Han, J. (2002). Profit mining: From
patterns to actions. Proceedings of the Eighth Interna-
tional Conference on Extending Database Technology
(pp. 70-87), Prague.
Woon, Y. K., Li, X., Ng, W. K., & Lu, W. F. (2003).
Parameterless data compression and noise filtering us-
ing association rule mining. Proceedings of the Fifth
International Conference on Data Warehousing and
Knowledge Discovery (pp. 278-287), Prague.
Woon, Y. K., Ng, W. K., & Das, A. (2001). Fast online
dynamic association rule mining. Proceedings of the
Second International Conference on Web Information
Systems Engineering (pp. 278-287), Japan.
Woon, Y. K., Ng, W. K., & Lim, E. P. (2002). Online and
incremental mining of separately grouped web access A
logs. Proceedings of the Third International Conference
on Web Information Systems Engineering (pp. 53-62),
Singapore.
Woon, Y. K., Ng, W. K., & Lim, E. P. (2004). A support-
ordered trie for fast frequent itemset discovery. IEEE
Transactions on Knowledge and Data Engineering,
16(5).
Yang, D. Y., Johar, A., Grama, A., & Szpankowski, W.
(2000). Summary structures for frequency queries on
large transaction sets. Proceedings of the Data Com-
pression Conference (pp. 420-429).
Yiu, M. L., & Mamoulis, N. (2003). Frequent-pattern based
iterative projected clustering. Proceedings of the Third
International Conference on Data Mining, USA.
Zaki, M. J. (2000). Scalable algorithms for association
mining. IEEE Transactions on Knowledge and Data
Engineering, 12(3), 372-390.
KEY TERMS
Apriori: A classic algorithm that popularized asso-
ciation rule mining. It pioneered a method to generate
candidate itemsets by using only frequent itemsets in
the previous pass. The idea rests on the fact that any
subset of a frequent itemset must be frequent as well.
This idea is also known as the downward closure prop-
erty.
Itemset: An unordered set of unique items, which
may be products or features. For computational effi-
ciency, the items are often represented by integers. A
frequent itemset is one with a support count that ex-
ceeds the support threshold, and a candidate itemset is
a potential frequent itemset. A k-itemset is an itemset
with exactly k items.
Key: A unique sequence of values that defines the
location of a node in a tree data structure.
Patricia Trie: A compressed binary trie. The
Patricia (Practical Algorithm to Retrieve Information
Coded in Alphanumeric) trie is compressed by avoiding
one-way branches. This is accomplished by including in
each node the number of bits to skip over before making
the next branching decision.
SOTrieIT: A dual-level trie whose nodes represent
itemsets. The position of a node is ordered by the support
count of the itemset it represents; the most frequent
63
TEAM LinG

itemsets are found on the leftmost branches of the
SOTrieIT.
Support Count of an Itemset: The number of transac-
tions that contain a particular itemset.
Support Threshold: A threshold value that is used to
decide if an itemset is interesting/frequent. It is defined by
the user, and generally, an association rule mining algo-
64
Association Rule Mining
rithm has to be executed many times before this value can
be well adjusted to yield the desired results.
Trie: An n-ary tree whose organization is based on
key space decomposition. In key space decomposition,
the key range is equally subdivided, and the splitting
position within the key range for each node is pre-
defined.
TEAM LinG

Association Rule Mining and Application to
MPIS
Raymond Chi-Wing Wong
The Chinese University of Hong Kong, Hong Kong
Ada Wai-Chee Fu
The Chinese University of Hong Kong, Hong Kong
INTRODUCTION
Association rule mining (Agrawal, Imilienski, & Swami,
1993) has been proposed for understanding the relation-
ships among items in transactions or market baskets. For
instance, if a customer buys butter, what is the chance that
he/she buys bread at the same time? Such information may
be useful for decision makers to determine strategies in a
store.
BACKGROUND
Given a set I = {I1, I2,, In} of items (e.g., carrot, orange and
knife) in a supermarket. The database contains a number
of transactions. Each transaction t is a binary vector with
t[k]=1 if t bought item Ik and t[k]=0 otherwise (e.g., {1,
0, 0, 1, 0}). An association rule is of the form X Ij, where
X is a set of some items in I, and Ij is a single item not in X
(e.g., {Orange, Knife} Plate).
A transaction t satisfies X if for all items Ik in X, t[k]
= 1. The support for a rule X Ij is the fraction of trans-
actions that satisfy the union of X and Ij. A rule X Ij has
confidence c% if and only if c% of transactions that
satisfy X also satisfy Ij.
The mining process of association rule can be divided
into two steps:
1. Frequent Itemset Generation: Generate all sets of
items that have support greater than or equal to a
certain threshold, called minsupport
2. Association Rule Generation: From the frequent
itemsets, generate all association rules that have
confidence greater than or equal to a certain thresh-
old called minconfidence
Step 1 is much more difficult compared with Step 2.
Thus, researchers have focused on the studies of fre-
quent itemset generation.
The Apriori Algorithm is a well-known approach,
which was proposed by Agrawal & Srikant (1994), to find
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
65
A
frequent itemsets. It is an iterative approach and there are
two steps in each iteration. The first step generates a set
of candidate itemsets. Then, the second step prunes all
disqualified candidates (i.e., all infrequent itemsets). The
iterations begin with size 2 itemsets and the size is
incremented at each iteration. The algorithm is based on
the closure property of frequent itemsets: if a set of items
is frequent, then all its proper subsets are also frequent.
The weaknesses of this algorithm are the generation of a
large number of candidate itemsets and the requirement to
scan the database once in each iteration.
A data structure called FP-tree and an efficient algo-
rithm called FP-growth are proposed by Han, Pei, & Yin
(2000) to overcome the above weaknesses. The idea of FP-
tree is fetching all transactions from the database and
inserting them into a compressed tree structure. Then,
algorithm FP-growth reads from the FP-tree structure to
mine frequent itemsets.
MAIN THRUST
Variations in Association Rules
Many variations on the above problem formulation have
been suggested. The association rules can be classified
based on the following (Han & Kamber, 2000):
1. Association Rules Based on the Type of Values of
Attribute: Based on the type of values of attributes,
there are two kinds Boolean association rule,
which is presented above, and quantitative associa-
tion rule. Quantitative association rule describes
the relationships among some quantitative attributes
(e.g., income and age). An example is
income(40K..50K) age(40..45). One proposed
method is grid-based dividing each attribute into
a fixed number of partitions [Association Rule Clus-
tering System (ARCS) in Lent, Swami & Widom
(1997)]. Srikant & Agrawal (1996) proposed to par-
tition quantitative attributes dynamically and to
TEAM LinG

merge the partitions based on a measure of partial
completeness. Another non-grid based approach is
found in Zhang, Padmanabhan, & Tuzhilin (2004).
2. Association Rules based on the Dimensionality of
Data: Association rules can be divided into single-
dimensional association rules and multi-dimen-
sional association rules. One example of single-
dimensional rule is buys({Orange, Knife})
buys(Plate), which contains only the dimension
buys. Multi-dimensional association rule is the one
containing attributes for more than one dimension.
For example, income(40K..50K) buys(Plate). One
mining approach is to borrow the concept of data
cube in the field of data warehousing. Figure 1
shows a lattice for the data cube for the dimensions
age, income and buys. Researchers (Kamber, Han,
& Chiang, 1997) applied the data cube model and
used the aggregate techniques for mining.
3. Association Rules based on the Level of Abstrac-
tions of Attribute: The rules discussed in previous
sections can be viewed as single-level association
rule. A rule that references different levels of ab-
straction of attributes is called a multilevel associa-
tion rule. Suppose there are two rules
income(10K..20K) buys(fruit) and
income(10K..20K) buys(orange). There are two
different levels of abstractions in these two rules
because fruit is a higher-level abstraction of or-
ange. Han & Fu (1995) apply a top-down strategy
to the concept hierarchy in the mining of frequent
itemsets.
Other Extensions to Association Rule
Mining
There are other extensions to association rule mining.
Some of them (Bayardo, 1998) find maxpattern (i.e., maxi-
mal frequent patterns) while others (Zaki & Hsiao, 2002)
find frequent closed itemsets. Maxpattern is a frequent
itemset that does not have a frequent item superset. A
frequent itemset is a frequent closed itemsets if there
Figure 1. A lattice showing the data cube for the
dimensions age, income, and buys
()
income
(age) (buys)
(age, income) income, buys)
age, buys)
(age, income, buys)
66
Association Rule Mining and Application to MPIS
Figure 2. A concept hierarchy of the fruit
fruit
apple orange banana
exists no itemset X such that (1) X X and (2) trans-
actions t, X is in t implies X is in t. These considerations
can reduce the resulting number of frequent itemsets
significantly.
Another variation of the frequent itemset problem is
mining top-K frequent itemsets (Cheung & Fu, 2004). The
problem is to find K frequent itemsets with the greatest
supports. It is often more reasonable to assume the
parameter K, instead of the data-distribution dependent
parameter of minsupport because the user typically would
not have the knowledge of the data distribution before
data mining.
The other variations of the problem are the incremen-
tal update of mining association rules (Hidber, 1999),
constraint-based rule mining (Grahne & Lakshmanan,
2000), distributed and parallel association rule mining
(Gilburd, Schuster, & Wolff, 2004), association rule min-
ing with multiple minimum supports/without minimum
support (Chiu, Wu, & Chen, 2004), association rule
mining with weighted item and weight support (Tao,
Murtagh, & Farid, 2003), and fuzzy association rule min-
ing (Kuok, Fu, & Wong, 1998).
Association rule mining has been integrated with
other data mining problems. There have been the integra-
tion of classification and association rule mining (Wang,
Zhou, & He, 2000) and the integration of association rule
mining with relational database systems (Sarawagi, Tho-
mas, & Agrawal, 1998).
Application of the Concept of
Association Rules to MPIS
Other than market basket analysis (Blischok, 1995), asso-
ciation rules can also help in applications such as intru-
sion detection (Lee, Stolfo, & Mok, 1999), heterogeneous
genome data (Satou et al., 1997), mining remotely sensed
images/data (Dong, Perrizo, Ding, & Zhou, 2000) and
product assortment decisions (Wong, Fu, & Wang, 2003;
Wong & Fu, 2004). Here we focus on the application on
product assortment decisions, as it is one of very few
examples where the association rules are not the end
mining results.
TEAM LinG
Association Rule Mining and Application to MPIS
Transaction database in some applications can be very
large. For example, Hedberg (1995) quoted that Wal-Mart
kept about 20 million sales transactions per day. Such data
requires sophisticated analysis. As pointed out by Blischok
(1995), a major task of talented merchants is to pick the
profit generating items and discard the losing items. It may
be simple enough to sort items by their profit and do the
selection. However, this ignores a very important aspect
in market analysis the cross-selling effect. There can be
items that do not generate much profit by themselves but
they are the catalysts for the sales of other profitable items.
Recently, some researchers (Kleinberg, Papadimitriou, &
Raghavan, 1998) suggest that concepts of association
rules can be used in the item selection problem with the
consideration of relationships among items.
One example of the product assortment decisions is
Maximal-Profit Item Selection (MPIS) with cross-selling
considerations (Wong, Fu, & Wang, 2003). Consider the
major task of merchants to pick profit-generating items and
discard the losing items. Assume we have a history record
of the sales (transactions) of all items. This problem is to
select a subset from the given set of items so that the
estimated profit of the resulting selection is maximal among
all choices.
Suppose a shop carries office equipment composed of
monitors, keyboards and telephones, with profits of
$1000K, $100K and $300K, respectively. If now the shop
decides to remove one of the three items from its stock, the
question is which two we should choose to keep. If we
simply examine the profits, we may choose to keep moni-
tors and telephones, and so the total profit is $1300K.
However, we know that there is strong cross-selling effect
between monitor and keyboard (see Table 1). If the shop
stops carrying keyboard, the customers of monitor may
choose to shop elsewhere to get both items. The profit
from monitor may drop greatly, and we may be left with
profit of $300K from telephones only. If we choose to keep
both monitors and keyboards, then the profit can be
expected to be $1100K, which is higher.
MPIS will give us the desired solution. MPIS utilizes
the concept of the relationship between selected items and
unselected items. Such relationship is modeled by the
cross-selling factor. Suppose d is the set of unselected
items and I is the selected item. A loss rule is proposed in
the form I d, where d means the purchase of any item
in d. The rule indicates that from the history, whenever a
customer buys the item I, he/she also buys at least one of
the items in d. Interpreting this as a pattern of customer
behavior, and assuming that the pattern will not change
even when some items were removed from the stock, if
none of the items in d are available then the customer also
will not purchase I. This is because if the customer still
purchases I, without purchasing any items in d, then the
pattern would be changed. Therefore, the higher the con-
fidence of I d, the more likely the profit of I should
not be counted. This is the reasoning behind the above A
definition. In the above example, suppose we choose
monitor and telephone. Then, d = {keyboard}. All profits
of monitor will be lost if, in the history, we find conf(I
d)=1, where I = monitor. This example illustrates the
importance of the consideration of cross-selling factor in
the profit estimation.
Wong, Fu, & Wang (2003) propose two algorithms to
deal with this problem. In the first algorithm, they ap-
proximate the total profit of the item selection in qua-
dratic form and solve a quadratic optimization problem.
The second one is a greedy approach called MPIS_Alg,
which prunes items iteratively according to an estimated
function based on the formula of the total profit of the
item selection until J items remain.
Another product assortment decision problem is stud-
ied by Wong & Fu, (2004), which addresses the problem
of selecting a set of marketing items in order to boost the
sales of the store.
FUTURE TRENDS
A new area for investigation of the problem of mining
frequent itemsets is mining data streaming for frequent
itemsets (Manku & Motwani, 2002; Yu, Chong, Lu, &
Zhou, 2004). In such a problem, the data is so massive
that all data cannot be stored in the memory of a computer
and the data cannot be processed by traditional algo-
rithms. The objective of all proposed algorithm is to store
as few as possible data and to minimize the error gener-
ated by some estimation in the model.
Privacy preservation on the association rule mining
is also rigorously studied in these few years (Vaidya &
Clifton, 2002; Agrawal, Evfimievski, & Srikant, 2003). The
problem is to mine from two or more different sources
without exposing individual transaction data to each
other.
CONCLUSION
Association rule mining plays an important role in the
literature of data mining. It poses many challenging
Table 1.
Monitor Keyboard Telephone
1 1 0
1 1 0
0 0 1
0 0 1
0 0 1
1 1 1
67
TEAM LinG

issues for the development of efficient and effective
methods. After taking a closer look, we find that the
application of association rules requires much more in-
vestigations in order to aid in more specific targets. We
may see a trend towards the study of applications of
association rules.
REFERENCES
Agrawal, R., Evfimievski, A., & Srikant, R. (2003). Informa-
tion sharing across private database. SIGMOD, 86-97.
Agrawal, R., Imilienski, T., & Swami. (1993). Mining asso-
ciation rules between sets of items in large databases.
SIGMOD, 129-140.
Agrawal, R., & Srikant, R. (1994). Fast algorithms for
mining association rules. In Proceedings of the 20th VLDB
Conference (pp. 487-499).
Bayardo, R.J. (1998). Efficiently mining long patterns
from databases. SIGMOD, 85-93.
Blischok, T. (1995). Every transaction tells a story. Chain
Store Age Executive with Shopping Center Age, 71(3),
50-57.
Cheung, Y.L., & Fu, A.W.-C. (2004). Mining association
rules without support threshold: With and without Item
Constraints. TKDE, 16(9), 1052-1069.
Chiu, D.-Y., Wu, Y.-H., & Chen, A.L.P. (2004). An efficient
algorithm for mining frequent sequences by a new strat-
egy without support counting. ICDE, 375-386.
Dong, J., Perrizo, W., Ding, Q., & Zhou, J. (2000). The
application of association rule mining to remotely sensed
data. In Proceedings of the 2000 ACM symposium on
Applied computing (pp. 340-345).
Gilburd, B., Schuster, A., & Wolff, R. (2004). A new
privacy model and association-rule mining algorithm for
large-scale distributed environments. SIGKDD.
Grahne, G., Lakshmanan, L., & Wang, X. (2000). Efficient
mining of constrained correlated sets. ICDE, 512-521
Han, J., & Fu, Y. (1995). Discovery of multiple-level asso-
ciation rules from large databases. In Proceedings of the
1995 International Conference on VLDB (pp. 420-431).
Han, J., & Kamber, M. (2000). Data mining: Concepts and
techniques. San Mateo, CA: Morgan Kaufmann Publishers.
Han, J., Pei, J., & Yin, Y. (2000). Mining frequent patterns
without candidate generation. SIGMOD, 1-12.
68
Association Rule Mining and Application to MPIS
Hedberg, S. (1995, October). The data gold rush. BYTE, 83-99.
Hidber, C. (1999), Online association rule mining. SIGMOD,
145-156.
Kamber, M., Han, J., & Chiang, J.Y. (1997). Metarule-
guided mining of multi-dimensional association rules
using data cubes. In Proceeding of the 3rd International
Conference on Knowledge Discovery and Data Mining
(pp. 207-210).
Kleinberg, J., Papadimitriou, C., & Raghavan, P. (1998). A
microeconomic view of data mining. Knowledge Discov-
ery Journal, 2(4), 311-324.
Kuok, C.M., Fu, A.W.C., & Wong, M.H., (1998). Mining
fuzzy association rules in databases. ACM SIGMOD
Record, 27(1), 41-46.
Lee, W., Stolfo, S.J., & Mok, K.W. (1999). A data mining
framework for building intrusion detection models. In
IEEE Symposium on Security and Privacy (pp. 120-132).
Lent, B., Swami, A.N., & Widom, J. (1997). Clustering
Association Rules. In ICDE (pp. 220-231).
Manku, G.S., & Motwani, R. (2002). Approximate fre-
quency counts over data streams. In Proceedings of the
20 th International Conference on VLDB (pp. 346-357).
Sarawagi, S., Thomas, S., & Agrawal, R. (1998). Integrat-
ing association rule mining with relational database sys-
tems: Alternatives and implications. SIGMOD, 343-354.
Satou, K., Shibayama, G., Ono, T., Yamamura, Y., Furuichi,
E., Kuhara, S., & Takagi, T. (1997). Finding association
rules on heterogeneous genome data. In Pacific Sympo-
sium on Biocomputing (PSB) (pp. 397-408).
Srikant, R., & Agrawal, R. (1996). Mining quantitative
association rules in large relational tables. SIGMOD, 1-12.
Tao, F., Murtagh, F., & Farid, M. (2003). Weighted asso-
ciation rule mining using weighted support and signifi-
cance framework. In The Ninth ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and Data
Mining (pp. 661-666).
Vaidya, J., & Clifton, C. (2002). Privacy preserving asso-
ciation rule mining in vertically partitioned data. In The
Eighth ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (pp. 639-644).
Wang, K., Zhou, S., & He, Y. (2000). Growing decision
trees on support-less association rules. In Sixth ACM
SIGKDD International Conference on Knowledge Dis-
covery & Data Mining (pp. 265-269).
TEAM LinG
Association Rule Mining and Application to MPIS
Wong, R.C.-W., & Fu, A.W.-C. (2004). ISM: Item selection
for marketing with cross-selling considerations. In Ad-
vances in Knowledge Discovery and Data Mining, 8th
Pacific-Asia Conference (PAKDD) (pp. 431-440), Lecture
Notes in Computer Science 3056. Berlin: Springer.
Wong, R.C.-W., Fu, A.W.-C., & Wang, K. (2003). MPIS:
Maximal-profit item selection with cross-selling consider-
ations. In IEEE International Conference on Data Min-
ing (ICDM) (pp. 371-378).
Yu, J.X., Chong, Z., Lu, H., & Zhou, A. (2004). False
positive or false negative: Mining frequent itemsets from
high speed transactional data streams. In Proceedings of
the Thirtieth International Conference on Very Large
Data Bases.
Zaki, M.J., & Hsiao, C.J. (2002). CHARM: An efficient
algorithm for closed itemset mining. In SIAM Interna-
tional Conference on Data Mining (SDM).
Zhang, H., Padmanabhan, B., & Tuzhilin, A. (2004). On the
discovery of significant statistical quantitative rules. In
Proceedings of the 10th ACM SIGKDD Knowledge Dis-
covery and Data Mining Conference.
KEY TERMS
Association Rule: A kind of rule in the form X Ij, where
X is a set of some items and Ij is a single item not in X.
Confidence: The confidence of a rule X I j, where X
is a set of items and Ij is a single item not in X, is the fraction
of the transactions containing all items in set X that also A
contain item Ij.
Frequent Itemset/Pattern: The itemset with support
greater than or equal to a certain threshold, called
minsupport.
Infrequent Itemset: Itemset with support smaller than
a certain threshold, called minsupport.
Itemset: A set of items
K-Itemset: Itemset with k items
Maximal-Profit Item Selection (MPIS): The problem
of item selection, which selects a set of items in order to
maximize the total profit with the consideration of cross-
selling effect
Support (Itemset) Or Frequency: The support of an
itemset X is the fraction of transactions containing all
items in X.
Support (Rule): The support of a rule X Ij, where X
is a set of items and Ij is a single item not in X, is the fraction
of the transactions containing all items in set X that also
contain item Ij.
Transaction: A record containing the items bought
by a customer.
69
TEAM LinG
70
Association Rule Mining of Relational Data
Anne Denton
North Dakota State University, USA
Christopher Besemann
North Dakota State University, USA
INTRODUCTION
Most data of practical relevance are structured in more
complex ways than is assumed in traditional data mining
algorithms, which are based on a single table. The concept
of relations allows for discussing many data structures
such as trees and graphs. Relational data have much
generality and are of significant importance, as demon-
strated by the ubiquity of relational database manage-
ment systems. It is, therefore, not surprising that popular
data mining techniques, such as association rule mining,
have been generalized to relational data. An important
aspect of the generalization process is the identification
of problems that are new to the generalized setting.
BACKGROUND
Several areas of databases and data mining contribute to
advances in association rule mining of relational data:
Relational Data Model: underlies most commercial
database technology and also provides a strong
mathematical framework for the manipulation of
complex data. Relational algebra provides a natural
starting point for generalizations of data mining
techniques to complex data types.
Inductive Logic Programming, ILP (Deroski &
Lavra , 2001): a form of logic programming, in
which individual instances are generalized to make
hypotheses about unseen data. Background knowl-
edge is incorporated directly.
Association Rule Mining, ARM (Agrawal,
Imielinski, & Swami, 1993): identifies associa-
tions and correlations in large databases. Associa-
tion rules are defined based on items, such as
objects in a shopping cart. Efficient algorithms are
designed by limiting output to sets of items that
occur more frequently than a given threshold.
Graph Theory: addresses networks that consist of
nodes, which are connected by edges. Traditional
graph theoretic problems typically assume no more
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
than one property per node or edge. Data associated
with nodes and edges can be modeled within the
relational algebra.
Association rule mining of relational data incorpo-
rates important aspects of these areas to form an innovative
data mining technique of important practical relevance.
MAIN THRUST
The general concept of association rule mining of rela-
tional data will be explored, as well as the special case of
mining a relationship that corresponds to a graph.
General Concept
Two main challenges have to be addressed when applying
association rule mining to relational data. Combined min-
ing of multiple tables leads to a search space that is
typically large even for moderately sized tables. Perfor-
mance is, thereby, commonly an important issue in rela-
tional data mining algorithms. A less obvious problem lies
in the skewing of results (Jensen & Neville, 2002). The
relational join operation combines each record from one
table with each occurrence of the corresponding record in
a second table. That means that the information in one
record is represented multiple times in the joined table.
Data mining algorithms that operate either explicitly or
implicitly on joined tables, thereby, use the same informa-
tion multiple times. Note that this problem also applies to
algorithms in which tables are joined on-the-fly by iden-
tifying corresponding records as they are needed. Further
specific issues may have to be addressed when reflexive
relationships are present. These issues will be discussed
in the section on relations that represent a graph.
A variety of techniques have been developed for data
mining of relational data (D eroski & Lavra , 2001). A
typical approach is called inductive logic programming,
ILP. In this approach relational structure is represented in
the form of Prolog queries, leaving maximum flexibility to
the user. While the notation of ILP differs from the
TEAM LinG
Association Rule Mining of Relational Data
relational notation it can be noted that all relational
operators can also be represented in ILP. The approach
does thereby not limit the types of problems that can be
addressed. It should, however, also be noted that while
relational database management system are developed
with performance in mind there may be a trade-off between
the generality of Prolog-based environments and their
limitations in speed.
Application of ARM within the ILP setting corre-
sponds to a search for frequent Prolog queries as a
generalization of traditional association rules (Dehaspe &
De Raedt, 1997). Examples of association rule mining of
relational data using ILP (Dehaspe & Toivonen, 2001)
could be shopping behavior of customers where relation-
ships between customers are included in the reasoning.
While ILP does not use a relational joining step as such,
it does also associate individual objects with multiple
occurrences of corresponding objects. Problems with
skewing are, thereby, also encountered in this approach.
An alternative to the ILP approach is to apply the
standard definition of association rule mining to relations
that are joined using the relational join operation. While
such an approach is less general it is often more efficient
since the join operation is highly optimized in standard
database systems. It is important to note that a join
operation typically changes the support of an item set,
and any support calculation should therefore be based on
the relation that uses the smallest number of join opera-
tions (Cristofor & Simovici, 2001). Equivalent changes in
item set weighting occur in ILP.
Interestingness of rules is an important issue in any
type of association rule mining. In traditional association
rule mining the problem of rule interest has been ad-
dressed in a variety of work on redundant rules, including
closed set generation (Zaki, 2000). Additional rule metrics
such as lift and conviction have been defined (Brin,
Motwani, Ullman, & Tsur, 1997). In relational association
rule mining the problem has been approached by the
definition of a deviation measure (Dehaspe & Toivonen,
2001). In general it can be noted that relational data mining
poses many additional problems related to skewing of
data compared with traditional mining on a single table
(Jensen & Neville, 2002).
Relations that Represent a Graph
One type of relational data set has traditionally received
particular attention, albeit under a different name. A
relation representing a relationship between entity in-
stances of the same type, also called a reflexive relation-
ship, can be viewed as the definition of a graph. Graphs
have been used to represent social networks, biological
networks, communication networks, and citation graphs,
just to name a few.
A typical example of an association rule mining prob-
lem is mining of annotation data of proteins in the pres- A
ence of a protein-protein interaction graph (Oyama, Kitano,
Satou, & Ito, 2002). Associations are extracted that relate
functions and localizations of one protein with those of
interacting proteins. Oyama et al. use association rule
mining, as applied to joined relations, for this work.
Another example could be association rule mining of
attributes associated with scientific publications on the
graph of their mutual citations.
A problem of the straight-forward approach of mining
joined tables directly becomes obvious upon further
study of the rules: In most cases the output is dominated
by rules that involve the same item as it occurs in different
entity instances that participate in a relationship. In the
example of protein annotations within the protein interac-
tion graph a protein in the nucleus is found to fre-
quently interact with another protein that is also located
in the nucleus. Similarities among relational neighbors
have been observed more generally for relational data-
bases (Macskassy & Provost, 2003). It can be shown that
filtering of output is not a consistent solution to this
problem, and items that are repeated for multiple nodes
should be eliminated in a preprocessing step (Besemann
& Denton, 2004). This is an example of a problem that does
not occur in association rule mining of a single table and
requires special attention when moving to multiple rela-
tions. The example also highlights the need to discuss
differences between sets of items of related objects are
(Besemann, Denton, Yekkirala, Hutchison, & Anderson, 2004).
Related Research Areas
A related research area is graph-based ARM (Inokuchi,
Washio, & Motoda, 2000; Yan & Han, 2002). Graph-based
ARM does not typically consider more than one label on
each node or edge. The goal of graph-based ARM is to
find frequent substructures based on that one label,
focusing on algorithms that scale to large subgraphs. In
relational ARM multiple item are associated with each
node and the main problem is to achieve scaling with
respect to the number of items per node. Scaling to large
subgraphs is usually irrelevant due to the small world
property of many types of graphs. For most networks of
practical interest any node can be reached from almost
any other by means of no more than some small number
of edges (Barabasi & Bonabeau, 2003). Association rules
that involve longer distances are therefore unlikely to
produce meaningful results.
There are other areas of research on ARM in which
related transactions are mined in some combined fashion.
Sequential pattern or episode mining (Agrawal & Srikant,
1995; Yan, Han, & Afshar, 2003) and inter-transaction
mining (Tung, Lu, Han, & Feng, 1999) are two main
71
TEAM LinG

categories. Generally the interest in association rule min-
ing is moving beyond the single-table setting to incorpo-
rate the complex requirements of real-world data.
FUTURE TRENDS
The consensus in the data mining community of the impor-
tance of relational data mining was recently paraphrased
by Dietterich (2003) as I.i.d. learning is dead. Long live
relational learning. The statistics, machine learning, and
ultimately data mining communities have invested de-
cades into sound theories based on a single table. It is now
time to afford as much rigor to relational data. When taking
this step it is important to not only specify generalizations
of existing algorithms but to also identify novel questions
that may be asked that are specific to the relational setting.
It is, furthermore, important to identify challenges that
only occur in the relational setting, including skewing due
to the application of the relational join operator, and
correlations that are frequent in relational neighbors.
CONCLUSION
Association rule mining of relational data is a powerful
frequent pattern mining technique that is useful for several
data structures including graphs. Two main approaches
are distinguished. Inductive logic programming provides
a high degree of flexibility, while mining of joined relations
is a fast technique that allows the study problems related
to skewed or uninteresting results. The potential compu-
tational complexity of relational algorithms and specific
properties of relational data make its mining an important
current research topic. Association rule mining takes a
special role in this process, being one of the most impor-
tant frequent pattern algorithms.
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A.N. (1993, May).
Mining association rules between sets of items in large
databases. In Proceedings of the ACM International
Conference on Management of Data (pp. 207-216), Wash-
ington, D.C.
Agrawal, R., & Srikant, R. (1995). Mining sequential pat-
terns. In Proceedings of the 11 th International Conference
on Data Engineering (pp. 3-14), IEEE Computer Society
Press, Taipei, Taiwan.
Barabasi, A.L., & Bonabeau, E. (2003). Scale-free net-
works. Scientific American, 288(5), 60-69.
72
Association Rule Mining of Relational Data
Besemann, C., & Denton, A. (2004, June). UNIC: UNique
item counts for association rule mining in relational
data. Technical Report, North Dakota State University,
Fargo, North Dakota.
Besemann, C., Denton, A., Yekkirala, A., Hutchison, R.,
& Anderson, M. (2004, Aug.). Differential association
rule mining for the study of protein-protein interaction
networks. In Proceedings ACM SIGKDD Workshop on
Data Mining in Bioinformatics, Seattle, WA.
Brin, S., Motwani, R., Ullman, J.D., & Tsur, S. (1997).
Dynamic itemset counting and implication rules for mar-
ket basket data. In Proceedings of the ACM SIGMOD
International Conference on Management of Data,
Tucson, AZ.
Cristofor, L., & Simovici, D. (2001). Mining association
rules in entity-relationship modeled databases. Tech-
nical Report, University of Massachusetts Boston.
Dehaspe, L., & De Raedt, L. (1997, Dec.). Mining associa-
tion rules in multiple relations. In Proceedings of the 7th
International Workshop on Inductive Logic Program-
ming (pp. 125-132), Prague, Czech Republic.
Dehaspe, L., & Toivonen, H. (2001). Discovery of rela-
tional association rules. In S. D eroski, & N. Lavra
(Eds.), Relational data mining. Berlin: Springer.
Dietterich, T. (2003, Nov.). Sequential supervised learn-
ing: Methods for sequence labeling and segmentation.
Invited Talk, 3rd IEEE International Conference on
Data Mining, Melbourne, FL, USA.
D eroski, S., & Lavra , N. (2001). Relational data min-
ing. Berlin: Springer.
Inokuchi, A., Washio, T., & Motoda, H. (2000) An apriori-
based algorithm for mining frequent substructures from
graph data. In Proceedings of the 4th European Confer-
ence on Principles of Data Mining and Knowledge
Discovery (pp. 13-23), Lyon, France.
Jensen, D., & Neville, J. (2002). Linkage and
autocorrelation cause feature selection bias in relational
learning. In Proceedings of the 19th International Con-
ference on Machine Learning (pp. 259-266), Sydney,
Australia.
Macskassy, S., & Provost, F. (2003). A simple relational
classifier. In Proceedings of the 2nd Workshop on Multi-
Relational Data Mining at KDD03, Washington, D.C.
Oyama, T., Kitano, K., Satou, K., & Ito, T. (2002). Extrac-
tion of knowledge on protein-protein interaction by as-
sociation rule discovery. Bioinformatics, 18(8), 705-714.
TEAM LinG
Association Rule Mining of Relational Data
Tung, A.K.H., Lu, H., Han, J., & Feng, L. (1999). Breaking
the barrier of transactions: Mining inter-transaction asso-
ciation rules. In Proceedings of the International Confer-
ence on Knowledge Discovery and Data Mining, San
Diego, CA.
Yan, X., & Han, J. (2002). gSpan: Graph-based substruc-
ture pattern mining. In Proceedings of the International
Conference on Data Mining, Maebashi City, Japan.
Yan, X., Han, J., & Afshar, R. (2003). CloSpan: Mining
closed sequential patterns in large datasets. In Proceed-
ings of the 2003 SIAM International Conference on Data
Mining, San Francisco, CA.
Zaki, M.J. (2000). Generating non-redundant association
rules. In Proceedings of the International Conference on
Knowledge Discovery and Data Mining (pp. 34-43), Boston,
MA.
KEY TERMS
Antecedent: The set of items A in the association rule
A B.
Apriori: Association rule mining algorithm that uses
the fact that the support of a non-empty subset of an item
set cannot be smaller than the support of the item set itself.
Association Rule: A rule of the form A B meaning
if the set of items A is present in a transaction, then the
set of items B is likely to be present too. A typical example
constitutes associations between items purchased at a
supermarket.
Confidence: The confidence of a rule is the support
of the item set consisting of all items in the rule (A ) A
divided by the support of the antecedent.
Entity-Relationship Model (E-R-Model): A model to
represent real-world requirements through entities, their
attributes, and a variety of relationships between them. E-
R-Models can be mapped automatically to the relational
model.
Inductive Logic Programming (ILP): Research area
at the interface of machine learning and logic program-
ming. Predicate descriptions are derived from examples
and background knowledge. All examples, background
knowledge and final descriptions are represented as logic
programs.
Redundant Association Rule: An association rule is
redundant if it can be explained based entirely on one or
more other rules.
Relation: A mathematical structure similar to a table in
which every row is unique, and neither rows nor columns
have a meaningful order.
Relational Database: A database that has relations
and relational algebra operations as underlying math-
ematical concepts. All relational algebra operations result
in relations as output. A join operation is used to combine
relations. The concept of a relational database was
introduced by E. F. Codd at IBM in 1970.
Support: The support of an item set is the fraction of
transactions that have all items in that item set.
73
TEAM LinG
74
Association Rules and Statistics
Martine Cadot
University of Henri Poincar/LORIA, Nancy, France
Jean-Baptiste Maj
LORIA/INRIA, France
Tarek Ziad
NUXEO, France
INTRODUCTION
A manager would like to have a dashboard of his company
without manipulating data. Usually, statistics have solved
this challenge, but nowadays, data have changed (Jensen,
1992); their size has increased, and they are badly struc-
tured (Han & Kamber, 2001). A recent methoddata
mininghas been developed to analyze this type of data
(Piatetski-Shapiro, 2000). A specific method of data min-
ing, which fits the goal of the manager, is the extraction of
association rules (Hand, Mannila & Smyth, 2001). This
extraction is a part of attribute-oriented induction (Guyon
& Elisseeff, 2003).
The aim of this paper is to compare both types of
extracted knowledge: association rules and results of
statistics.
BACKGROUND
Statistics have been used by people who want to extract
knowledge from data for one century (Freeman, 1997).
Statistics can describe, summarize and represent the data.
In this paper data are structured in tables, where lines are
called objects, subjects or transactions and columns are
called variables, properties or attributes. For a specific
variable, the value of an object can have different types:
quantitative, ordinal, qualitative or binary. Furthermore,
statistics tell if an effect is significant or not. They are
called inferential statistics.
Data mining (Srikant, 2001) has been developed to
precede a huge amount of data, which is the result of
progress in digital data acquisition, storage technology,
and computational power. The association rules, which
are produced by data-mining methods, express links on
database attributes. The knowledge brought by the asso-
ciation rules is shared in two different parts. The first
describes general links, and the second finds specific
links (knowledge nuggets) (Fabris & Freitas, 1999;
Padmanabhan & Tuzhilin, 2000). In this article, only the
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
first part is discussed and compared to statistics. Further-
more, in this article, only data structured in tables are used
for association rules.
MAIN THRUST
The problem differs with the number of variables. In the
sequel, problems with two, three, or more variables are
discussed.
Two Variables
The link between two variables (A and B) depends on the
coding. The outcome of statistics is better when data are
quantitative. A current model is linear regression. For
instance, the salary (S) of a worker can be expressed by the
following equation:
S = 100 Y + 20000 + (1)
where Y is the number of years in the company, and is
a random number. This model means that the salary of a
newcomer in the company is $20,000 and increases by
$100 per year.
The association rule for this model is: YS. This
means that there are a few senior workers with a small
paycheck. For this, the variables are translated into binary
variables. Y is not the number of years, but the property
has seniority, which is not quantitative but of type Yes/
No. The same transformation is applied to the salary S,
which becomes the property has a big salary.
Therefore, these two methods both provide the link
between the two variables and have their own instruments
for measuring the quality of the link. For statistics, there
are the tests of regression model (Baillargeon, 1996), and
for association rules, there are measures like support,
confidence, and so forth (Kodratoff, 2001). But, depend-
ing on the type of data, one model is more appropriate than
the other (Figure 1).
TEAM LinG
Association Rules and Statistics

Figure 1. Coding and analysis methods variable has a particular effect on the link between Y and
S, called interaction (Winer, Brown & Michels, 1991). A
Quantitative Ordinal Qualitative Yes/No The association rules for this model are:

- Association Rules + YS, ES for the equation (2)

+ Statistics -
Three Variables
If a third variable E, the experience of the worker, is
integrated, the equation (1) becomes:
S = 100 Y + 2000 E + 19000 +
E is the property has experience. If E=1, a new
experienced worker gets a salary of $21,000, and if E=0, a
new non-experienced worker gets a salary of $19,000. The
increase of the salary, as a function of seniority (Y), is the
same in both cases of experience.
S = 50 Y + 1500 E + 50 E Y + 19500 +
Now, if E=1, a new experienced worker gets a salary of
$21,000, and if E=0, a new non-experienced worker gets a
salary of $19,500. The increase of the salary, as a function
of seniority (Y), is $50 higher for experienced workers.
These regression models belong to a linear model of
statistics (Prum, 1996), where, in the equation (3), the third
YS, ES, YES for the equation (3)
The statistical test of the regression model allows to
choose with or without interaction (2) or (3). For the
association rules, it is necessary to prune the set of three
rules, because their measures do not give the choice
between a model of two rules and a model of three rules
(Zaki, 2000; Zhu, 1998).
More Variables
(2)
With more variables, it is difficult to use statistical models
to test the link between variables (Megiddo & Srikant,
1998). However, there are still some ways to group vari-
ables: clustering, factor analysis, and taxonomy (Govaert,
2003). But the complex links between variables, like inter-
actions, are not given by these models and decrease the
quality of the results.
(3)
Comparison
Table 1 briefly compares statistics with the association
rules. Two types of statistics are described: by tests and
by taxonomy. Statistical tests are applied to a small amount
of variables and the taxonomy to a great amount of

Table 1. Comparison between statistics and association rules

Statistics Data Mining

Tests Taxonomy Association rules
Decision Tests (+) Threshold defined (-) Threshold defined (-)
Level of Knowledge Low (-) High and simple (+) High and complex (+)
No. of Variables Small (-) High (+) Small and high (+)
Complex Link Yes (-) No (+) No (-)

Figure 2. (a) Regression equations; (b) taxonomy; (c) association rules

a) b) c)

75

TEAM LinG

variables. In statistics, the decision is easy to make out of
test results, unlike association rules, where a difficult
choice on several indices thresholds has to be performed.
For the level of knowledge, the statistical results need
more interpretation relative to the taxonomy and the asso-
ciation rules.
Finally, graphs of the regression equations (Hayduk,
1987), taxonomy (Foucart, 1997), and association rules
(Gras & Bailleul, 2001) are depicted in Figure 2.
FUTURE TRENDS
With association rules, some researchers try to find the
right indices and thresholds with stochastic methods.
More development needs to be done in this area. Another
sensitive problem is the set of association rules that is not
made for deductive reasoning. One of the most common
solutions is the pruning to suppress redundancies, con-
tradictions and loss of transitivity. Pruning is a new method
and needs to be developed.
CONCLUSION
With association rules, the manager can have a fully
detailed dashboard of his or her company without manipu-
lating data. The advantage of the set of association rules
relative to statistics is a high level of knowledge. This
means that the manager does not have the inconvenience
of reading tables of numbers and making interpretations.
Furthermore, the manager can find knowledge nuggets
that are not present in statistics.
The association rules have some inconvenience; how-
ever, it is a new method that still needs to be developed.
REFERENCES
Baillargeon, G. (1996). Mthodes statistiques de
lingnieur: Vol. 2. Trois-Riveres, Quebec: Editions SMG.
Fabris,C., & Freitas, A. (1999). Discovery surprising pat-
terns by detecting occurrences of Simpsons paradox:
Research and development in intelligent systems XVI.
Proceedings of the 19th Conference of Knowledge-Based
Systems and Applied Artificial Intelligence, Cambridge,
UK.
Foucart, T. (1997). Lanalyse des donnes, mode demploi. McGraw-Hill.
Rennes, France: Presses Universitaires de Rennes.
Freedman, D. (1997). Statistics. W.W. New York: Norton &
Company.
76
Association Rules and Statistics
Govaert, G. (2003). Analyse de donnes. Lavoisier, France:
Hermes-Science.
Gras, R., & Bailleul, M. (2001). La fouille dans les donnes
par la mthode danalyse statistique implicative.
Colloque de Caen. Ecole polytechnique de lUniversit
de Nantes, Nantes, France.
Guyon, I., & Elisseeff, A. (2003). An introduction to
variable and feature selection: Special issue on variable
and feature selection. Journal of Machine Learning
Research, 3, 1157-1182.
Han, J., & Kamber, M. (2001). Data mining: Concepts
and techniques. San Francisco, CA: Morgan Kaufmann.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of
data mining. Cambridge, MA: MIT Press.
Hayduk, L.A. (1987). Structural equation modelling
with LISREL. Maryland: John Hopkins Press.
Jensen, D. (1992). Induction with randomization test-
ing: Decision-oriented analysis of large data sets [doc-
toral thesis]. Washington University, Saint Louis, MO.
Kodratoff, Y. (2001). Rating the interest of rules induced
from data and within texts. Proceedings of the 12th IEEE
International Conference on Database and Expert Sys-
tems Aplications-Dexa, Munich, Germany.
Megiddo, N., & Srikant, R. (1998). Discovering predictive
association rules. Proceedings of the Conference on
Knowledge Discovery in Data, New York.
Padmanabhan, B., & Tuzhilin, A. (2000). Small is beauti-
ful: Discovering the minimal set of unexpected patterns.
Proceedings of the Conference on Knowledge Discov-
ery in Data. Boston, Massachusetts.
Piatetski-Shapiro, G. (2000). Knowledge discovery in da-
tabases: 10 years after. Proceedings of the Conference on
Knowledge Discovery in Data, Boston, Massachusetts.
Prum, B. (1996). Modle linaire: Comparaison de
groupes et rgression. Paris, France: INSERM.
Srikant, R. (2001). Association rules: Past, present, fu-
ture. Proceedings of the Workshop on Concept Lattice-
Based Theory, Methods and Tools for Knowledge Dis-
covery in Databases, California.
Winer, B.J., Brown, D.R., & Michels, K.M.(1991). Statis-
tical principles in experimental design. New York:
Zaki, M.J. (2000). Generating non-redundant association
rules. Proceedings of the Conference on Knowledge
Discovery in Data, Boston, Massachusetts.
TEAM LinG
Association Rules and Statistics
Zhu, H. (1998). On-line analytical mining of association
rules [doctoral thesis]. Simon Fraser University, Burnaby,
Canada.
KEY TERMS
Attribute-Oriented Induction: Association rules, clas-
sification rules, and characterization rules are written with
attributes (i.e., variables). These rules are obtained from
data by induction and not from theory by deduction.
Badly Structured Data: Data, like texts of corpus or
log sessions, often do not contain explicit variables. To
extract association rules, it is necessary to create vari-
ables (e.g., keyword) after defining their values (fre-
quency of apparition in corpus texts or simply apparition/
non apparition).
Interaction: Two variables, A and B, are in interaction
if their actions are not seperate.
Linear Model: A variable is fitted by a linear combina-
tion of other variables and interactions between them. A
Pruning: The algorithms of extraction for the associa-
tion rule are optimized in computationality cost but not in
other constraints. This is why a suppression has to be
performed on the results that do not satisfy special
constraints.
Structural Equations: System of several regression
equations with numerous possibilities. For instance, a
same variable can be made into different equations, and
a latent (not defined in data) variable can be accepted.
Taxonomy: This belongs to clustering methods and is
usually represented by a tree. Often used in life categori-
zation.
Tests of Regression Model: Regression models and
analysis of variance models have numerous hypothesis,
e.g. normal distribution of errors. These constraints allow
to determine if a coefficient of regression equation can be
considered as null with a fixed level of significance.
77
TEAM LinG
78
Automated Anomaly Detection
Brad Morantz
Georgia State University, USA
INTRODUCTION
Preparing a dataset is a very important step in data mining.
If the input to the process contains problems, noise, or
errors, then the results will reflect this, as well. Not all
possible combinations of the data should exist, as the data
represent real-world observations. Correlation is expected
among the variables. If all possible combinations were
represented, then there would be no knowledge to be
gained from the mining process.
The goal of anomaly detection is to identify and/or
remove questionable or incorrect observations. These
occur because of keyboard error, measurement or record-
ing error, human mistakes, or other causes. Using knowl-
edge about the data, some standard statistical techniques,
and a little programming, a simple data-scrubbing program
can be written that identifies or removes faulty records.
Duplicates can be eliminated, because they contribute no
new knowledge. Real valued variables could be within
measurement error or tolerance of each other, yet each
could represent a unique rule. Statistically categorizing the
data would eliminate or, at least, greatly reduce this.
In application of this process with actual datasets,
accuracy has been increased significantly, in some cases
double or more.
BACKGROUND
Data mining is an exploratory process looking for as yet
unknown patterns (Westphal & Blaxton, 1998). The data
represent real-world occurrences, and there is correlation
among the variables. Some are principled in their con-
struction, one event triggering another. Sometimes events
occur in a certain order (Westphal & Blaxton, 1998). Not
all possible combinations of the data are to be expected.
If this were not the case, then we would learn nothing from
this data. These methods allow us to see patterns and
regularities in large datasets (Mitchell, 1999).
Credit reporting agencies have been examining large
datasets of credit histories for quite some time, trying to
determine rules that will help discern between problematic
and responsible consumers (Mitchell, 1999). Datasets
have been mined looking for indications for boiler explo-
sion probabilities to high-risk pregnancies to consumer
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
purchasing patterns. This is the semiotics of data, as we
transform data to information and finally to knowledge.
Dirty data, or data containing errors, are a major
problem in this process. The old saying is, garbage in,
garbage out (Statsoft, 2004). Heuristic estimates are that
60-80% of the effort should go into preparing the data for
mining, and only the small remaining portion actually is
required for the data-mining effort itself. These data
records that are deviations from the common rule are
called anomalies.
Data are always dirty and have been called the curse
of data mining (Berry & Linoff, 2000). Several factors can
be responsible for attenuating the quality of the data,
among them errors, missing values, and outliers (Webb,
2002). Missing data have many causes, varying from
recording error to illegible writing to just not supplied.
This is closely related to incorrect values that also can be
caused by poor penmanship as well as measurement error,
keypunch mistakes, different or incorrect metrics, mis-
placed decimal, and other similar causes.
Fuzzy definitions, where the meaning of a value is
either unclear or inconsistent, are another problem (Berry
& Linoff, 2000). Often, when something is being measured
and recorded, mistakes happen. Even automated processes
can produce dirty data (Bloom, 1998). Micro-array data has
errors due to base pairs on the probe not matching correctly
to genes in the test material (Shavlik et al., 2004). The
sources of error are large, and it is necessary to have a
process that finds these anomalies and identifies them.
In real valued datasets, the possible combinations are
(almost) unlimited. A dataset with eight variables, each
with four significant digits, could yield as many as 1032
combinations. Mining such a dataset would not only be
tedious and time-consuming, but possibly could yield an
overly large number of patterns. Using (six-range) cat-
egorical data, the same problem would only have 1.67 x 106
combinations. Gauss normally distributed data can be
separated into plus or minus 1, 2, or 3 sigma. Other
distributions can use Chebyshev or other distributions
with similar dividing points. There is no real loss of data,
yet the process is greatly simplified.
Finding the potentially bad observations or records
is the first problem. The second problem is what to do once
they are found. In many cases it is possible to go back and
verify the value, correcting it, if necessary. If this is
TEAM LinG
Automated Anomaly Detection
possible, the program should flag values that are to be
verified. This may not always be possible, or it may be too
expensive. Not all situations repeat within a reasonable
time, if at all (i.e., observation of Halleys comet).
There are two schools of thought, the first being to
substitute the mean value for the missing or wrong value.
The problem with this is that it might not be a reasonable
value, and it can create a new rule, one that could be false
(i.e., shoe size for a giant is not average). It might introduce
sample bias, as well (Berry & Linoff, 2000).
Deleting the observation is the other common solu-
tion. Quite often, in large datasets, a duplicate exists, so
deleting causes no loss. The cost of improper commission
is greater than that of omission. Sometimes an outlier tells
a story. So, one has to be careful about deletions.
THE AUTOMATED ANOMALY
DETECTION PROCESS
Methodology
To illustrate the process, a public dataset is used. This
particular one is available from the University of California
at Irvine Machine Learning Repository (University of Cali-
fornia, 2003). Known as the Abalone dataset, it consists of
4,400 observations of abalones that were captured in the
wild with several measurements of each one. Natural varia-
tion exists, as well as human error, both in making the
measurements and in the recording. Also listed on the Web
site were some studies that used the data and their results.
Accuracy in the form of hit rate varied between 0-35%.
While it may seem overly simple and obvious, plot-
ting the data is the first step. These graphical views can
provide much insight into the data (Webb, 2002). The data
for each variable can be plotted vs. frequency of occur-
rence to visually determine distribution. Combining this
with knowledge of the research will help to determine the
correct distribution to use for each included variable. A
sum of independent terms would tend to support a Gauss
normal distribution, while the product of a number of
independent terms might suggest using log normal. This
plotting also might suggest necessary transformations.
It is necessary to understand the acceptable range for
each field. Some values obtained might not be reasonable.
If there is a zero in a field, is it indicative of a missing value,
or is it an acceptable value? No value is not the same as
zero. Some values, while within bounds, might not be
possible. It is also necessary to check for obvious mis-
takes, inconsistencies, or out of bounds.
Knowledge about the subject of study is necessary.
From this, rules can be made. In the example of the abalone,
the animal in the shell must weigh more than when it is
shucked (removed from the shell) for obvious reasons.
Other such rules from domain knowledge can be created A
(abalone.net, 2004; University of Capetown, 2004; World
Aquaculture, 2004). Sometimes, they may seem too obvi-
ous, but they are effective. The rules can be programmed
into a subroutine specific to the dataset.
Regression can be used to check for variables that are
not statistically significant. Step-wise regression is a
handy tool for identifying significant variables. Other
ratio variables can be created and then checked for signifi-
cance using regression. Again, domain knowledge can
help create these variables, as well as insight and some
luck. Insignificant variables can be deleted from the
dataset, and new ones can be added.
If the dataset is real valued, it is possible that records
exist that are within tolerance or measurement error of
each other. There are two ways to reduce the number of
unique observations. (1) Attenuate the accuracy by round-
ing to reduce the number of significant digits. Each
variable rounding to one less significant digit reduces the
number of possible patterns by an order of magnitude. (2)
Calculate a mean and standard deviation for the cleaned
dataset. Using an appropriate distribution, sort the values
by standard deviations from the mean. Testing to see if the
chosen distribution is correct is accomplished by using a
Chi square test, a Kolmogorof Smirnoff test, or the empiri-
cal test. The number of standard deviations replaces the
real valued data, and a simple categorical dataset will exist.
This allows for simple comparisons between observa-
tions. Otherwise, records with values as little as .0001%
differences would be considered unique and different.
While some of the precision of the original data is lost, this
process is exploratory and finds the general patterns that
are in the data. This allows one to gain insight into the
database using a combination of statistics and artificial
intelligence (Pazzani, 2000), using human knowledge and
skill as the catalyst to improve the results.
The final step before mining the data is to remove
duplicates, as they add no additional information. As the
collection of observations gets increasingly larger, it gets
harder to introduce new experiences. This process can be
incorporated into the computer program by a simple
process that is similar to bubblesort. Instead of comparing
to see which row is greater, it just looks for differences. If
none are found, then the row is deleted.
Example Results
A few variables were plotted producing, some very un-
usual graphs. These were definitely not the graphs that
were expected. This was the first indication that the
dataset was noisy. Abalones are born in very large num-
bers, but with an extremely high infant mortality rate (over
79
TEAM LinG
 Automated Anomaly Detection

99%) (Bamfield Marine Science Centre, 2004). This graph
did not reflect that.
An initial scan of the data showed some inconsistent
points, like a five-year-old infant, a shucked animal weigh-
ing more than a complete one, and other similar abnormali-
ties. Another problem with most analyses of these datasets
is that gender is not ratio or ordinal data and, therefore, had
to be converted to a dummy variable.
Step-wise regression removed all but five variables.
The remaining variables were: diameter, height, whole
weight, shucked weight, and viscera weight. Two new
variables were created: shell ratio (whole weight divided
by shell weight) and weight to diameter ratio. Since the
diameter is directly proportional to volume, this variable is
proportional to density. The proof of its significance was
a t value of 39 and an F value of 1561. These are both
statistically significant. A plot of shell ratio vs. frequency
yielded a fairly Gauss normal looking curve.
As these are real valued data with four digits given, it
is possible to have observations that vary by as little as
0.01%. This value is even less than the accuracy of the
measuring instruments. In other words, there are really a
relatively small number of possibilities, described by a
large number of almost identical examples, some within
measurement tolerance of each other.
The mean and standard deviation were calculated for
each of the remaining and new variables of the dataset. The
empirical test was done to verify approximate meeting of
Gauss normal distribution. Each value then was replaced
by the integer number of standard deviations it is from the
mean, creating a categorical dataset. Simple visual inspec-
tion showed two things: (1) there was, indeed, correlation
among the observations; and (2) it became increasingly
more difficult to introduce a new pattern.
Duplicate removal process was the next step. As
expected, the first 50 observations only had 22% dupli-
cates, but by the time the entire dataset was processed,
65% of the records were removed, because it presented no
new information.
To better understand the quality of the data, least
squares regression was performed. The model produced
an ANOVA F value of 22.4, showing good confidence in
it. But the Pearsonian correlation coefficient R2 of only 0.25
indicated that there was some problem. Visual observation
of the dataset and its plots led to some suspicion of the
group with one ring (age = 2.5 years). OLS regression was
performed on this group, yielding an F of 27, but an R 2 of
only 0.03. This tells us that this portion of the data is only
muddying the water and attenuating the performance of
our model.
Upon removal of this group of observations, OLS
regression was performed on the remaining data, giving
an improved F of 639 (showing that, indeed, it is a good
model) and an R2 of 0.53, an acceptable level and one that
can adequately describe the variation in the criterion.
The results listed at the Web site where the dataset
was obtained are as follows:
Sam Waugh in the Computer Science Department at
the University of Tasmania used this dataset in 1995 for
his doctoral dissertation (University of California, 2003).
His results, while the first recorded attempt, did not have
good accuracy at predicting the age. The problem was
encoded as a classification task.
24.86% Cascade Correlation (no hidden nodes)
26.25% Cascade Correlation (five hidden
nodes)
21.5% C4.5
0.0% Linear Discriminant Analysis
3.57% k=5 Nearest Neighbor
Clark, et al. (1996) did further work on this dataset.
They split the ring classification into three groups: 1 to
8, 9 to 10, and 11 and up. This reduced the number of
targets and made each one bigger, in effect making each
easier to hit.
Their results were much better, as shown in the
following:
64% Back propagation
55% Dystal
The results obtained from the answer tree using the
new cleaned dataset are shown in Table 1.
All of the one-ring observations were filtered out in
a previous step, and the extraction was 100% accurate in
not predicting any as being one-ring. The hit rates are as
follows:

Table 1. Hit Rate

Actual Category
Predicted 1 2 3 4 Total
1 0 0 0 0 0
2 11 269 79 2 361
3 231 140 953 280 1604
4 22 3 27 81 133
Total 264 412 1059 363 2098

80

TEAM LinG
Automated Anomaly Detection
1 ring 100.0% correct
2 ring 74.5% correct
3 ring 59.4% correct
4 ring 60.9% correct
Overall accuracy 62.1% correct
FUTURE TRENDS
A program could be written that would input simple things
like the number of variables, the number of observations,
and some classification results. A rule input mechanism
would accept the domain-specific rules and make them
part of the analysis. Further improvements would be the
inclusion of fuzzy logic. Type I would allow the use of
lingual variables (i.e., big, small, hot, cold) in the records,
and type II would allow for some fuzzy overlap and fit.
CONCLUSION
Data mining is an exploratory process to see what is in the
data and what patterns can be found. Noise and errors in
the dataset are reflected in the results from the mining
process. Cleaning the data and identifying anomalies
should be performed. Marked observations should be
verified and corrected, if possible. If this cannot be done,
they should be deleted. In real valued datasets, the values
can be categorized with accepted statistical techniques.
Anomaly detection, after some manual viewing and analy-
sis, can be automated. Part of the process is specific to the
knowledge domain of the dataset, and part could be
standardized. In our example problem, this cleaning pro-
cess improved results, and the mining produced a more
accurate rule set.
REFERENCES
Abalone.net. (2003). All about abalone: An online guide.
Retrieved from http://www.abalone.net
Bamfield Marine Sciences Centre Public Education
Programme. (2004). Oceanlink. Retrieved from http://
oceanlink.island.net/oinfo/Abalone/abalone.html
Berry, M.J.A., & Linoff, G.S. (2000). Mastering data min-
ing: The art and science of customer relationship man-
agement. New York, NY: Wiley & Sons, Inc.
Bloom, D. (1998). Technology, experimentation, and the
quality of survey data. Science, 280(5365), 847-848.
Clark, D., Schreter, Z., & Adams, A. (1996). A quantitative
comparison of dystal and backpropagation. Proceedings
of the Australian Conference on Neural Networks (ACNN
96), Canberra, Australia. A
Mitchell, T. (1999). Machine learning and data mining.
Communications of the ACM, 42(11), 30-36.
Pazzani, M.J. (2000). Knowledge discovery from data?
IEEE Intelligent Systems, 15(2), 10-13.
Shavlik, J., Molla, M., Waddell, M., & Page, D. (2004).
Using machine learning to design and interpret gene-
expression microarrays. American Association for Artifi-
cial Intelligence, 25(1), 23-44.
Statsoft, Inc. (2004). Electronic statistics textbook. Re-
trieved from http://www.statsoft.com
Tasmanian Abalone Council Ltd. (2004). http://
tasabalone.com.au
University of California at Irvine. (2003). Machine learn-
ing repository, abalone database. Retrieved from http:/
/ics.uci.edu/~mlearn/MLRepository
University of Capetown Zoology Department. (2004).
http://web.uct.ac.za/depts/zoology/abnet
Webb, A. (2002). Statistical pattern recognition. West
Sussex, England: Wiley & Sons.
Westphal, C., & Blaxton, T. (1998). Data mining solutions
methods and tools for solving real-world problems. New
York: Wiley & Sons.
World Aquaculture. (2004). http://www7.taosnet.com/
platinum/data/light/species/abalone.html
KEY TERMS
Anomaly: A value or observation that deviates from
the rule or analogy. A potentially incorrect value.
ANOVA or Analysis of Variance: A powerful statis-
tical method for studying the relationship between a
response or criterion variable and a set of one or more
predictor or independent variable(s).
Correlation: Amount of relationship between two
variables, how they change relative to each other, range:
-1 to +1.
F Value: Fisher value, a statistical distribution, used
here to indicate the probability that an ANOVA model is
good. In the ANOVA calculations, it is the ratio of squared
variances. A large number translates to confidence in the
model.
81
TEAM LinG

Ordinal Data: Data that is in order but has no relation-
ship between the values or to an external value.
Pearsonian Correlation Coefficient: Defines how
much of the variation in the criterion variable(s) is caused
by the model. Range: 0 to 1.
Ratio Data: Data that is in order and has fixed spacing;
a relationship between the points that is relative to a fixed
external point.
82
Automated Anomaly Detection
Step-Wise Regression: An automated procedure on
statistical programs that adds one predictor variable at a
time, and if it is not statistically significant, it removes it
from the model. Some work in both directions either by
adding or removing from the model, one at a time.
T Value, also called Students t: A statistical
distribution for smaller sample sizes. In regression rou-
tines in statistical programs, it indicates whether a predic-
tor variable is statistically significant or if it truly is
contributing to the model. A value more than about 3 is
required for this indication.
TEAM LinG

Automatic Musical Instrument Sound
Classification
Alicja A. Wieczorkowska
Polish-Japanese Institute of Information Technology, Poland
INTRODUCTION
The aim of musical instrument sound classification is to
process information from audio files by a classificatory
system and accurately identify musical instruments play-
ing the processed sounds. This operation and its results
are called automatic classification of musical instru-
ment sounds.
BACKGROUND
Musical instruments are grouped into the following
categories (Hornbostel & Sachs, 1914):
Idiophones: Made of solid, non-stretchable, so-
norous material.
Membranophones: Skin drums; membranophones,
and idiophones are called percussion.
Chordophones: Stringed instruments.
Aerophones: Wind instruments: woodwinds
(single-reed, double-reed, flutes) and brass (lip-
vibrated)
Idiophones are classified according to the material,
number of idiophones and resonators in a single instru-
ment, and whether pitch or tuning is important. Subcat-
egories include idiophones struck together by concus-
sion (e.g., castanets), struck (gong), rubbed (musical
glasses), scraped (washboards), stamped (hard floors
stamped with tap shoes), shaken (rattles), and plucked
(jews harp).
Membranophones are classified according to their
shape, material, number of heads, if they have snares, etc.,
whether and how the drum is tuned, and how the skin is
fixed and played. Subcategories include drums (cylindri-
cal, conical, barrel, hourglass, goblet, footed, long, kettle,
frame, friction drum, and mirliton/kazoo).
Chordophones are classified with respect to the
relationship of the strings to the body of the instrument,
if they have frets (low bridges on the neck or body,
where strings are stopped) or movable bridges, number
of strings, and how they are played and tuned. Subcat-
egories include zither, lute plucked, and bowed (e.g.,
guitars, violin), harp, lyre, and bow.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
83
A
Aerophones are classified according to how the air is
set in motion, mainly depending on the mouthpiece: blow
hole, whistle, reed, and lip-vibrated. Subcategories in-
clude flutes (end-blown, side-blown, nose, globular,
multiple), panpipes, whistle mouthpiece (recorder), single-
and double-reed (clarinet, oboe), air chamber (pipe or-
gans), lip-vibrated (trumpet or horn), and free aerophone
(bullroarers) (SIL, 1999).
The description of properties of musical instrument
sounds is usually given in vague subjective terms, like
sharp, nasal, bright, and so forth, and only some of them
(i.e., brightness) have numerical counterparts. There-
fore, one of the main problems in this research is to
prepare the appropriate numerical sound description for
instrument recognition purposes.
Automatic classification of musical instrument
sounds aims at classifying audio data accurately into
appropriate groups representing instruments. This clas-
sification can be performed at instrument level, instru-
ment family level (e.g., brass), or articulation (i.e., how
sound is struck, sustained, and released, e.g. vibrato -
varying the pitch of a note up and down) (Smith, 2000). As
a preprocessing, the audio data are usually parameterized
(i.e., numerical or other parameters or attributes are as-
signed, and then data mining techniques are applied to the
parameterized data). Accuracy of classification varies,
depending on the audio data used in the experiments,
number of instruments, parameterization, classification,
and validation procedure applied. Automatic classifica-
tion compares favorably with human performance. Listen-
ers identify musical instruments with accuracy far from
perfect, with results depending on the sounds chosen and
experience of listeners. Classification systems allow in-
strument identification without participation of human
experts. Therefore, such systems can be valuable assis-
tance for users of audio data searching for specific timbre,
especially if they are not experienced musicians and when
the amount of available audio data is huge, thus making
manual searching impractical, if possible at all. When
combined with a melody-searching system, automatic
instrument classification may provide a handy tool for
finding favorite tunes performed by favorite instruments
in audio databases.
TEAM LinG

MAIN THRUST
Research on automatic classification of musical instru-
ments so far has been performed mainly on isolated,
singular sounds; works on polyphonic sounds usually
aim at source separation and operations like pitch track-
ing of these sounds (Viste & Evangelista, 2003). Most
commonly used data include MUMS compact discs
(Opolko & Wapnick, 1987), the University of Iowa
samples (Fritts, 1997), and IRCAMs Studio on Line
(IRCAM, 2003).
A broad range of data mining techniques was applied
in this research, aiming at extraction of information
hidden in audio data (i.e., sound features that are com-
mon for a given instrument and differentiate it from the
others). Descriptions of musical instrument sounds are
usually subjective, and finding appropriate numeric de-
scriptors (parameters) is a challenging task. Sound pa-
rameterization is arbitrarily chosen by the researchers,
and the parameters may reflect features that are known
to be important for the human in the instrument recog-
nition task, like descriptors of sound evolution in time
(i.e., onset features, depth of sound vibration, etc.),
subjective timbre features (i.e., brightness of the sound),
and so forth. Basically, parameters characterize coeffi-
cients of sound analysis, since they are relatively easy
to calculate.
On the basis of parameterization, further research
can be performed. Clustering applied to parameter vec-
tors reveals similarity among sounds and adds a new
glance on instrument classification, usually based on
instrument construction or sound articulation. Deci-
sion rules and trees allow identification of the most
descriptive sound features. Transformation of sound pa-
rameters may produce new descriptors better suited for
automatic instrument classification. The classification
can be performed hierarchically, taking instrument fami-
lies and articulation into account. Classifiers represent a
broad range of methods, from simple statistic tools to
new advanced algorithms rooted in artificial intelligence.
Parameterization Methods
Sound is a physical disturbance in the medium (e.g., air)
through which it is propagated (Whitaker & Benson,
2002). Periodic fluctuations are perceived as sound
having pitch. The audible frequency range is about 20-
20,000 Hz (hertz or cycles per second). The parameter-
ization aims at capturing most distinctive sound fea-
tures regarding sound amplitude evolution in time, static
spectral features (frequency contents) of the most stable
part of the sound, and evolution of frequency content in
time. These features are based on the Fourier spectrum
and time-frequency sound representations like wavelet
84
Automatic Musical Instrument Sound Classification
transform. Some analyses are adjusted to the properties of
human hearing, which perceives changes of sound ampli-
tude and frequency in a logarithmic-like manner (e.g.,
frequency contents analysis in mel scale). The results
based on such analysis are easier to interpret in subjective
terms. Also, statistic and mathematic operations are ap-
plied to the sound representation, yielding good results,
too. Some descriptors require calculating pitch of the
sound, and any inaccuracies in pitch calculation (e.g.,
octave errors) may lead to erroneous results.
Parameter sets investigated in the research are usu-
ally a mixture of various types, since such combinations
allow capturing more representative sound description
for instrument classification purposes.
The following analysis and parameterization meth-
ods are used to describe musical instrument sounds:
Autocorrelation and cross-correlation functions
investigating periodicity of the signal and statisti-
cal parameters of spectrum obtained via Fourier
transform: average amplitude and frequency varia-
tions (wide in vibrated sounds), standard devia-
tions (Ando & Yamaguchi, 1993).
Contents of selected groups of partials in the
spectrum (Pollard & Jansson, 1982;
Wieczorkowska, 1999a), including amount of even
and odd harmonics (Martin & Kim, 1998), allow-
ing identification of clarinet sounds.
Vibrato strength and other changes of sound fea-
tures in time (Martin & Kim, 1998; Wieczorkowska
et al., 2003) and temporal envelope of the sound.
Statistical moments of the time wave, spectral
centroid (gravity center), coefficients of cepstrum
(i.e., the Fourier transform applied to the loga-
rithm of amplitude plot of the spectrum), con-
stant-Q coefficients (i.e., for logarithmically-
spaced spectral bins) (Brown, 1999; Brown et al.,
2001).
Wavelet analysis, providing time-frequency plot
based on decomposition of sound signal into func-
tions called wavelets (Kostek & Czyzewski, 2001;
Wieczorkowska, 1999b).
Mel-frequency coefficients (i.e., in mel scale,
adjusted to the properties of human hearing) and
linear prediction cepstral coefficients, where fu-
ture values are estimated as a linear function of
previous values (Eronen, 2001).
Multidimensional Scaling Analysis (MSA) trajec-
tories obtained through Principal Component
Analysis (PCA) applied to the constant-Q spectral
snapshots to determine the most significant at-
tributes of each sound (Kaminskyj, 2002). PCA
transforms a set of variables into a smaller set of
TEAM LinG
Automatic Musical Instrument Sound Classification
uncorrelated variables, which keep as much of the
variability in the data as possible.
MPEG-7 audio descriptors, including log-attack time
(i.e., logarithm of onset duration, fundamental fre-
quencypitch, spectral envelope, and spread, etc.)
(ISO, 2003; Peeters, et al., 2000).
Feature vectors obtained via parameterization of
musical instrument sounds are used as inputs for classi-
fiers, both for training and recognition purposes.
Classification Techniques
Automatic classification is the process by which a clas-
sificatory system processes information in order to
automatically classify data accurately, or the result of
such a process. A class may represent an instrument,
articulation, instrument family, and so forth. Classifiers
applied to this task range from probabilistic and statisti-
cal algorithms through methods based on learning by
example, where classification is based on the distance
between the observed sample and the nearest known
neighbor, to methods originating from artificial intelli-
gence like neural networks, which mimic neural connec-
tions in the brain. Each classifier yields a new sound
description (representation). Some classifiers produce an
explicit set of classification rules (e.g., decision trees or
rough set based algorithms), giving insight into relation-
ships between specific sound timbres and the calculated
features. Since human-performed recognition of musical
instruments is based on subjective criteria and difficult to
formalize, learning algorithms that allow extraction of pre-
cise rules of sound classification are broadening our knowl-
edge and giving formal representation of subjective sound
features.
The following algorithms can be applied to musical
instrument sound classification:
Bayes decision rule (i.e., probabilistic classifica-
tion method of assignment of unknown samples) to
the classes. In Brown (1999), training data were
grouped into clusters obtained through k-means
algorithm, and Gaussian probability density func-
tions were formed from the mean and variance of
each cluster.
K-Nearest Neighbor (k-NN) algorithm, where the
class (instrument) for a tested sound sample is
assigned on the basis of the distances between the
vector of parameters for this sample and the major-
ity of k nearest vectors representing known samples
(Kaminskyj, 2002; Martin & Kim, 1998). To im-
prove performance, genetic algorithms are addi-
tionally applied to find the optimal set of weights
for the parameters (Fujinaga & McMillan, 2000).
A statistical pattern-recognition technique; maxi-
mum a posteriori classifier based on Gaussian mod- A
els (introducing prior probabilities), obtained via
Fisher multiple discriminant analysis that projects
the high-dimensional feature space into a space of
one dimension fewer than the number of classes in
which the classes are separated maximally (Martin
& Kim, 1998).
Neural networks designed by analogy with a sim-
plified model of the neural connections in the
brain and trained to find relationships in the data;
multi-layer nets and self-organizing feature maps
have been used (Cosi et al., 1994; Kostek &
Czyzewski, 2001).
Decision trees, where nodes are labeled with
sound parameters, edges are labeled with param-
eter values, and leaves represent classes
(Wieczorkowska, 1999b).
Rough-set-based algorithms; rough sets are de-
fined by upper approximation, containing ele-
ments that belong to the set for sure, and lower
approximation containing elements that may be-
long to the set (Wieczorkowska, 1999a).
Support vector machines that aim at finding the
hyperplane that best separates observations be-
longing to different classes (Agostini et al., 2003).
Hidden Markov Models (HMM) used for repre-
senting sequences of states; in this case, can be
used for representing long sequences of feature
vectors that define an instrument sound (Herrera
et al., 2000).
Classifiers are first trained and then tested with
respect to their generalization purposes (i.e., whether
they work properly on unknown samples.
Validation and Results
Parameterization and classification methods yield vari-
ous results, depending on the sound data and validation
procedure when classifiers are tested on unseen
samples. Usually, the available data are divided into
training and test sets. For instance, 70% of the data is
used for training and the remaining 30% for testing;
this procedure is usually repeated a number of times,
and the final result is the average of all runs. Other
popular divisions are in proportions 80/20 or 90/10.
Also, leave-one-out procedure is used, where only one
sample is used for testing. Generally, the higher per-
centage of the training data is in proportion to the test
data and the smaller the number of classes, the higher
the accuracy that is obtained. Some instruments are
easily identified with high accuracy, whereas, others
frequently are misclassified, especially with those from
85
TEAM LinG

the same family. Classification of instruments is sometimes
performed hierarchically: articulation or family is recog-
nized first, and then the instrument is identified.
Following is an overview of results obtained so far in
the research on musical instrument sound classification:
Brown (1999) reported an average 84.1% recog-
nition accuracy for two classesoboe and saxo-
phoneusing cepstral coefficients as features
and Bayes decision rules for clusters obtained via
k-means algorithm.
Brown, Houix, and McAdams (2001), in experi-
ments with four classes, obtained 7984% accu-
racy for bin-to-bin differences of constant-Q co-
efficients, and cepstral and autocorrelation coef-
ficients using Bayesian method.
K-NN classification applied to mel-frequency and
linear prediction cepstral coefficients (Eronen,
2001), with training on 29 orchestral instruments
and testing on 16 instruments from various re-
cordings, yielded 35% accuracy for instruments
and 77% for families. K-NN, combined with ge-
netic algorithms (Fujinaga & McMillan, 2000),
yielded 50% correctness in leave-one-out tests on
spectral features representing 23 orchestral in-
struments played with various articulations.
Kaminskyj (2002) applied k-NN to constant-Q
and cepstral coefficients, MSA trajectories, am-
plitude envelope, and spectral centroid. He ob-
tained 89-92% accuracy for instruments, 96% for
families, and 100% in identifying impulsive vs.
sustained sounds in leave-one-out tests for MUMS
data. Tests on other recordings initially yielded
33-61% accuracy, and 87-90% after improve-
ments.
Multilayer neural networks applied to wavelet and
Fourier-based parameterization yielded 72-99%
accuracy for various groups of four instruments
(Kostek & Czyzewski, 2001).
The statistical pattern-recognition technique and
k-NN algorithm, applied to sounds representing
14 orchestral instruments played with various ar-
ticulation, yielded 71.6% accuracy for instruments,
86.9% for families, and 98.8% in discriminating
continuant sounds vs. pizzicato (Martin & Kim,
1998) in 70/30 tests. The features included pitch,
spectral centroid, ratio of odd-to-even harmonic
energy, onset asynchrony, and the strength of vi-
brato and tremolo (quick changes of sound ampli-
tude or note repetitions).
Discriminant analysis and support vector machines
yielded about 70% accuracy in leave-one-out tests
with spectral features for 27 instruments (Agostini
et al., 2003).
86
Automatic Musical Instrument Sound Classification
Rough-set-based classifiers and decision trees ap-
plied to the data representing 18 classes (11 orches-
tral instruments, various articulation), parameter-
ized using Fourier and wavelet-based attributes,
yielded 68-77 % accuracy in 90/10 test, and 64-68%
in 70/30 tests (Wieczorkowska, 1999b).
K-NN and rough-set-based classifiers, applied to
spectral and temporal sound parameterization,
yielded 68% accuracy in 80/20 tests for 18 classes,
representing 11 orchestral instruments and vari-
ous articulation (Wieczorkowska et al., 2003).
Generally, instrument families or sustained/impul-
sive sounds are identified with accuracy exceeding 90%,
whereas instruments, if there are more than 10, are
identified with accuracy reaching about 70%. These
results compare favorably with human performance and
exceed results obtained for inexperienced listeners.
FUTURE TRENDS
Automatic indexing and searching of audio files is gain-
ing increasing interest. MPEG-7 standard addresses the
issue of content description in multimedia data, and
audio descriptors provided in this standard form a basis
for further research. Constant growth of audio resources
available on the Internet causes an increasing need for
content-based search of audio data. Therefore, we can
expect intensification of research in this domain and
progress of studies on automatic classification of mu-
sical instrument sounds.
CONCLUSION
Results obtained so far in automatic musical instrument
sound classification vary, depending on the size of the data,
sound parameterization, classifier, and testing method.
Also, some instruments are identified easily with high
accuracy, whereas others are misclassified frequently, in
case of both human and machine performance.
Increasing interest in content-based searching
through audiovisual data and growth of amount of mul-
timedia data available via the Internet raises the need and
perspective for further progress in automatic classifi-
cation of audio data.
REFERENCES
Agostini, G., Longari, M., & Pollastri, E. (2003). Musi-
cal instrument timbres classification with spectral fea-
TEAM LinG
Automatic Musical Instrument Sound Classification
tures. EURASIP Journal on Applied Signal Processing,
1, 1-11.
Ando, S., & Yamaguchi, K. (1993). Statistical study of
spectral parameters in musical instrument tones. Journal
of the Acoustical Society of America, 94(1), 37-45.
Brown, J.C. (1999). Computer identification of musical
instruments using pattern recognition with cepstral co-
efficients as features. Journal of the Acoustical Soci-
ety of America, 105, 1933-1941.
Brown, J.C., Houix, O., & McAdams, S. (2001). Feature
dependence in the automatic identification of musical
woodwind instruments. Journal of the Acoustical Soci-
ety of America, 109, 1064-1072.
Cosi, P., De Poli, G., & Lauzzana, G. (1994). Auditory
modelling and self-organizing neural networks for tim-
bre classification. Journal of New Music Research, 23,
71-98.
Eronen, A. (2001). Comparison of features for musical
instrument recognition. Proceedings of the IEEE Work-
shop on Applications of Signal Processing to Audio
and Acoustics WASPAA 2001, New York, NY, USA.
Fritts, L. (1997). The University of Iowa musical instru-
ment samples. Retrieved 2004 from http://there
min.music.uiowa.edu/MIS.html
Fujinaga, I., & McMillan, K. (2000). Realtime recogni-
tion of orchestral instruments. Proceedings of the Inter-
national Computer Music Conference, Berlin, Germany.
Herrera, P., Amatriain, X., Batlle, E., & Serra, X. (2000).
Towards instrument segmentation for music content
description: A critical review of instrument classifica-
tion techniques. Proceedings of the International Sym-
posium on Music Information Retrieval ISMIR 2000,
Plymouth, Massachusetts.
Hornbostel, E.M.V., & Sachs, C. (1914). Systematik der
Musikinstrumente. Ein Versuch. Zeitschrift fr
Ethnologie, 46(4-5), 553-90.
IRCAM, Institute de Recherche et Coordination Acoustique/
Musique. (2003). Studio on line. Retrieved 2004 from http://
forumnet.ircam.fr/rubrique.php3?id_rubr ique=107
ISO, International Organisation for Standardisation. (2003).
MPEG-7 Overview. Retrieved 2004 from http://
www.chiariglio ne.org/mpeg/standards/mpeg-7/mpeg-
7.htm
Kaminskyj, I. (2002). Multi-feature musical instrument
sound classifier w/user determined generalisation per-
formance. Proceedings of the Australasian Computer
Music Association Conference ACMC 2002, Melbourne,
Australia.
Kostek, B., & Czyzewski, A. (2001). Representing musical
instrument sounds for their automatic classification. Jour- A
nal of the Audio Engineering Society, 49(9), 768-785.
Martin, K.D., & Kim, Y.E. (1998). Musical instrument
identification: A pattern-recognition approach. Pro-
ceedings of the 136th Meeting of the Acoustical Society
of America, Norfolk, Virginia.
Opolko, F., & Wapnick, J. (1987). MUMSMcGill Univer-
sity master samples. [CD-ROM]. McGill University,
Montreal, Quebec, Canada.
Peeters, G., McAdams, S., & Herrera, P. (2000). Instrument
sound description in the context of MPEG-7. Proceedings
of the International Computer Music Conference
ICMC2000, Berlin, Germany.
Pollard, H.F., & Jansson, E.V. (1982). A tristimulus method
for the specification of musical timbre. Acustica, 51, 162-
171.
SIL. (1999). LinguaLinks library. Retrieved 2004 from
http://www.sil.org/LinguaLinks/Anthropology/Expndd
EthnmsclgyCtgrCltrlMtrls/MusicalInstrumentsSub
categorie.htm
Smith, R. (2000). Rods encyclopedic dictionary of tradi-
tional music. Retrieved 2004 from http://
www.sussexfolk.freeserve.co.uk/ency/a.htm
Viste, H., & Evangelista, G. (2003). Separation of har-
monic instruments with overlapping partials in multi-
channel mixtures. Proceedings of the IEEE Workshop
on Applications of Signal Processing to Audio and
Acoustics WASPAA-03, New Paltz, New York.
Whitaker, J.C. & Benson, K.B. (Eds.). (2002). Stan-
dard handbook of audio and radio engineering. New
York: McGraw-Hill.
Wieczorkowska, A. (1999a). Rough sets as a tool for
audio signal classification. Foundations of Intelligent
Systems LNCS/LNAI 1609, 11th Symposium on Method-
ologies for Intelligent Systems, Proceedings/ISMIS99,
Warsaw, Poland.
Wieczorkowska, A. (1999b). Skuteczno rozpoznawania
dwikw instrumentw muzycznych w zalenoci od
sposobu parametryzacji i rodzaju klasyfikatora (Effi-
ciency of musical instrument sounds recognition de-
pending on parameterization and classifier) [doctoral
thesis] [in Polish]. Gdansk: Technical University of
Gdansk.
Wieczorkowska, A., Wrblewski, J., Synak, P., & lzak ,
D. (2003). Application of temporal descriptors to musical
instrument sound recognition. Journal of Intelligent
Information Systems, 21(1), 71-93.
87
TEAM LinG

KEY TERMS
Aerophones: The category of musical instruments,
called wind instruments, producing sound by the vibra-
tion of air. Woodwinds and brass (lip-vibrated) instru-
ments belong to this category, including single reed
woodwinds (e.g., clarinet), double reeds (oboe), flutes,
and brass (trumpet).
Articulation: The process by which sounds are formed;
the manner in which notes are struck, sustained, and
released. Examples: staccato (shortening and detaching
of notes), legato (smooth), pizzicato (plucking strings),
vibrato (varying the pitch of a note up and down), muted
(stringed instruments, by sliding a block of rubber or
similar material onto the bridge; brass (by inserting a
conical device into the bell).
Automatic Classification: The process by which a
classificatory system processes information in order to
classify data accurately; also, the result of such process.
Chordophones: The category of musical instru-
ments producing sound by means of a vibrating string;
stringed instruments. Examples: guitar, violin, piano, harp,
lyre, musical bow.
88
Automatic Musical Instrument Sound Classification
Idiophones: The category of musical instruments made
of solid, non-stretchable sonorous material. Subcatego-
ries: idiophones struck together by concussion (e.g., cas-
tanets), struck (gong), rubbed (musical glasses), scraped
(washboards), stamped (hard floors stamped with tap
shoes), shaken (rattles), and plucked (jews harp).
Membranophones: The category of musical instru-
ments; skin drums. Examples: daraboukka, tambourine.
Parameterization: The assignment of parameters
to represent processes that usually are not easily de-
scribed by equations.
Sound: A physical disturbance in the medium through
which it is propagated. This fluctuation may change
routinely, and such periodic sound is perceived as hav-
ing pitch. The audible frequency range is about 20
20,000 Hz (hertz, or cycles per second). Harmonic
sound wave consists of frequencies being integer mul-
tiples of the first component (fundamental frequency),
corresponding to pitch.
Spectrum: The distribution of component frequen-
cies of the sound, each being a sine wave, with their
amplitudes and phases (time locations of these compo-
nents). These frequencies can be determined through
Fourier analysis.
TEAM LinG

Bayesian Networks
Ahmad Bashir
University of Texas at Dallas, USA
Latifur Khan
University of Texas at Dallas, USA
Mamoun Awad
University of Texas at Dallas, USA
INTRODUCTION
A Bayesian network is a graphical model that finds
probabilistic relationships among variables of a system.
The basic components of a Bayesian network include a set
of nodes, each representing a unique variable in the
system, their inter-relations, as indicated graphically by
edges, and associated probability values. By using these
probabilities, termed conditional probabilities, and their
interrelations, we can reason and calculate unknown
probabilities. Furthermore, Bayesian networks have dis-
tinct advantages compared to other methods, such as
neural networks, decision trees, and rule bases, which we
shall discuss in this paper.
BACKGROUND
Bayesian classification is based on Nave Bayesian clas-
sifiers, which we discuss in this section. Naive Bayesian
classification is the popular name for a probabilistic
classification. The term Naive Bayes refers to the fact that
the probability model can be derived by using Bayes
theorem and that it incorporates strong independence
assumptions that often have no bearing in reality, hence
they are deliberately nave. Depending on the model,
Nave Bayes classifiers can be trained very efficiently in
a supervised learning setting. In many practical applica-
tions, parameter estimation for Nave Bayes models uses
the method of maximum likelihood.
Abstractly, the desired probability model for a classi-
fier is a conditional model
P(C | F1,,Fn)
over a dependent class variable C with a small number of
outcomes, or classes, conditional on several feature vari-
ables F1 through Fn. The nave conditional independence
assumptions play a role at this stage, when the probabili-
ties are being computed. In this model, the assumption
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
89
B
is that each feature Fi is conditionally independent of
every other feature Fj. This situation is mathematically
represented as:
P(Fi | C, Fj) = P(Fi | C)
Such nave models are easier to compute, because
they factor into P(C) and a series of independent probabil-
ity distributions. The Nave Bayes classifier combines
this model with a decision rule. The common rule is to
choose the label that is most probable, known as the
maximum a posteriori or MAP decision rule.
In a supervised learning setting, one wants to estimate
the parameters of the probability model. Because of the
independent feature assumption, it suffices to estimate
the class prior and the conditional feature models inde-
pendently by using the method of maximum likelihood.
The Nave Bayes classifier has several properties that
make it simple and practical, although the independence
assumptions are often violated. The overall classifier is
the robust to serious deficiencies of its underlying nave
probability model, and in general, the Nave Bayes ap-
proach is more powerful than might be expected from the
extreme simplicity of its model; however, in the presence
of nonindependent attributes wi, the Nave Bayesian
classifier must be upgraded to the Bayesian classifier,
which will more appropriately model the situation.
MAIN THRUST
The basic concept in the Bayesian treatment of certainties
in causal networks is conditional probability. When the
probability of an event A, P(A), is known, then it is
conditioned by other known factors. A conditional prob-
ability statement has the following form:
Given that event B has occurred, the probability of
the event A occurring is x.
TEAM LinG

Graphical Models
A graphical model visually illustrates conditional inde-
pendencies among variables in a given problem. Two
variables that are conditionally independent have no
direct impact on each others values. Furthermore, the
graphical model shows any intermediary variables that
separate two conditionally independent variables.
Through these intermediary variables, two conditionally
independent variables affect one another.
A graph is composed of a set of nodes, which repre-
sent variables, and a set of edges. Each edge connects two
nodes, and an edge can have an optional direction as-
signed to it. For X1 and X2, if a causal relationship between
the variables exists, the edge will be directional, leading
from the case variable to the effect variable; if just a
correlation between the variables exists, the edge will be
undirected.
We use an example with three variables to illustrate
these concepts. In this example, two conditionally inde-
pendent variables, A and C, are directly related to another
variable, B. To represent this situation, an edge must exist
between the nodes of the variables that are directly
related, that is, between A and B and between B and C.
Furthermore, the relationships between A and B and B and
C are correlations as opposed to causal relations; hence,
the respective edges will be undirected. Figure 1 illus-
trates this example. Due to conditional independence,
nodes A and C still have an indirect influence on one
another; however, variable B encodes the information
from A that impacts C, and vice versa.
A Bayesian network is a specific type of graphical
model, with directed edges and no cycles (Stephenson,
2000). The edges in Bayesian networks are viewed as
causal connections, where each parent node causes an
effect on its children.
In addition, nodes in a Bayesian network contain a
conditional probability table, or CPT, which stores all
probabilities that may be used to reason or make infer-
ences within the system.
Figure 1. Graphical model of two independent variables
A and C that are directly related to a third variable B
B manipulation of X leads to a change in the probability
A C
90
Bayesian Networks
Bayesian Probabilities
Probability calculus does not require that the probabili-
ties be based on theoretical results or frequencies of
repeated experiments, commonly known as relative fre-
quencies. Probabilities may also be completely subjective
estimates of the certainty of an event.
Consider an example of a basketball game. If one were
to bet on an upcoming game between Team A and Team
B, it is important to know the probability of Team A
winning the game. This probability is definitely not a ratio,
a relative frequency, or even an estimate of a relative
frequency; the game cannot be repeated many times
under exactly the same conditions. Rather, the probability
represents only ones belief concerning Team As chances
of winning. Such a probability is termed a Bayesian or
subjective probability and makes use of Bayes theorem
to calculate unknown probabilities.
A Bayesian probability may also be referred to as a
personal probability. The Bayesian probability of an
event x is a persons degree of belief in that event. A
Bayesian probability is a property of the person who
assigns the probability, whereas a classical probability
is a physical property of the world, meaning it is the
physical probability of an event.
An important difference between physical probability
and Bayesian probability is that repeated trials are not
necessary to measure the Bayesian probability. The Baye-
sian method can assign a probability for events that may
be difficult to experimentally determine. An oft-voiced
criticism of the Bayesian approach is that probabilities
seem arbitrary, but this is a probability assessment issue
that does not take away from the many possibilities that
Bayesian probabilities provide.
Causal Influence
Bayesian networks require an operational method for
identifying causal relationships in order for accurate
domain modeling. Hence, causal influence is defined in
the following manner: If the action of making variable X
take some value sometimes changes the value taken by
variable Y, then X is assumed to be responsible for some-
times changing Ys value, and one may conclude that X is
a cause of Y. More formally, X is manipulated when we
force X to take some value, and we say X causes Y if some
distribution of Y.
Furthermore, if manipulating X leads to a change in the
probability distribution of Y, then X obtaining a value by
any means whatsoever also leads to a change in the
probability distribution of Y. Hence, one can make the
natural conclusion that causes and their effects are statis-
TEAM LinG
Bayesian Networks
tically correlated. However, note that variables can be
correlated in less direct ways; that is, one variable may not
necessarily cause the other. Rather, some intermediaries
may be involved.
Bayesian Networks: A First Look
This section provides a detailed definition of Bayesian
networks in an attempt to merge the probabilistic, logic,
and graphical modeling ideas that we have presented thus
far.
Causal relations, apart from the reasoning that they
model, also have a quantitative side, namely their strength.
This is expressed by associating numbers to the links. Let
the variable A be a parent of the variable B in a causal
network. Using probability calculus, it will be normal to let
the conditional probability be the strength of the link
between these variables. On the other hand, if the variable
C is also a parent of the variable B, then conditional
probabilities P(B|A) and P(B|C) do not provide any infor-
mation on how the impacts from variable A and variable B
interact. They may cooperate or counteract in various
ways. Therefore, the specification of P(B|A,C) is required.
If loops exist within the reasoning, the domain model
may contain feedback cycles; these cycles are difficult to
model quantitatively. For such networks, no calculus cop-
ing with feedback cycles has been developed. Therefore,
the network must be loop free. In fact, from a practical
perspective, it should be modeled as closely to a tree as
possible.
Evidential Reasoning
As stated previously, Bayesian networks accomplish such
an economy by pointing out, for each variable Xi, the
conditional probabilities P(Xi | pai) where pai is the set of
parents (of Xi) that render Xi independent of all its other
parents. After giving this specification, the joint probabil-
ity distribution can be calculated by the product
P( x1,..., xn ) = P( xi | pai )
i
Using this product, all probabilistic queries can be
found coherently with probability calculus. Passing evi-
dence up and down a Bayesian network is known as belief
propagation, and exact probability calculations is NP-
hard, meaning that the calculation time to compute exact
probabilities grows exponentially with the size of the
Bayesian network. A number of algorithms are available
for probabilistic calculations in Bayesian networks
(Huang, King, & Lyu, 2002), beginning with Pearls
message passing algorithm.
Advantages
B
A Bayesian network is a graphical model that finds
probabilistic relationships among variables of the sys-
tem, but a number of models are available for data analy-
sis, including rule bases, decision trees, and artificial
neural networks. Several techniques for data analysis
also exist, including classification, density estimation,
regression, and clustering. However, Bayesian networks
have distinct advantages that we discuss here.
One of the biggest advantages of Bayesian networks
is that they have a bidirectional message passing archi-
tecture. Learning from the evidence can be interpreted as
unsupervised learning. Similarly, expectation of an ac-
tion can be interpreted as unsupervised learning. Be-
cause Bayesian networks pass data between nodes and
see the expectations from the world model, they can be
considered as bidirectional learning systems (Helsper &
Gaag, 2002). In addition to bidirectional message pass-
ing, Bayesian networks have several important features,
such as the allowance of subjective a priori judgments,
direct representation of causal dependence, and the
ability to imitate the human thinking process.
Bayesian networks handle incomplete data sets with-
out difficulty because they discover dependencies
among all the variables. When one of the inputs is not
observed, most other models will end up with an inaccu-
rate prediction because they do not calculate the corre-
lation between the input variables. Bayesian networks
suggest a natural way to encode these dependencies.
For this reason, they are a natural choice for image
classification or annotation, because the lack of a clas-
sification can also be viewed as missing data.
Considering the Bayesian statistical techniques,
Bayesian networks facilitate the combination of domain
knowledge and data (Neapolitan, 2004). Prior or domain
knowledge is crucially important if one performs a real-
world analysis, particularly when data are inadequate or
expensive. The encoding of causal prior knowledge is
straightforward because Bayesian networks have causal
semantics. Additionally, Bayesian networks encode the
strength of causal relationships with probabilities. There-
fore, prior knowledge and data can be put together with
well studied techniques from Bayesian statistics for both
forward and backward reasoning with the Bayesian net-
work.
Bayesian networks also ease many of the theoretical
and computational difficulties of rule-based systems by
utilizing graphical structures for representing and man-
aging probabilistic knowledge. Independencies can be
dealt with explicitly; they can be articulated, encoded
graphically, read off the network, and reasoned about,
and yet they forever remain robust to numerical impres-
91
TEAM LinG

sion. Moreover, graphical representations uncover sev-
eral opportunities for efficient computation and serve as
understandable logic diagrams.
Bayesian networks can simulate humanlike reason-
ing; this fact is not, however, due to any structural
similarities with the human brain. Rather, it is because of
the resemblance between the ways Bayesian networks
and humans reason. The resemblance is more psychologi-
cal than biological but nevertheless a true benefit.
Bayesian Inference
Inference is the task of computing the probability of each
value of a node in a Bayesian network when other vari-
ables values are known (Jensen, 1999). This concept is
what makes Bayesian networks so powerful, as it allows
the user to apply knowledge toward forward or backward
reasoning. Suppose that a specific value for one or more
of the variables in the network has been observed. If one
variable has a definite value, or evidence, the probabili-
ties, or belief values, for the other variables need to be
revised, as this variable is not a defined value. This
calculation of the updated probabilities for system vari-
ables that are based on new evidence is precisely the
definition of inference.
FUTURE TRENDS
The future of Bayesian networks lies in determining new
ways to tackle the following issues of Bayesian inferencing
and in building a Bayesian structure that accurately
represents a particular system. As we discuss in this
paper, conditional dependencies can be mapped into a
graph in several ways, each with subtle semantic and
statistical differences. Future research will give way to
Bayesian networks that can understand system seman-
tics and adapt accordingly, not only with respect to the
conditional probabilities within each node but also with
respect to the graph itself.
CONCLUSION
A Bayesian network consists of the following elements:
A set of variables and a set of directed edges
between variables
A finite set of mutually exclusive states for each
variable
A directed acyclic graph (DAG), constructed from
the variables coupled with the directed edges
A conditional probability table (CPT) P(A | B1, B2, ,
92
Bayesian Networks
Bn) that is associated with each variable A with
parents B1, B2 Bn
Bayesian networks continue to play a vital role in
prediction and classification within data mining
(Niedermeyer, 1998). They are a marriage between prob-
ability theory and graph theory, providing a natural tool
for dealing with two problems that occur throughout
applied mathematics and engineering: uncertainty and
complexity. Also, Bayesian networks play an increasingly
important role in the design and analysis of machine
learning algorithms, serving as a promising way to ap-
proach present and future problems related to artificial
intelligence and data mining (Choudhary, Rehg, Pavlovic,
& Pentland, 2002; Doshi, Greenwald, & Clarke, 2002;
Fenton, Cates, Forey, Marsh, Neil, & Tailor, 2003).
REFERENCES
Choudhury, T., Rehg, J. M., Pavlovic, V., & Pentland, A.
(2002). Boosting and structure learning in dynamic Baye-
sian networks for audio-visual speaker detection. Pro-
ceedings of the International Conference on Pattern
Recognition (ICPR), Canada, III (pp. 789-794).
Doshi, P., Greenwald, L., & Clarke, J. (2002). Towards
effective structure learning for large Bayesian networks.
Proceedings of the AAAI Workshop on Probabilistic
Approaches in Search, Canada (pp. 16-22).
Fenton, N., Cates, P., Forey, S., Marsh, W., Neil, M., &
Tailor, M. (2003). Modelling risk in complex software
projects using Bayesian networks (Tech. Rep. No. RA-
DAR Tech Repo). London: Queen Mary University.
Helsper, E.M. & Gaag, L.C. van der. (2002). Building
Bayesian networks through ontologies. Proceedings of
the 15th Eureopean Conference on Artificial Intelli-
gence, Lyon, France (pp. 680-684).
Huang, K., King, I., & R. Lyu, M. (2002). Learning maximum
likelihood semi-naive Bayesian network classifier. Pro-
ceedings of the IEEE International Conference on Sys-
tems, Man, and Cybernetics, 3, 6, Hamammet, Tunisia.
Jensen, F. (1999). Gradient descent training of Bayesian
networks. Proceedings of the European Conference on
Symbolic and Quantitative Approaches to Reasoning
and Uncertainty (pp. 190-200).
Neapolitan, R. E. (2004). Learning Bayesian networks.
Upper Saddle River, NJ: Prentice-Hall.
Niedermayer, D. (1998). An introduction to Bayesian
networks and their contemporary applications. Re-
TEAM LinG
Bayesian Networks
trieved October 2004, from http://www.niedermayer.ca/
papers/bayesian/
Stephenson, T. (2000). An introduction to Bayesian net-
work theory and usage. Retrieved October 2004, from
http://www.idiap.ch/publications/todd00a.bib.abs.html
KEY TERMS
Bayes Theorem: Result in probability theory that
states the conditional probability of a variable A, given B,
in terms of the conditional probability of variable B, given
A, and the marginal probability of A alone.
Conditional Probability: Probability of some event A,
assuming event B, written mathematically as P(A|B).
Data Mining: The application of analytical methods
and tools to data for the purpose of identifying patterns
and relationships such as classification, prediction, esti-
mation, or affinity grouping.
Independent: Two random variables are independent
when knowing something about the value of one of them B
does not yield any information about the value of the
other.
Joint Probability: The probability of two events oc-
curring in conjunction.
Maximum Likelihood: Method of point estimation
using as an estimate of an unobservable population
parameter the member of the parameter space that maxi-
mizes the likelihood function.
Neural Networks: Learning systems that are designed
by analogy with a simplified model of the neural connec-
tions in the brain and can be trained to find nonlinear
relationships in data.
Supervised Learning: A machine learning technique
for creating a function from training data; the task of the
supervised learner is to predict the value of the function
for any valid input object after having seen only a small
number of training data.
93
TEAM LinG
94
Best Practices in Data Warehousing from the
Federal Perspective
Les Pang
National Defense University, USA
INTRODUCTION
Data warehousing has been a successful approach for
supporting the important concept of knowledge man-
agement one of the keys to organizational success at
the enterprise level. Based on successful implementa-
tions of warehousing projects, a number of lessons
learned and best practices were derived from these
project experiences. The scope was limited to projects
funded and implemented by federal agencies, military
institutions and organizations directly supporting them.
Projects and organizations reviewed include the fol-
lowing:
Census 2000 Cost and Progress System
Defense Dental Standard System
Defense Medical Logistics Support System Data
Warehouse Program
Department of Agriculture Rural Development
Data Warehouse
DOD Computerized Executive Information Sys-
tem
Department of Transportation (DOT) Executive
Reporting Framework System
Environmental Protection Agency (EPA)
Envirofacts Warehouse
Federal Credit Union
Health and Urban Development (HUD) Enterprise
Data Warehouse
Internal Revenue Service (IRS) Compliance Data
Warehouse
U.S. Army Operational Testing and Evaluation Com-
mand
U.S. Coast Guard Executive Information System
U.S. Navy Type Commanders Readiness Manage-
ment System
BACKGROUND
Data warehousing involves the consolidation of data
from various transactional data sources in order to
support the strategic needs of an organization. This
approach links the various silos of data that is distrib-
uted throughout an organization. By applying this ap-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
proach, an organization can gain significant competitive
advantages through the new level of corporate knowledge.
Various agencies in the Federal Government at-
tempted to implement a data warehousing strategy in
order to achieve data interoperability. Many of these
agencies have achieved significant success in improving
internal decision processes as well as enhancing the
delivery of products and services to the citizen. This
chapter aims to identify the best practices that were
implemented as part of the successful data warehousing
projects within the federal sector.
MAIN THRUST
Each best practice (indicated in boldface) and its ratio-
nale are listed below. Following each practice is a
description of illustrative project or projects (indicated
in italics), which support the practice.
Ensure the Accuracy of the Source
Data to Maintain the Users Trust of
the Information in a Warehouse
The user of a data warehouse needs to be confident that
the data in a data warehouse is timely, precise, and
complete. Otherwise, a user that discovers suspect data
in warehouse will likely cease using it, thereby reducing
the return on investment involved in building the ware-
house. Within government circles, the appearance of
suspect data takes on a new perspective.
HUD Enterprise Data Warehouse - Gloria Parker,
HUD Chief Information Officer, spearheaded data ware-
housing projects at the Department of Education and at
HUD. The HUD warehouse effort was used to profile
performance, detect fraud, profile customers, and do
what if analysis. Business areas served include Fed-
eral Housing Administration loans, subsidized proper-
ties, and grants. She emphasizes that the public trust of
the information is critical. Government agencies do not
want to jeopardize our public trust by putting out bad
data. Bad data will result in major ramifications not only
from citizens but also from the government auditing
arm, the General Accounting Office, and from Congress
(Parker, 1999).
TEAM LinG
Best Practices in Data Warehousing from the Federal Perspective
EPA Envirofacts Warehouse - The Envirofacts data
warehouse comprises of information from 12 different
environmental databases for facility information, in-
cluding toxic chemical releases, water discharge permit
compliance, hazardous waste handling processes,
Superfund status, and air emission estimates. Each pro-
gram office provides its own data and is responsible for
maintaining this data. Initially, the Envirofacts ware-
house architects noted some data integrity problems,
namely, issues with accurate data, understandable data,
properly linked data and standardized data. The archi-
tects had to work hard to address these key data issues so
that the public can trust that the quality of data in the
warehouse (Garvey, 2003).
U.S. Navy Type Commander Readiness Manage-
ment System The Navy uses a data warehouse to
support the decisions of its commanding officers. Data
at the lower unit levels is aggregated to the higher levels
and then interfaced with other military systems for a
joint military assessment of readiness as required by the
Joint Chiefs of Staff. The Navy found that it was spend-
ing too much time to determine its readiness and some
of its reports contained incorrect data. The Navy devel-
oped a user friendly, Web-based system that provides
quick and accurate assessment of readiness data at all
levels within the Navy. The system collects, stores,
reports and analyzes mission readiness data from air, sub
and surface forces for the Atlantic and Pacific Fleets.
Although this effort was successful, the Navy learned that
data originating from the lower levels still needs to be
accurate. The reason is that a number of legacy systems,
which serves as the source data for the warehouse, lacked
validation functions (Microsoft, 2000).
Standardize the Organizations Data
Definitions
A key attribute of a data warehouse is that it serves as a
single version of the truth. This is a significant im-
provement over the different and often conflicting ver-
sions of the truth that come from an environment of
disparate silos of data. To achieve this singular version
of the truth, there needs to be consistent definitions of
data elements to afford the consolidation of common
information across different data sources. These con-
sistent data definitions are captured in a data warehouses
metadata repository.
DoD Computerized Executive Information System
(CEIS) is a 4-terabyte data warehouse holds the medical
records of the 8.5 million active members of the U.S.
military health care system who are treated at 115
hospitals and 461 clinics around the world. The Defense
Department wanted to convert its fixed-cost health care
system to a managed-care model to lower costs and
increase patient care for the active military, retirees and
their dependents. Over 12,000 doctors, nurses and ad-
ministrators use it. Frank Gillett, an analyst at Forrester B
Research, Inc., stated that, What kills these huge data
warehouse projects is that the human beings dont agree
on the definition of data. Without that . . . all that $450
million [cost of the warehouse project] could be thrown
out the window (Hamblen, 1998).
Be Selective on what Data Elements to
Include in the Warehouse
Users are unsure of what they want so they place an
excessive number of data elements in the warehouse.
This results in an immense, unwieldy warehouse in
which query performance is impaired.
Federal Credit Union - The data warehouse archi-
tect for this organization suggests that users know which
data they use most, although they will not always admit
to what they use least (Deitch, 2000).
Select the Extraction-Transformation-
Loading (ETL) Strategy Carefully
Having an effective ETL strategy that extracts data from
the various transactional systems, transforms the data to
a common format, and loads the data into a relational or
multidimensional database is the key to a successful
data warehouse project. If the ETL strategy is not effec-
tive, it will mean delays in refreshing the data ware-
house, contaminating the data warehouse with dirty data,
and increasing the costs in maintaining the warehouse.
IRS Compliance Warehouse supports research and
decision support, allows the IRS to analyze, develop,
and implement business strategies for increasing volun-
tary compliance, improving productivity and managing
the organization. It also provides projections, forecasts,
quantitative analysis, and modeling. Users are able to
query this data for decision support.
A major hurdle was to transform the large and di-
verse legacy online transactional data sets for effective
use in an analytical architecture. They needed a way to
process custom hierarchical data files and convert to
ASCII for local processing and mapping to relational
databases. They ended up with developing a script pro-
gram that will do all of this. ETL is a major challenge and
may be a showstopper for a warehouse implementa-
tion (Kmonk, 1999).
Leverage the Data Warehouse to
Provide Auditing Capability
An overlooked benefit of data warehouses is its capabil-
ity of serving as an archive of historic knowledge that
can be used as an audit trail for later investigations.
95
TEAM LinG
 Best Practices in Data Warehousing from the Federal Perspective
U.S. Army Operational Testing and Evaluation Com-
mand (OPTEC) is charged with developing test criteria
and evaluating the performance of extremely complex weap-
ons equipment in every conceivable environment and
condition. Moreover, as national defense policy is under-
going a transformation, so do the weapon systems, and
thus the testing requirements. The objective of their ware-
house was to consolidate a myriad of test data sets to
provide analysts and auditors with access to the specific
information needed to make proper decisions.
OPTEC was having fits when audit agencies, such as
the General Accounting Office (GAO), would show up to
investigate a weapon system. For instance, if problems
with a weapon show up five years after it is introduced
into the field, people are going to want to know what tests
were performed and the results of those tests. A ware-
house with its metadata capability made data retrieval
much more efficient (Microsoft, 2000).
Leverage the Web and Web Portals for
Warehouse Data to Reach Dispersed
Users
In many organizations, users are geographically distrib-
uted and the World Wide Web has been very effective as
a gateway for these dispersed users to access the key
resources of their organization, which include data ware-
houses and data marts.
U.S. Army OPTEC developed a Web-based front end
for its warehouse so that information can be entered and
accessed regardless of the hardware available to users. It
supports the geographically dispersed nature of OPTECs
mission. Users performing tests in the field can be
anywhere from Albany, New York to Fort Hood, Texas.
That is why the browser client the Army developed is so
important to the success of the warehouse (Microsoft,
2000).
DoD Defense Dental Standard System supports more
than 10,000 users at 600 military installations world-
wide. The solution consists of three main modules:
Dental Charting, Dental Laboratory Management, and
Workload and Dental Readiness Reporting. The charting
module helps dentists graphically record patient infor-
mation. The lab module automates the workflow between
dentists and lab technicians. The reporting module al-
lows users to see key information though Web-based
online reports, which is a key to the success of the
defense dental operations.
IRS Compliance Data Warehouse includes a Web-
based query and reporting solution that provides high-value,
easy-to-use data access and analysis capabilities, be quickly
and easily installed and managed, and scale to support
hundreds of thousands of users. With this portal, the IRS
96
found that portals provide an effective way to access
diverse data sources via a single screen (Kmonk, 1999).
Make Warehouse Data Available to All
Knowledgeworkers (Not Only to
Managers)
The early data warehouses were designed to support
upper management decision-making. However, over
time, organizations have realized the importance of
knowledge sharing and collaboration and its relevance
to the success of the organizational mission. As a
result, upper management has become aware of the
need to disseminate the functionality of the data ware-
house throughout the organization.
IRS Compliance Data Warehouse supports a diver-
sity of user types economists, research analysts, and
statisticians all of whom are searching for ways to
improve customer service, increase compliance with
federal tax laws and increase productivity. It is not just
for upper management decision making anymore
(Kmonk, 1999).
Supply Data in a Format Readable by
Spreadsheets
Although online analytical tools such as those sup-
ported by Cognos and Business Objects are useful for
data analysis, the spreadsheet is still the basic tool used
by most analysts.
U.S. Army OPTEC wanted users to transfer data and
work with information on applications that they are
familiar with. In OPTEC, they transfer the data into a
format readable by spreadsheets so that analysts can
really crunch the data. Specifically, pivot tables found in
spreadsheets allows the analysts to manipulate the infor-
mation to put meaning behind the data (Microsoft, 2000).
Restrict or Encrypt Classified/
Sensitive Data
Depending on requirements, a data warehouse can con-
tain confidential information that should not be re-
vealed to unauthorized users. If privacy is breached, the
organization may become legally liable for damages
and suffer a negative reputation with the ensuing loss of
customers trust and confidence. Financial conse-
quences can result.
DoD Computerized Executive Information System
uses an online analytical processing tool from a popu-
lar vendor that could be used to restrict access to
certain data, such as HIV test results, so that any confi-
dential data would not be disclosed (Hamblen, 1998).
TEAM LinG
Best Practices in Data Warehousing from the Federal Perspective
Considerations must be made in the architecting of a data
warehouse. One alternative is to use a roles-based archi-
tecture that allows access to sensitive data by only
authorized users and the encryption of data in the event
of data interception.
Perform Analysis During the Data
Collection Process
Most data analyses involve completing data collection
before analysis can begin. With data warehouses, a new
approach can be undertaken.
Census 2000 Cost and Progress System was built to
consolidate information from several computer sys-
tems. The data warehouse allowed users to perform
analyses during the data collection process; something
was previously not possible. The system allowed execu-
tives to take a more proactive management role. With
this system, Census directors, regional offices, manag-
ers, and congressional oversight committees have the
ability to track the 2000 census, which never been done
before (SAS, 2000).
Leverage User Familiarity with
Browsers to Reduce Training
Requirements
The interface of a Web browser is very familiar to most
employees. Navigating through a learning management
system using a browser may be more user friendly than
using the navigation system of a proprietary training
software.
U.S. Department of Agriculture (USDA) Rural De-
velopment, Office of Community Development, admin-
isters funding programs for the Rural Empowerment
Zone Initiative. There was a need to tap into legacy
databases to provide accurate and timely rural funding
information to top policy makers. Through Web acces-
sibility using an intranet system, there were dramatic
improvements in financial reporting accuracy and timely
access to data. Prior to the intranet, questions such as
What were the Rural Development investments in 1997
for the Mississippi Delta region? required weeks of
laborious data gathering and analysis, yet yielded obso-
lete answers with only an 80 percent accuracy factor.
Now, similar analysis takes only a few minutes to per-
form, and the accuracy of the data is as high as 98
percent. More than 7,000 Rural Development employ-
ees nationwide can retrieve the information at their
desktops, using a standard Web browser. Because em-
ployees are familiar with the browser, they did not need
training to use the new data mining system (Ferris,
2003).
Use Information Visualization
Techniques Such as Geographic B
Information Systems (GIS)
A GIS combines layers of data about a physical location
to give users a better understanding of that location. GIS
allows users to view, understand, question, interpret,
and visualize data in ways simply not possible in para-
graphs of text or in the rows and columns of a spread-
sheet or table.
EPA Envirofacts Warehouse includes the capability
of displaying its output via the EnviroMapper GIS sys-
tem. It maps several types of environmental informa-
tion, including drinking water, toxic and air releases,
hazardous waste, water discharge permits, and Superfund
sites at the national, state, and county levels (Garvey,
2003). Individuals familiar with Mapquest and other
online mapping tools can easily navigate the system and
quickly get the information they need.
FUTURE TRENDS
Data warehousing will continue to grow as long as there
are disparate silos of data sources throughout an organi-
zation. However, the irony is that there will be a prolif-
eration of data warehouses as well as data marts, which
will not interoperate within an organization. Some ex-
perts predict the evolution toward a federated architec-
ture for the data warehousing environment. For ex-
ample, there will be a common staging area for data
integration and, from this source, data will flow among
several data warehouses. This will ensure that the single
truth requirement is maintained throughout the organi-
zation (Hackney, 2000).
Another important trend in warehousing is one away
from historic nature of data in warehouses and toward
real-time distribution of data so that information vis-
ibility will be instantaneous (Carter, 2004). This is a key
factor for business decision-making in a constantly
changing environment. Emerging technologies, namely
service-oriented architectures and Web services, are
expected to be the catalyst for this to occur.
CONCLUSION
An organization needs to understand how it can leverage
data from a warehouse or mart to improve its level of
service and the quality of its products and services.
Also, the organization needs to recognize that its most
valuable resource, the workforce, needs to be adequately
trained in accessing and utilizing a data warehouse. The
97
TEAM LinG
 Best Practices in Data Warehousing from the Federal Perspective
workforce should recognize the value of the knowledge
that can be gained from data warehousing and how to
apply it to achieve organizational success.
A data warehouse should be part of an enterprise
architecture, which is a framework for visualizing the
information technology assets of an enterprise and how
these assets interrelate. It should reflect the vision and
business processes of an organization. It should also
include standards for the assets and interoperability
requirements among these assets.
REFERENCES
AMS. (1999). Military marches toward next-genera-
tion health care service: The Defense Dental Stan-
dard System.
Carter, M. (2004). The death of data warehousing. Loosely
Coupled.
Deitch, J. (2000). Technicians are from Mars, users are
from Venus: Myths and facts about data warehouse
administration (Presentation).
Ferris, N. (1999). 9 hot trends for 99. Government Execu-
tive.
Ferris, N. (2003). Information is power. Government Ex-
ecutive.
Garvey, P. (2003). Envirofacts warehouse public ac-
cess to environmental data over the Web (Presenta-
tion).
Gerber, C. (1996). Feds turn to OLAP as reporting tool.
Federal Computer Week.
Hackney, D. (2000). Data warehouse delivery: The fed-
erated future. DM Review.
Hamblen, M. (1998). Pentagon to deploy huge medical
data warehouse. Computer World.
Kirwin, B. (2003). Management update: Total cost of
ownership analysis provides many benefits. Gartner
Research, IGG-08272003-01.
Kmonk, J. (1999). Viador information portal provides Web
data access and reporting for the IRS. DM Review.
Matthews, W. (2000). Digging digital gold. Federal Com-
puter Week.
Microsoft Corporation. (2000). OPTEC adopts data ware-
housing strategy to test critical weapons systems.
Microsoft Corporation. (2000). U.S. Navy ensures readi-
ness using SQL Server.
98
Parker, G. (1999). Data warehousing at the federal govern-
ment: A CIO perspective. In Proceedings from Data
Warehouse Conference 99.
PriceWaterhouseCoopers. (2001). Technology forecast.
SAS. (2000). The U.S. Bureau of the Census counts on a
better system.
Schwartz, A. (2000). Making the Web Safe. Federal Com-
puter Week.
KEY TERMS
ASCII: American Standard Code for Information
Interchange. Serves a code for representing English
characters as numbers with each letter assigned a num-
ber from 0 to 127.
Data Warehousing: A compilation of data designed
to for decision support by executives, managers, ana-
lysts and other key stakeholders in an organization. A
data warehouse contains a consistent picture of busi-
ness conditions at a single point in time.
Database: A collection of facts, figures, and objects
that is structured so that it can easily be accessed,
organized, managed, and updated.
Enterprise Architecture: A business and perfor-
mance-based framework to support cross-agency col-
laboration, transformation, and organization-wide im-
provement.
Extraction-Transformation-Loading (ETL): A
key transitional set of steps in migrating data from the
source systems to the database housing the data ware-
house. Extraction refers to drawing out the data from
the source system, transformation concerns converting
the data to the format of the warehouse and loading
involves storing the data into the warehouse.
Geographic Information Systems: Map-based
tools used to gather, transform, manipulate, analyze, and
produce information related to the surface of the Earth.
Hierarchical Data Files: Database systems that
are organized in the shape of a pyramid with each row of
objects linked to objects directly beneath it. This
approach has generally been superceded by relationship
database systems.
Knowledge Management: A concept where an organi-
zation deliberately and comprehensively gathers, orga-
nizes, and analyzes its knowledge, then shares it inter-
nally and sometimes externally.
TEAM LinG
Best Practices in Data Warehousing from the Federal Perspective
Legacy System: Typically, a database management
system in which an organization has invested consider-
able time and money and resides on a mainframe or
minicomputer.
Outsourcing: Acquiring services or products from
an outside supplier or manufacturer in order to cut costs
and/or procure outside expertise.
Performance Metrics: Key measurements of sys-
tem attributes that is used to determine the success of
the process.
Pivot Tables: An interactive table found in most
spreadsheet programs that quickly combines and com-
pares typically large amounts of data. One can rotate its
rows and columns to see different arrangements of the
source data, and also display the details for areas of
interest.
Terabyte: A unit of memory or data storage capacity
equal to roughly 1,000 gigabytes. B
Total Cost of Ownership: Developed by Gartner
Group, an accounting method used by organizations
seeking to identify their both direct and indirect sys-
tems costs.
NOTE
The views expressed in this article are those of the
author and do not reflect the official policy or position
of the National Defense University, the Department of
Defense or the U.S. Government.
99
TEAM LinG
100
Bibliomining for Library Decision-Making
Scott Nicholson
Syracuse University School of Information Studies, USA
Jeffrey Stanton
Syracuse University School of Information Studies, USA
INTRODUCTION
Most people think of a library as the little brick building
in the heart of their community or the big brick building in
the center of a college campus. However, these notions
greatly oversimplify the world of libraries. Most large
commercial organizations have dedicated in-house li-
brary operations, as do schools; nongovernmental orga-
nizations; and local, state, and federal governments. With
the increasing use of the World Wide Web, digital librar-
ies have burgeoned, serving a huge variety of different
user audiences. With this expanded view of libraries, two
key insights arise. First, libraries are typically embedded
within larger institutions. Corporate libraries serve their
corporations, academic libraries serve their universities,
and public libraries serve taxpaying communities who
elect overseeing representatives. Second, libraries play a
pivotal role within their institutions as repositories and
providers of information resources. In the provider role,
libraries represent in microcosm the intellectual and learn-
ing activities of the people who comprise the institution.
This fact provides the basis for the strategic importance
of library data mining: By ascertaining what users are
seeking, bibliomining can reveal insights that have mean-
ing in the context of the librarys host institution.
Use of data mining to examine library data might be
aptly termed bibliomining. With widespread adoption of
computerized catalogs and search facilities over the past
quarter century, library and information scientists have
often used bibliometric methods (e.g., the discovery of
patterns in authorship and citation within a field) to
explore patterns in bibliographic information. During the
same period, various researchers have developed and
tested data-mining techniques, which are advanced sta-
tistical and visualization methods to locate nontrivial
patterns in large datasets. Bibliomining refers to the use
of these bibliometric and data-mining techniques to ex-
plore the enormous quantities of data generated by the
typical automated library.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
Forward-thinking authors in the field of library science
began to explore sophisticated uses of library data some
years before the concept of data mining became popular-
ized. Nutter (1987) explored library data sources to sup-
port decision making but lamented that the ability to
collect, organize, and manipulate data far outstrips the
ability to interpret and to apply them (p. 143). Johnston
and Weckert (1990) developed a data-driven expert sys-
tem to help select library materials, and Vizine-Goetz,
Weibel, and Oskins (1990) developed a system for auto-
mated cataloging based on book titles (see also Morris,
1992, and Aluri & Riggs, 1990). A special section of
Library Administration and Management, Mining your
automated system, included articles on extracting data
to support system management decisions (Mancini, 1996),
extracting frequencies to assist in collection decision
making (Atkins, 1996), and examining transaction logs to
support collection management (Peters, 1996).
More recently, Banerjeree (1998) focused on describ-
ing how data mining works and how to use it to provide
better access to the collection. Guenther (2000) discussed
data sources and bibliomining applications but focused
on the problems with heterogeneous data formats.
Doszkocs (2000) discussed the potential for applying
neural networks to library data to uncover possible asso-
ciations between documents, indexing terms, classifica-
tion codes, and queries. Liddy (2000) combined natural
language processing with text mining to discover informa-
tion in digital library collections. Lawrence, Giles, and
Bollacker (1999) created a system to retrieve and index
citations from works in digital libraries. Gutwin, Paynter,
Witten, Nevill-Manning, and Frank (1999) used text min-
ing to support resource discovery.
These projects all shared a common focus on improv-
ing and automating two of the core functions of a library:
acquisitions and collection management. A few authors
have recently begun to address the need to support
management by focusing on understanding library users:
TEAM LinG
Bibliomining for Library Decision-Making
Schulman (1998) discussed using data mining to examine
changing trends in library user behavior; Sallis, Hill,
Jancee, Lovette, and Masi (1999) created a neural network
that clusters digital library users; and Chau (2000) dis-
cussed the application of Web mining to personalize
services in electronic reference.
The December 2003 issue of Information Technology
and Libraries was a special issue dedicated to the
bibliomining process. Nicholson presented an overview
of the process, including the importance of creating a data
warehouse that protects the privacy of users. Zucca
discussed the implementation of a data warehouse in an
academic library. Wormell; Surez-Balseiro, Iribarren-
Maestro, & Casado; and Geyer-Schultz, Neumann, &
Thede used bibliomining in different ways to understand
the use of academic library sources and to create appro-
priate library services.
We extend these efforts by taking a more global view
of the data generated in libraries and the variety of
decisions that those data can inform. Thus, the focus of
this work is on describing ways in which library and
information managers can use data mining to understand
patterns of behavior among library users and staff and
patterns of information resource use throughout the insti-
tution.
MAIN THRUST
Integrated Library Systems and Data
Warehouses
Most managers who wish to explore bibliomining will
need to work with the technical staff of their Integrated
Library System (ILS) vendors to gain access to the data-
bases that underlie the system and create a data ware-
house. The cleaning, preprocessing, and anonymizing of
the data can absorb a significant amount of time and effort.
Only by combining and linking different data sources,
however, can managers uncover the hidden patterns that
can help them understand library operations and users.
Exploration of Data Sources
Available library data sources are divided into three
groups for this discussion: data from the creation of the
library, data from the use of the collection, and data from
external sources not normally included in the ILS.
ILS Data Sources from the Creation of
the Library System B
Bibliographic Information
One source of data is the collection of bibliographic
records and searching interfaces that represents materi-
als in the library, commonly known as the Online Public
Access Catalog (OPAC). In a digital library environment,
the same type of information collected in a bibliographic
library record can be collected as metadata. The concepts
parallel those in a traditional library: Take an agreed-upon
standard for describing an object, apply it to every object,
and make the resulting data searchable. Therefore, digital
libraries use conceptually similar bibliographic data
sources to traditional libraries.
Acquisitions Information
Another source of data for bibliomining comes from
acquisitions, where items are ordered from suppliers and
tracked until they are received and processed. Because
digital libraries do not order physical goods, somewhat
different acquisition methods and vendor relationships
exist. Nonetheless, in both traditional and digital library
environments, acquisition data have untapped potential
for understanding, controlling, and forecasting informa-
tion resource costs.
ILS Data Sources from Usage of the
Library System
User Information
In order to verify the identity of users who wish to use
library services, libraries maintain user databases. In
libraries associated with institutions, the user database is
closely aligned with the organizational database. Sophis-
ticated public libraries link user records through zip codes
with demographic information in order to learn more about
their user population. Digital libraries may or may not have
any information about their users, based upon the login
procedure required. No matter what data are captured
about the patron, it is important to ensure that the iden-
tification information about the patron is separated from
the demographic information before this information is
stored in a data warehouse; doing so protects the privacy
of the individual.
101
TEAM LinG

Circulation and Usage Information
The richest sources of information about library user
behavior are circulation and usage records. Legal and
ethical issues limit the use of circulation data, however. A
data warehouse can be useful in this situation, because
basic demographic information and details about the cir-
culation could be recorded without infringing upon the
privacy of the individual.
Digital library services have a greater difficulty in
defining circulation, as viewing a page does not carry the
same meaning as checking a book out of the library,
although requests to print or save a full text information
resource might be similar in meaning. Some electronic full-
text services already implement the server-side capture of
such requests from their user interfaces.
Searching and Navigation Information
The OPAC serves as the primary means of searching for
works owned by the library. Additionally, because most
OPACs use a Web browser interface, users may also
access bibliographic databases, the World Wide Web,
and other online resources during the same session; all
this information can be useful in library decision making.
Digital libraries typically capture logs from users who are
searching their databases and can track, through
clickstream analysis, the elements of Web-based services
visited by users. In addition, the combination of a login
procedure and cookies allows the connection of user
demographics to the services and searches they used in a
session.
External Data Sources
Reference Desk Interactions
In the typical face-to-face or telephone interaction with a
library user, the reference librarian records very little infor-
mation about the interaction. Digital reference transac-
tions, however, occur through an electronic format, and
the transaction text can be captured for later analysis,
which provides a much richer record than is available in
traditional reference work. The utility of these data can be
increased if identifying information about the user can be
captured as well, but again, anonymization of these trans-
actions is a significant challenge.
Item Use Information
Fussler and Simon (as cited in Nutter, 1987) estimated that
75 to 80% of the use of materials in academic libraries is in
house. Some types of materials never circulate, and there-
102
Bibliomining for Library Decision-Making
fore, tracking in-house use is also vital in discovering
patterns of use. This task becomes much easier in a
digital library, as Web logs can be analyzed to discover
what sources the users examined.
Interlibrary Loan and Other Outsourcing
Services
Many libraries use interlibrary loan and/or other
outsourcing methods to get items on a need-by-need
basis for users. The data produced by this class of
transactions will vary by service but can provide a
window to areas of need in a library collection.
Applications of Bibliomining through a
Data Warehouse
Bibliomining can provide an understanding of the indi-
vidual sources listed previously in this article; however,
much more information can be discovered when sources
are combined through common fields in a data ware-
house.
Bibliomining to Improve Library Services
Most libraries exist to serve the information needs of
users, and therefore, understanding the needs of indi-
viduals or groups is crucial to a librarys success. For
many decades, librarians have suggested works; market
basket analysis can provide the same function through
usage data in order to aid users in locating useful works.
Bibliomining can also be used to determine areas of
deficiency and to predict future user needs. Common
areas of item requests and unsuccessful searches may
point to areas of collection weakness. By looking for
patterns in high-use items, librarians can better predict
the demand for new items.
Virtual reference desk services can build a database
of questions and expert-created answers, which can be
used in a number of ways. Data mining could be used to
discover patterns for tools that will automatically assign
questions to experts based upon past assignments. In
addition, by mining the question/answer pairs for pat-
terns, an expert system could be created that can provide
users an immediate answer and a pointer to an expert for
more information.
Bibliomining for Organizational Decision
Making Within the Library
Just as the user behavior is captured within the ILS, the
behavior of library staff can also be discovered by con-
TEAM LinG
Bibliomining for Library Decision-Making
necting various databases to supplement existing perfor-
mance review methods. Although monitoring staff through
their performance may be an uncomfortable concept,
tighter budgets and demands for justification require
thoughtful and careful performance tracking. In addition,
research has shown that incorporating clear, objective
measures into performance evaluations can actually im-
prove the fairness and effectiveness of those evaluations
(Stanton, 2000).
Low-use statistics for a work may indicate a problem
in the selection or cataloging process. Looking at the
associations between assigned subject headings, call
numbers, and keywords, along with the responsible party
for the catalog record, may lead to a discovery of system
inefficiencies. Vendor selection and price can be exam-
ined in a similar fashion to discover if a staff member
consistently uses a more expensive vendor when cheaper
alternatives are available. Most libraries acquire works
both by individual orders and through automated order-
ing plans that are configured to fit the size and type of that
library. Although these automated plans do simplify the
selection process, if some or many of the works they
recommend go unused, then the plan might not be cost
effective. Therefore, merging the acquisitions and circu-
lation databases and seeking patterns that predict low use
can aid in appropriate selection of vendors and plans.
Bibliomining for External Reporting and
Justification
The library may often be able to offer insights to their
parent organization or community about their user base
through patterns detected with bibliomining. In addition,
library managers are often called upon to justify the
funding for their library when budgets are tight. Likewise,
managers must sometimes defend their policies, particu-
larly when faced with user complaints. Bibliomining can
provide the data-based justification to back up the anec-
dotal evidence usually used for such arguments.
Bibliomining of circulation data can provide a number
of insights about the groups who use the library. By
clustering the users by materials circulated and tying
demographic information into each cluster, the library can
develop conceptual user groups that provide a model of
the important constituencies of the institutions user
base; this grouping, in turn, can fulfill some common
organizational needs for understanding where common
interests and expertise reside in the user community. This
capability may be particularly valuable within large orga-
nizations where research and development efforts are
dispersed over multiple locations.
FUTURE TRENDS
B
Consortial Data Warehouses
One future path of bibliomining is to combine the data
from multiple libraries through shared data warehouses.
This merger will require standards if the libraries use
different systems. One such standard is the COUNTER
project (2004), which is a standard for reporting the use of
digital library resources. Libraries working together to
pool their data will be able to gain a competitive advantage
over publishers and have the data needed to make better
decisions. This type of data warehouse can power evi-
dence-based librarianship, another growing area of re-
search (Eldredge, 2000).
Combining these data sources will allow library sci-
ence research to move from making statements about a
particular library to making generalizations about
librarianship. These generalizations can then be tested on
other consortial data warehouses and in different settings
and may be the inspiration for theories. Bibliomining and
other forms of evidence-based librarianship can therefore
encourage the expansion of the conceptual and theoreti-
cal frameworks supporting the science of librarianship.
Bibliomining, Web Mining, and Text
Mining
Web mining is the exploration of patterns in the use of
Web pages. Bibliomining uses Web mining as its base but
adds some knowledge about the user. This aids in one of
the shortcomings of Web mining many times, nothing
is known about the user. This lack still holds true in some
digital library applications; however, when users access
password-protected areas, the library has the ability to
map some information about the patron onto the usage
information. Therefore, bibliomining uses tools from Web
usage mining but has more data available for pattern
discovery.
Text mining is the exploration of the context of text in
order to extract information and understand patterns. It
helps to add information to the usage patterns discovered
through bibliomining. To use terms from information
science, bibliomining focuses on patterns in the data that
label and point to the information container, while text
mining focuses on the information within that container.
In the future, organizations that fund digital libraries can
look to text mining to greatly improve access to materials
beyond the current cataloging/metadata solutions.
The quality and speed of text mining continues to
improve. Liddy (2000) has researched the extraction of
103
TEAM LinG

information from digital texts; implementing these tech-
nologies can allow a digital library to move from suggest-
ing texts that might contain the answer to just providing
the answer by extracting it from the appropriate text or
texts. The use of such tools risks taking textual material
out of context and also provides few hints about the
quality of the material, but if these extractions were links
directly into the texts, then context could emerge along
with an answer. This situation could provide a substantial
asset to organizations that maintain large bodies of tech-
nical texts, because it would promote rapid, universal
access to previously scattered and/or uncataloged mate-
rials.
Example of Hybrid Approach
Hwang and Chuang (in press) have recently combined
bibliomining, Web mining, and text mining in a
recommender system for an academic library. They started
by using data mining on Web usage data for articles in a
digital library and combining that information with infor-
mation about the users. They then built a system that
looked at patterns between works based on their content
by using text mining. By combing these two systems into
a hybrid system, they found that the hybrid system
provides more accurate recommendations for users than
either system taken separately. This example is a perfect
representation of the future of bibliomining and how it can
be used to enhance the text-mining research projects
already in progress.
CONCLUSION
Libraries have gathered data about their collections and
users for years but have not always used those data for
better decision making. By taking a more active approach
based on applications of data mining, data visualization,
and statistics, these information organizations can get a
clearer picture of their information delivery and manage-
ment needs. At the same time, libraries must continue to
protect their users and employees from the misuse of
personally identifiable data records. Information discov-
ered through the application of bibliomining techniques
gives the library the potential to save money, provide
more appropriate programs, meet more of the users infor-
mation needs, become aware of the gaps and strengths of
their collection, and serve as a more effective information
source for its users. Bibliomining can provide the data-
based justifications for the difficult decisions and fund-
ing requests library managers must make.
104
Bibliomining for Library Decision-Making
REFERENCES
Atkins, S. (1996). Mining automated systems for collec-
tion management. Library Administration & Manage-
ment, 10(1), 16-19.
Banerjee, K. (1998). Is data mining right for your library?
Computer in Libraries, 18(10), 28-31.
Chau, M. Y. (2000). Mediating off-site electronic reference
services: Human-computer interactions between libraries
and Web mining technology. IEEE Fourth International
Conference on Knowledge-Based Intelligent Engineer-
ing Systems & Allied Technologies, USA, 2 (pp. 695-699).
Chaudhry, A. S. (1993). Automation systems as tools of
use studies and management information. IFLA Journal,
19(4), 397-409.
COUNTER (2004). COUNTER: Counting online usage of
networked electronic resources. Retrieved from http://
www.projectcounter.org/about.html
Doszkocs, T. E. (2000). Neural networks in libraries: The
potential of a new information technology. Retrieved
from http://web.simmons.edu/~chen/nit/NIT%2791/
027~dos.htm
Eldredge, J. (2000). Evidence-based librarianship: An
overview. Bulletin of the Medical Library Association,
88(4), 289-302.
Geyer-Schulz, A., Neumann, A., & Thede, A. (2003). An
architecture for behavior-based library recommender sys-
tems. Information Technology and Libraries, 22(4), 165-
174.
Guenther, K. (2000). Applying data mining principles to
library data collection. Computers in Libraries, 20(4), 60-
63.
Gutwin, C., Paynter, G., Witten, I., Nevill-Manning, C., &
Frank, E. (1999). Improving browsing in digital libraries
with keyphrase indexes. Decision Support Systems, 2I,
81-104.
Hwang, S., & Chuang, S. (in press). Combining article
content and Web usage for literature recommendation in
digital libraries. Online Information Review.
Johnston, M., & Weckert, J. (1990). Selection advisor: An
expert system for collection development. Information
Technology and Libraries, 9(3), 219-225.
Lawrence, S., Giles, C. L., & Bollacker, K. (1999). Digital
libraries and autonomous citation indexing. IEEE Com-
puter, 32(6), 67-71.
TEAM LinG
Bibliomining for Library Decision-Making
Liddy, L. (2000, November/December). Text mining. Bul-
letin of the American Society for Information Science, 13-
14.
Mancini, D. D. (1996). Mining your automated system for
systemwide decision making. Library Administration &
Management, 10(1), 11-15.
Morris, A. (Ed.). (1992). Application of expert systems in
library and information centers. London: Bowker-Saur.
Nicholson, S. (2003). The bibliomining process: Data
warehousing and data mining for library decision-making.
Information Technology and Libraries, 22(4), 146-151.
Nutter, S. K. (1987). Online systems and the management
of collections: Use and implications. Advances in Library
Automation Networking, 1, 125-149.
Peters, T. (1996). Using transaction log analysis for library
management information. Library Administration &
Management, 10(1), 20-25.
Sallis, P., Hill, L., Janee, G., Lovette, K., & Masi, C. (1999).
A methodology for profiling users of large interactive
systems incorporating neural network data mining tech-
niques. Proceedings of the 1999 Information Resources
Management Association International Conference (pp.
994-998).
Schulman, S. (1998). Data mining: Life after report genera-
tors. Information Today, 15(3), 52.
Stanton, J. M. (2000). Reactions to employee performance
monitoring: Framework, review, and research directions.
Human Performance, 13, 85-113.
Surez-Balseiro, C. A., Iribarren-Maestro, I., Casado, E. S.
(2003). A study of the use of the Carlos III University of
Madrid Librarys online database service in Scientific
Endeavor. Information Technology and Libraries, 22(4),
179-182.
Vizine-Goetz, D., Weibel, S., & Oskins, M. (1990). Auto-
mating descriptive cataloging. In R. Aluri, & D. Riggs
(Eds.), Expert systems in libraries (pp. 123-127). Norwood,
NJ: Ablex Publishing Corporation.
Wormell, I. (2003). Matching subject portals with the
research environment. Information Technology and Li-
braries, 22(4), 158-166.
Zucca, J. (2003). Traces in the clickstream: Early work on
a management information repository at the University of
Pennsylvania. Information Technology and Libraries,
22(4), 175-178.
KEY TERMS
B
Bibliometrics: The study of regularities in citations,
authorship, subjects, and other extractable facets from
scientific communication by using quantitative and visu-
alization techniques. This study allows researchers to
understand patterns in the creation and documented use
of scholarly publishing.
Bibliomining: The application of statistical and pat-
tern-recognition tools to large amounts of data associ-
ated with library systems in order to aid decision making
or to justify services. The term bibliomining comes from
the combination of bibliometrics and data mining, which
are the two main toolsets used for analysis.
Data Warehousing: The gathering and cleaning of
data from disparate sources into a single database, which
is optimized for exploration and reporting. The data ware-
house holds a cleaned version of the data from operational
systems, and data mining requires the type of cleaned
data that live in a data warehouse.
Evidence-Based Librarianship: The use of the best
available evidence, combined with the experiences of
working librarians and the knowledge of the local user
base, to make the best decisions possible (Eldredge,
2000).
Integrated Library System: The automation system
for libraries that combines modules for cataloging, acqui-
sition, circulation, end-user searching, database access,
and other library functions through a common set of
interfaces and databases.
Online Public Access Catalog (OPAC): The module
of the Integrated Library System designed for use by the
public to allow discovery of the librarys holdings through
the searching of bibliographic surrogates. As libraries
acquire more digital materials, they are linking those
materials to the OPAC entries.
NOTE
This work is based on Nicholson, S., & Stanton, J. (2003).
Gaining strategic advantage through bibliomining: Data
mining for management decisions in corporate, special,
digital, and traditional libraries. In H. Nemati & C. Barko
(Eds.). Organizational data mining: Leveraging enter-
prise data resources for optimal performance (pp. 247
262). Hershey, PA: Idea Group.
105
TEAM LinG
106
Biomedical Data Mining Using RBF Neural
Networks
Feng Chu
Nanyang Technological University, Singapore
Lipo Wang
Nanyang Technological University, Singapore
INTRODUCTION
Accurate diagnosis of cancers is of great importance for
doctors to choose a proper treatment. Furthermore, it also
plays a key role in the searching for the pathology of
cancers and drug discovery. Recently, this problem at-
tracts great attention in the context of microarray technol-
ogy. Here, we apply radial basis function (RBF) neural
networks to this pattern recognition problem. Our experi-
mental results in some well-known microarray data sets
indicate that our method can obtain very high accuracy
with a small number of genes.
BACKGROUND
Microarray is also called gene chip or DNA chip. It is a
newly appeared biotechnology that allows biomedical
researchers monitor thousands of genes simultaneously
(Schena, Shalon, Davis, & Brown, 1995). Before the ap-
pearance of microarrays, a traditional molecular biology
experiment usually works on only one gene or several
genes, which makes it difficult to have a whole picture
of an entire genome. With the help of microarrays, re-
searchers are able to monitor, analyze and compare ex-
pression profiles of thousands of genes in one experiment.
On account of their features, microarrays have been
used in various tasks such as gene discovery, disease
diagnosis, and drug discovery. Since the end of the last
century, cancer classification based on gene expression
profiles has attracted great attention in both the biologi-
cal and the engineering fields. Compared with traditional
cancer diagnostic methods based mainly on the morpho-
logical appearances of tumors, the method using gene
expression profiles is more objective, accurate, and reli-
able. More importantly, some types of cancers have
subtypes with very similar appearances that are very hard
to be classified by traditional methods. It has been proven
that gene expression has a good capability to clarify this
previously muddy problem.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Thus, to develop accurate and efficient classifiers
based on gene expression becomes a problem of both
theoretical and practical importance. Recent approaches
on this problem include artificial neural networks (Khan et
al., 2001), support vector machines (Guyon, Weston,
Barnhill, & Vapnik, 2002), k-nearest neighbor (Olshen &
Jain, 2002), nearest shrunken centroids (Tibshirani, Hastie,
Narashiman, & Chu, 2002), and so on.
A solution to this problem is to find out a group of
important genes that contribute most to differentiate
cancer subtypes. In the meantime, we should also provide
proper algorithms that are able to make correct prediction
based on the expression profiles of those genes. Such
work will benefit early diagnosis of cancers. In addition,
it will help doctors choose proper treatment. Furthermore,
it also throws light on the relationship between the can-
cers and those important genes.
From the point of view of machine learning and statis-
tical learning, cancer classification using gene expression
profiles is a challenging problem. The reason lies in the
following two points. First, typical gene expression data
sets usually contain very few samples (from several to
several tens for each type of cancers). In other words, the
training data are scarce. Second, such data sets usually
contain a large number of genes, for example, several
thousands. That is, the data are high dimensional. There-
fore, this is a special pattern recognition problem with
relatively small number of patterns and very high dimen-
sionality. To provide such a problem with a good solution,
appropriate algorithms should be designed.
In fact, a number of different approaches such as k-
nearest neighbor (Olshen and Jain, 2002), support vector
machines (Guyon et al.,2002), artificial neural networks
(Khan et al., 2001) and some statistical methods have
been applied to this problem since 1995. Among these
approaches, some obtained very good results. For ex-
ample, Khan et al. (2001) classified small round blue cell
tumors (SRBCTs) with 100% accuracy by using 96 genes.
Tibshirani et al. (2002) successfully classified SRBCTs
with 100% accuracy by using only 43 genes. They also
TEAM LinG
Biomedical Data Mining Using RBF Neural Networks
classified three different subtypes of lymphoma with
100% accuracy by using 48 genes. (Tibshirani, Hastie,
Narashiman, & Chu, 2003)
However, there are still a lot of things can be done to
improve present algorithms. In this work, we use and
compare two gene selection schemes, i.e., principal com-
ponents analysis (PCA) (Simon, 1999) and a t-test-based
method (Tusher, Tibshirani, & Chu, 2001). After that, we
introduce an RBF neural network (Fu & Wang, 2003) as the
classification algorithm.
MAIN THRUST
After a comparative study of gene selection methods, a
detailed description of the RBF neural network and some
experimental results are presented in this section.
Microarray Data Sets
We analyze three well-known gene expression data sets,
i.e., the SRBCT data set (Khan et al., 2001), the lymphoma
data set (Alizadeh et al., 2000), and the leukemia data set
(Golub et al., 1999).
The lymphoma data set (http://llmpp.nih.gov/lym-
phoma) (Alizadeh et al., 2000) contains 4026 well mea-
sured clones belonging to 62 samples. These samples
belong to following types of lymphoid malignancies:
diffuse large B-cell lymphoma (DLBCL, 42 samples), fol-
licular lymphoma (FL, nine samples) and chronicle lym-
phocytic leukemia (CLL, 11 samples). In this data set, a
small part of data is missing. A k-nearest neighbor algo-
rithm was used to fill those missing values (Troyanskaya
et al., 2001).
The SRBCT data set (http://research.nhgri.nih.gov/
microarray/Supplement/) (Khan et al., 2001) contains the
expression data of 2308 genes. There are totally 63 training
samples and 25 testing samples. Five of the testing samples
are not SRBCTs. The 63 training samples contain 23 Ewing
family of tumors (EWS), 20 rhabdomyosarcoma (RMS), 12
neuroblastoma (NB), and eight Burkitt lymphomas (BL).
And the 20 testing samples contain six EWS, five RMS, six
NB, and three BL.
The leukemia data set (http://www-genome.wi.mit.edu/
cgi-\\bin /cancer/publications) (Golub et al., 1999) has
two types of leukemia, i.e., acute myeloid leukemia (AML)
and acute lymphoblastic leukemia (ALL). Among these
samples, 38 of them are for training; the other 34 blind
samples are for testing. The entire leukemia data set
contains the expression data of 7,129 genes. Different
with the cDNA microarray data, the leukemia data are
oligonucleotide microarray data. Because such expres-
sion data are raw data, we need to normalize them to reduce
the systemic bias induced during experiments. We follow
the normalization procedure used by Dudoit, Fridlyand, B
and Speed (2002). Three preprocessing steps were ap-
plied: (a) thresholding with floor of 100 and ceiling of
16000; (b) filtering, exclusion of genes with max/min<5 or
(max-min)<500. max and min refer to the maximum and the
minimum of the gene expression values, respectively; and
(c) base 10 logarithmic transformation. There are 3571
genes survived after these three steps. After that, the data
were standardized across experiments, i.e., minus the
mean and divided by the standard deviation of each
experiment.
Methods for Gene Selection
As mentioned in the former part, the gene expression data
are very high-dimensional. The dimension of input pat-
terns is determined by the number of genes used. In a
typical microarray experiment, usually several thousands
of genes take part in. Therefore, the dimension of patterns
is several thousands. However, only a small number of the
genes contribute to correct classification; some others
even act as noise. Gene selection can eliminate the
influence of such noise. Furthermore, the fewer the
genes used, the lower the computational burden to the
classifier. Finally, once a smaller subset of genes is iden-
tified as relevant to a particular cancer, it helps biomedical
researchers focus on these genes that contribute to the
development of the cancer. The process of gene selection
is ranking genes discriminative ability first and then
retaining the genes with high ranks.
As a critical step for classification, gene selection has
been studied intensively in recent years. There are two
main approaches, one is principal component analysis
(PCA) (Simon, 1999), perhaps the most widely used method;
the other is a t-test-based approach which has been more
and more widely accepted. In the important papers
(Alizadeh et al., 2000; Khan et al., 2001), PCA was used.
The basic idea of PCA is to find the most informative
genes that contain most of the information in the data set.
Another approach is based on t-test that is able to mea-
sure the difference between two groups. Thomas, Olsen,
Tapscott, and Zhao. (2001) recommended this method.
Tusher et al. (2001) and Pan (2002) also proposed their
method based on t-test, respectively. Besides these two
main methods, there are also some other methods. For
example, a method called Markov blanket was proposed
by Xing, Jordan, and Karp (2001). Li, Weinberg, Darden,
and Pedersen (2001) applied another method which com-
bined genetic algorithm and K-nearest neighbor.
PCA (Simon, 1999) aims at reducing the input dimen-
sion by transforming the input space into a new space
described by principal components (PCs). All the PCs are
107
TEAM LinG

orthogonal and they are ordered according to the absolute
value of their eigenvalues. The k-th PC is the vector with
the k-th largest eigenvalue. By leaving out the vectors with
small eigenvalues, the input spaces dimension is reduced.
In fact, the PCs indicate the directions with largest
variations of input vectors. Because PCA chooses vectors
with largest eigenvalues, it covers directions with largest
variations of vectors. In the directions determined by the
vectors with small eigenvalues, the variations of vectors
are very small. In a word, PCA intends to capture the most
informative directions (Simon, 1999).
We tested PCA in the lymphoma data set (Alizadeh et
al., 2000). We obtained 62 PCs from the 4026 genes in the
data set by using PCA. Then, we ranked those PCs accord-
ing to their eigenvalues (absolute values). Finally, we used
our RBF neural network that will be introduced in the latter
part to classify the lymphoma data set.
At first, we randomly divided the 62 samples into two
parts, 31 samples for training and the other 31 samples for
testing. We then input the 62 PCs one by one to the RBF
network according to their eigenvalue ranks starting with
the PC ranked one. That is, we first used only a single PC
that is ranked 1 as the input to the RBF network. We trained
the network with the training data and subsequently tested
the network with the testing data. We repeated this pro-
cess with the top two PCs, then the top three PCs, and so
on. Figure 1 shows the testing error. From this result, we
found that the RBF network can not reach 100% accuracy.
The best testing accuracy is 93.55% that happened when 36
or 61 PCs were input to the classifier. The classification result
using the t-test-based gene selection method will be shown
in the next section, which is much better than PCA approach.
The t-test-based gene selection measures the differ-
ence of genes distribution using a t-test based scoring
scheme, i.e., t-score (TS). After that, only the genes with
the highest TSs are to be put into our classifier. The TS of
gene i is defined as follows (Tusher et al., 2001):
x ik x i
TS i = max , k = 1,2,...K
d
k i s
x ik = jC k x ij / nk
n
x i = xij / n
j =1
(x
where: 1 2
si2 = ij
nK k
jC k
d k = 1 / nk + 1 / n
There are K classes. max {yk, k = 1,2,..K} is the maximum
of all y k, k = 1,2,..K. Ck refers to class k that includes nk
samples. xij is the expression value of gene i in sample j. x ik
is the mean expression value in class k for gene i. n is the
108
Biomedical Data Mining Using RBF Neural Networks
total number of samples. x i is the general mean expres-
sion value for gene i. s i is the pooled within-class
standard deviation for gene i. Actually, the t-score used
here is a t-statistics between a specific class and the
overall centroid of all the classes.
To compare the t-test-based method with PCA, we
also applied it to the lymphoma data set with the same
procedure as what we did by using PCA. This method
obtained 100% accuracy with only the top six genes. The
results are shown in Figure 1. This comparison indicated
that the t-test-based method was much better than PCA
in this problem.
An RBF Neural Network
An RBF neural network (Haykin, 1999) has three layers.
The first layer is an input layer; the second layer is a
hidden layer that includes some radial basis functions,
also known as hidden kernels; the third layer is an output
layer. An RBF neural network can be regarded as a
mapping of the input domain X onto the output domain
Y. Mathematically, an RBF neural network can be de-
scribed as follows:
N
y m ( x) = wmi G ( x t i ) + bm , i=1,2,,N; m=1,2,M
i =1
Here stands for the Euclidean norm. M is the
number of outputs. N is the number of hidden kernels.
y m (x) is the output m corresponding to the input x. t i is
the position of kernel i. wmi is the weight between the
kernel i and the output m. bm is the bias on the output m.
G ( x t i ) is the kernel function. Usually, an RBF neural
network uses Gaussian kernel functions as follows:
Figure 1. Classification results of using PCA and the t-
test-based method as gene selection methods
1
)
0. 8
x ik
0. 6
PCA
Er r or
t - t est
0. 4
0. 2
0
1 6 11 16 21 26 31 36 41 46 51 56 61
Number of genes
TEAM LinG
Biomedical Data Mining Using RBF Neural Networks
2 FUTURE TRENDS
x t i
G ( x t i ) = exp( 2
)
2 i
where i is the radius of the kernel i.
The main steps to construct an RBF neural network
include: (a) determining the positions of all the kernels (ti);
(b) determining the radius of each kernel ( i ); and (c)
calculating the weights between each kernel and each
output node.
In this paper, we use a novel RBF neural network
proposed by Fu and Wang (Fu and Wang, 2003), which
allows for large overlaps of hidden kernels belonging to
the same class.
Results
In the SRBCT data set, we first ranked the entire 2308
genes according to their TSs (Tusher et al., 2001). Then
we picked out 96 genes with the highest TSs. We applied
our RBF neural network to classify the SRBCT data set.
The SRBCT data set contains 63 samples for training and
20 blind samples for testing. We input the selected 96
genes one by one to the RBF network according to their
TS ranks starting with the gene ranked one. We repeated
this process with the top two genes, then the top three
genes, and so on. Figure 2 shows the testing errors with
respect to the number of genes. The testing error decreased
to 0 when the top seven genes were input into the RBF
network.
In the leukemia data set, we chose 56 genes with the
highest TSs (Tusher et al., 2001). We followed the same
procedure as in the SRBCT data set. We did classification
with 1 gene, then two genes, then three genes and so on.
Our RBF neural network got an accuracy of 97.06%, i.e.
one error in all 34 samples, when 12, 20, 22, 32 genes were
input, respectively.
Figure 2. The testing result in the SRBCT data set
0. 5
0. 4
0. 3
Er r or
0. 2
0. 1
0 0
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96
Number of genes
B
Until now, the focus of work is investigating the informa-
tion with statistical importance in microarray data sets. In
the near future, we will try to incorporate more biological
knowledge into our algorithm, especially the correlations
of genes.
In addition, with more and more microarray data sets
produced in laboratories around the world, we will try to
mine multi-data-set with our RBF neural network, i.e., we
will try to process the combined data sets. Such an attempt
will hopefully bring us a much broader and deeper insight
into those data sets.
CONCLUSION
Through our experiments, we conclude that the t-test-
based gene selection method is an appropriate feature
selection/dimension reduction approach, which can find
more important genes than PCA can.
The results in the SRBCT data set and the leukemia
data set proved the effectiveness of our RBF neural
network. In the SRBCT data set, it obtained 100% accu-
racy with only seven genes. In the leukemia data set, it
made only one error with 12, 20, 22, and 32 genes, respec-
tively. In view of this, we also conclude that our RBF
neural network outperforms almost all the previously
published methods in terms of accuracy and the number
of genes required.
REFERENCES
Alizadeh, A.A. et al. (2000). Distinct types of diffuse large
B-cell lymphoma identified by gene expression profiling.
Nature, 403, 503-511.
Figure 3. The testing result in the leukemia data set
0. 2
0. 15
Er r or
0. 1
0. 05
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55
Number of genes
109
TEAM LinG

Dudoit, S., Fridlyand, J., & Speed, J. (2002). Comparison
of discrimination methods for the classification of tumors
using gene expression data. Journal of American Statis-
tics Association, 97, 77-87.
Fu, X., & Wang, L. (2003). Data dimensionality reduction
with application to simplifying RBF neural network struc-
ture and improving classification performance. IEEE Trans.
Syst., Man, Cybernetics. Part B: Cybernetics, 33, 399-409.
Golub, T.R. et al. (1999). Molecular classification of can-
cer: class discovery and class prediction by gene expres-
sion monitoring. Science, 286, 531-537.
Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. (2002).
Gene selection for cancer classification using support
vector machines. Machine Learning, 46, 389-422.
Haykin, S. (1999). Neural network, a comprehensive
foundation (2nd ed.). New Jersey, U.S.A: Prentice-Hall, Inc.
Khan, J.M. et al. (2001). Classification and diagnostic
prediction of cancers using gene expression profiling and
artificial neural networks. Nature Medicine, 7, 673-679.
Li, L., Weinberg, C.R., Darden, T.A., & Pedersen, L.G.
(2001). Gene selection for sample classification based on
gene expression data: Study of sensitivity to choice of
parameters of the GA/KNN method. Bioinformatics, 17,
1131-1142.
Olshen, A.B., & Jain, A.N. (2002). Deriving quantitative
conclusions from microarray expression data.
Bioinformatics, 18, 961-970.
Pan, W. (2002). A comparative review of statistical meth-
ods for discovering differentially expressed genes in repli-
cated microarray experiments. Bioinformatics, 18, 546-554.
Schena, M., Shalon, D., Davis, R.W., & Brown, P.O. (1995).
Quantitative monitoring of gene expression patterns with
a complementary DNA microarray. Science, 270, 467-470.
Thomas, J.G., Olsen, J.M., Tapscott, S.J. & Zhao, L.P.
(2001). An efficient and robust statistical modeling ap-
proach to discover differentially expressed genes using ge-
nomic expression profiles. Genome Research, 11, 1227-1236.
Tibshirani, R., Hastie, T., Narashiman, B., & Chu, G. (2002).
Diagnosis of multiple cancer types by shrunken centroids of
gene expression. Proc. Natl. Acad. Sci. USA, 99, 6567-6572.
Tibshirani, R., Hastie, T., Narashiman, B., & Chu, G. (2003).
Class predication by nearest shrunken centroids with applica-
tions to DNA microarrays. Statistical Science, 18, 104-117.
Troyanskaya, O., Cantor, M, & Sherlock, G., et al. (2001).
Missing value estimation methods for DNA microarrays.
Bioinformatics, 17, 520-525.
110
Biomedical Data Mining Using RBF Neural Networks
Tusher, V.G., Tibshirani, R., & Chu, G. (2001). Significance
analysis of microarrays applied to the ionizing radiation
response. Proc. Natl. Acad. Sci. USA, 98, 5116-5121.
Xing, E.P., Jordan, M.I., & Karp, R.M. (2001). Feature
selection for high-dimensional genomic microarray data.
Proceedings of the Eighteenth International Conference
on Machine Learning (pp. 601-608). Morgan Kaufmann
Publishers, Inc.
KEY TERMS
Feature Extraction: Feature extraction is the process
to obtain a group of features with the characters we need
from the original data set. It usually uses a transform (e.g.
principal component analysis) to obtain a group of fea-
tures at one time of computation.
Feature Selection: Feature selection is the process to
select some features we need from all the original features.
It usually measures the character (e.g. t-test score) of each
feature first, then, chooses some features we need.
Gene Expression Profile: Through microarray chips,
an image that describes to what extent genes are ex-
pressed can be obtained. It usually uses red to indicate the
high expression level and uses green to indicate the low
expression level. This image is also called a gene expres-
sion profile.
Microarray: A Microarray is also called a gene chip
or a DNA chip. It is a newly appeared biotechnology that
allows biomedical researchers monitor thousands of genes
simultaneously.
Principal Components Analysis: Principal compo-
nents analysis transforms one vector space into a new
space described by principal components (PCs). All the
PCs are orthogonal to each other and they are ordered
according to the absolute value of their eigenvalues. By
leaving out the vectors with small eigenvalues, the dimen-
sion of the original vector space is reduced.
Radial Basis Function (RBF) Neural Network: An
RBF neural network is a kind of artificial neural network.
It usually has three layers, i.e., an input layer, a hidden
layer, and an output layer. The hidden layer of an RBF
neural network contains some radial basis functions, such
as Gaussian functions or polynomial functions, to trans-
form input vector space into a new non-linear space. An
RBF neural network has the universal approximation abil-
ity, i.e., it can approximate any function to any accuracy,
as long as there are enough hidden neurons.
TEAM LinG
Biomedical Data Mining Using RBF Neural Networks

T-Test: T-test is a kind of statistical method that Training a Neural Network: Training a neural net-
measures how large the difference is between two groups work means using some known data to build the structure B
of samples. and tune the parameters of this network. The goal of training
is to make the network represent a mapping or a regression we
Testing a Neural Network: To know whether a trained need.
neural network is the mapping or the regression we need,
we test this network with some data that have not been
used in the training process. This procedure is called
testing a neural network.

111

TEAM LinG
112
Building Empirical-Based Knowledge for
Design Recovery
Hee Beng Kuan Tan
Nanyang Technological University, Singapore
Yuan Zhao
Nanyang Technological University, Singapore
INTRODUCTION
Although the use of statistically probable properties is
very common in the area of medicine, it is not so in
software engineering. The use of such properties may
open a new avenue for the automated recovery of de-
signs from source codes. In fact, the recovery of de-
signs can also be called program mining, which in turn
can be viewed as an extension of data mining to the
mining in program source codes.
BACKGROUND
Today, most of the tasks in software verification, test-
ing, and re-engineering remain manually intensive
(Beizer, 1990), time-consuming, and error prone. As
many of these tasks require the recognition of designs
from program source codes, automation of the recogni-
tion is an important means to improve these tasks.
However, many designs are difficult (if not impossible)
to recognize automatically from program source codes
through theoretical knowledge alone (Biggerstaff,
Mitbander, & Webster, 1994; Kozaczynski, Ning, &
Engberts, 1992). Most of the approaches proposed for
the recognition of designs from program source codes
are based on plausible inference (Biggerstaff et al.,
1994; Ferrante, Ottenstein, & Warren, 1987;
Kozaczynski et al., 1992). That is, they are actually
based on empirical-based knowledge (Kitchenham,
Pfleeger, Pickard, Jones, Hoaglin, Emam, & Rosenberg,
2002). However, to the best of our knowledge, the
building of empirical-based knowledge to supplement
theoretical knowledge for the recognition of designs
from program source code has not been formally dis-
cussed in the literature.
This paper introduces an approach for the building
and applying of empirical-based knowledge to supple-
ment theoretical knowledge for the recognition of de-
signs from program source codes. The first section
introduces the proposed approach. The second section
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
discusses the application of the proposed approach for the
recovery of functional dependencies enforced in database
transactions. The final section shows our conclusion.
MAIN THRUST
The Approach
Many types of designs are usually implemented through
a few methods. The use of a method has a direct influ-
ence on the programs that implement the designs. As a
result, these programs may have some certain charac-
teristics. And we may be able to recognize the designs or
their properties through recognizing these characteris-
tics, from either a theoretical or empirical basis or the
combination of the two.
An overview of the approach for building empirical-
based knowledge for design recovery through program
analysis is shown in Figure 1. In the figure, arcs show
interactions between tasks. In the approach, we first
research the designs or their properties, which can be
recognized from some characteristics in the programs
that implement them through automated program analy-
sis. This task requires domain knowledge or experience.
The reason for using design properties is that some
designs could be too complex to recognize directly. In
the latter case, we first recognize the properties, then
use them to infer the designs. We aim for characteris-
tics that are not only sufficient but also necessary for
recognizing designs or their properties. If the charac-
teristics are sufficient but not necessary, even if we
cannot infer the target designs, they do not imply the
nonexistence of the designs. If we cannot find charac-
teristics from which a design can be formally proved,
then we will look for characteristics that have signifi-
cant statistical evidence. These empirical-based char-
acteristics are taken as hypotheses. With the use of
hypotheses and theoretical knowledge, a theory is built
for the inference of designs.
TEAM LinG
Building Empirical-Based Knowledge for Design Recovery

Secondly, we design experiments to validate the hy- Figure 1. An overview of the proposed design recovery
potheses. Some software tools should be developed to B
automate or semiautomate the characterization of the Identify Designs
or
properties stated in the hypotheses. We may merge their Properties
multiple hypotheses together as a single hypothesis for
the convenience of hypothesis testing. An experiment is
Identity Program
designed to conduct a binomial test (Gravetter & Wallnau, Charactertics and Formulate
Hypotheses
2000) for each resulting hypothesis. If altogether we
have k hypotheses denoted by H1,., Hk and we would
like the probability of validity of the proposed design Develop Software Tools
Develop Theory
to aid Experiments for
recovery to be more than or equal to q, then we must Hypotheses Testing
for the Inference of Designs

choose p1,., pk such that p1,., pk q. For each

hypothesis Hj (1 j k), the null and alternate hypoth-
esis of the binomial test states that less than pj*100% Conduct Experiements for
Hypotheses Testing
and equal or more than pj*100%, respectively, of the
cases that Hj holds. That is:
accept hypothesis Develop Algorithms
Hj0 (null hypothesis): probability that Hj holds < p for Automated Design
Recovery
Hj1 (alternate hypothesis): probability that Hj holds reject hypothesis

p
Develop Software Tool
to Implement the Algorithms
For the use of normal approximation for the bino-
mial test, both npj and n(1-pj) must be greater than or
equal to 10. As such, the sample size n must be greater Conduct Experiment
than or equal to max (10/pj, 10/(1-pj)). The experiment to validate the Effectiveness

is designed to draw a random sample of size n to test the

hypothesis. For each case in the sample, the validity of
the hypothesis is examined. The total number of cases,
X, that the hypothesis holds is recorded and substituted
in the following binomial test statistics:
X /n p
j
z= ( p (1 p ) / n )
j j
Let be the Type I error probability. If z is greater
than z, we reject the null hypothesis; otherwise, we
accept the null hypothesis, where the probability of
standard normal model for z z is .
Thirdly, we develop the required software tools and
conduct the experiments to test each hypothesis ac-
cording to the design drawn in the previous step.
Fourthly, if all the hypotheses are accepted, algo-
rithms will be developed for the use of the theory to
automatically recognize the designs from program source
codes. A software tool will also be developed to imple-
ment the algorithms. Some experiments should also be
conducted to validate the effectiveness of the method.
Applying the Proposed Approach for
the Recovery of Functional
Dependencies
Let R be a record type and X be a sequence of attributes
of R. For any record r in R, its sequence of values of the
attributes in X is referred as the X-value of r. Let R be a
record type, and X and Y be sequences of attributes of R.
We say that the functional dependency (FD), X Y of R,
holds at time t, if at time t, for any two R records r and s,
the X-values of r and s are identical, then the Y-values of
r and s are also identical. We say that the functional
dependency holds in a database if, except in the midst of
a transaction execution (which updates some record types
involved in the dependency), the dependency always
holds (Ullman, 1982).
Many of the worlds database applications have been
built on old generation DBMSs (database management
systems). Due to the nature of system development, many

113

TEAM LinG
 Building Empirical-Based Knowledge for Design Recovery
functional dependencies are not discovered in the initial
system development. They are only identified during the
system maintenance stage.
Although keys can be used to implement functional
dependencies in old generation DBMSs, due to the effort
in restructuring databases during the system maintenance
stage, most functional dependencies identified during this
stage are not defined explicitly as keys in the databases.
They are enforced in transactions. Furthermore, most of
the conventional files and relational databases allow only
the definition of one key. As such, most of the candidate
keys are enforced in transactions. As a result, many func-
tional dependencies in a legacy database are enforced in
the transactions that update the database. Our previous
work (Tan, 2004) has proven that if all the transactions that
update a database satisfy a set of properties with reference
to a functional dependency, X Y of R, then the func-
tional dependency holds in the database. Before proceed-
ing further, we shall discuss these properties.
For generality, we shall express a program path in the
form (a1, , an), where ai is either a node or in the form
{ nii ,.., nik }, in which nii ,.., nik are nodes. If ai is in the
form { nii ,.., nik }, then in (a1, , a n), after the predeces-
sor node of a i, the subsequence of nodes, nii ,.., nik ,
may occur any number of times (possibly zero), before
proceeding to the successor of a i.
Before proceeding further, we shall introduce some
notations to represent certain types of nodes in control
flow graphs of transactions that will be used throughout
this paper:
rd(R, W == b): A node that reads or selects an R
record in which the W-value is b if it exists.
mdf(R, W == b, Z1 := c1,.., Z n := cn): A node that
modifies the Z 1-value, .. , Zn-value of an R record,
in which the original W-value is b, to c1,.., cn,
respectively.
ins(R, Z1 := c1,.., Zn := cn): A node that inserts an
R record, in which its Z1-value,.. , Zn-value are
set to c1,.., c n, respectively.
Here, R is a record type. W, Z 1, , Zn are sequences of
R attributes. The values of those attributes that are not
mentioned in the mdf and ins nodes can be modified and
set to any value.
We shall also use xclu_fd(X Y of R) to denote a
node in the control flow graph of a transaction that does
not perform any updating that may lead to the violation of
the functional dependency X Y of R.
A program path in the form (rd(R, X == x), {xclu_fd(X
Y of R)}, {{xclu_fd(X Y of R)}, ins(R, X := x, Y
114
:= y), {xclu_fd(X Y of R)}}), such that if the rd node
successfully reads an R record specified, and y is iden-
tical to the Y-value of the record read, is called an
insertion pattern for enforcing the FD, X Y of R.
A program path in the form ({{xclu_fd(X Y of R)},
mdf(R, X == x0, Y := y), {xclu_fd(X Y of R)}}), such that
all the R records in which the X-values are equal to x 0 are
modified by the mdf node, is called a Y-modification
pattern for enforcing the FD, X Y of R.
A program path in the form (rd(R, X == x), {xclu_fd(X
Y of R)}, {{xclu_fd(X Y of R)}, mdf(R, X == x0, X :=
x, Y := unchange), {xclu_fd(X Y of R)}}), such that the
mdf node is only executed if the rd node does not
successfully read an R record specified, is called an X-
modification pattern for enforcing the FD, X Y of
R.
We have proven the following rules by Tan (2004):
Nonviolation of FD: In a transaction, if all the
nodes that insert R records or modify the at-
tribute X or Y in any program path from the start
node to the end node are always contained in a
sequence of subpaths in the patterns for enforc-
ing the functional dependency, X Y of R, then
the transaction does not violate the functional
dependency.
FD in Database: Each transaction that updates
any record type involved in a functional depen-
dency does not violate the functional dependency
if and only if is a functional dependency
designed in the database.
Theoretically, the property stated in the first rule is
not a necessary property in order for the functional
dependency to hold. As such, we may not be able to
recover all functional dependencies enforced by rec-
ognizing these properties.
Fortunately, other than very exceptional cases, most
of the enforcement of functional dependency does
result to the previously mentioned property. As such,
empirically, the property is usually also necessary for
the functional dependency, X Y of R, to hold in the
database. Thus, we take the following hypothesis.
Hypothesis 1: If a transaction does not violate
the functional dependency, X Y of R, then all
the nodes that insert R records or modify the
attribute X or Y in any program path from the start
node to the end node are always contained in a
sequence of subpaths in the patterns for enforc-
ing the functional dependency.
With the hypothesis, the result discussed by Tan and
Thein (in press) can be extended to the following theorem.
TEAM LinG
Building Empirical-Based Knowledge for Design Recovery
Theorem 1: A functional dependency, X Y of R,
holds in a database if and only if in each transaction
that updates any record type involved in the func-
tional dependency, all the nodes that insert R records
or modify the attribute X or Y in any program path
from the start node to the end node are always
contained in a sequence of subpaths in the patterns
for enforcing the functional dependency.
The proof of sufficiency can be found in previous
work (see Tan & Thein, 2004). The necessity is implied
directly from the hypothesis.
This theorem can be used to recover all the functional
dependencies in a database. We have developed an
algorithm the recovery. The algorithm is similar to the
algorithm presented by Tan and Thein (2004).
We have conducted an experiment to validate the
hypothesis. In this experiment, we get each of our 71
postgraduate students who take the software require-
ments analysis and design course to design the follow-
ing three transactions in pseudocode to update the table,
plant-equipment (plant-name, equipment-name, equip-
ment-manufacturer, manufacturer-address), such that
each transaction ensures that the functional dependency,
equipment-manufacturer manufacturer-address of
plant-equipment holds:
Insert a new tuple in the table to store a user input
each time. The input has four fields, inp-plant-
name, inp-equipment-name, inp-equipment-manu-
facturer and inp-manufacturer-address. In the tuple,
the value of each attribute is set according to the
corresponding input field.
For each user input that has a single field, inp-
equipment-manufacturer, delete the tuple in the
table in which the value of the attribute, equip-
ment-manufacturer, is identical to the value of
inp-equipment-manufacturer.
For each user input that has two fields, old-equip-
ment-manufacturer and new-equipment-manufac-
turer, update the equipment-manufacturer in each
tuple in which the value of equipment-manufac-
turer is identical to the value of old-equipment-
Table 1. The statistics of an experiment
Enforcement of the
Transaction Functional Dependency
Correct Wrong
1 55 16
2 65 6 codes (Embury & Shao, 2001; Tan, Ling, & Goh, 2002;
3 36 35
manufacturer to the value of new-equipment-manu-
facturer. B
We also analyzed 50 transactions from three exist-
ing database applications. Table 1 summarizes the result
of the transactions designed by the students.
We examined each transaction that enforces the
functional dependency. We found that in each of these
transactions, all the nodes that insert R records or
modify the attribute X or Y in any program path from the
start node to the end node are always contained in a
sequence of subpaths in the patterns for enforcing the
functional dependency. As such, Hypothesis 1 holds in
each of these transactions. Because each transaction
that does not enforce the functional dependency cor-
rectly violates the functional dependency, Hypothesis 1
always holds in such a transaction. Therefore, Hypoth-
esis 1 holds in all the 213 transactions. For the 50
transactions that we drew from three existing database
applications, each database application enforces only
one functional dependency in their transactions (other
functional dependencies are implemented as keys in the
databases). We checked each transaction on Hypothesis
1 with respect to the functional dependency that is
enforced in the transactions in the database application
to which the transaction belongs. And Hypothesis 1 held
in all the transactions. Therefore, Hypothesis 1 holds
for all the 263 transactions in our sample.
Taking 0.001 () as the Type I error probability, if
the binomial test statistics z computed from the formula
discussed previously in this paper is greater than 3.09,
we reject the null hypothesis; otherwise, we accept the
null hypothesis. In our experiment, we have n = 263, X
= 263, and p = 0.96, so our z score is 3.31. Note that our
sample size is greater than max(10/0.96, 10/(1-0.96)).
Thus, we reject the null hypothesis, and the test gives
evidence that Hypothesis 1 holds for equal or more than
96% of the cases at the 0.1 % level of significance.
Because only one hypothesis is involved in the approach
for the inference of functional dependencies, the test
gives evidence that the approach holds for equal or more
than 96% of the cases at the 0.1 % level of significance.
FUTURE TRENDS
In general, many designs are very difficult (if not impos-
sible) to formally prove from program characteristics
(Chandra, Godefroid, & Palm, 2002; Clarke, Grumberg,
& Peled,1999; Deng & Kothari, 2002). Therefore, the
use of empirical-based knowledge is important for the
automated recognition of designs from program source
Wong, 2001). We believe that the integration of empiri-
115
TEAM LinG
 Building Empirical-Based Knowledge for Design Recovery
cal-based properties into existing program analysis and
model-checking techniques will be a fruitful direction
in the future.
CONCLUSION
Empirical-based knowledge has been used in the recog-
nition of designs from source codes through automated
program analysis. It is a promising research direction.
This chapter introduces the approach for building em-
pirical-based knowledge, a vital part for such research
exploration. We have also applied it in the recognition
of functional dependencies enforced in database trans-
actions from the transactions. We believe that our ap-
proach will encourage more exploration of the discov-
ery and use of empirical-based knowledge in this area.
Recently, we have completed our work on the recovery
of posttransaction user-input error (PTUIE) handling
for database transaction. This approach appeared in IEEE
Transactions on Knowledge and Data Engineering
(Tan & Thein, 2004).
REFERENCES
Basili, V. R. (1996). The role of experimentation in
software engineering: Past, current, and future. The 18th
International Conference on Software Engineering (pp.
442-449), Germany.
Beizer, B. (1990). Software testing techniques. New York:
Van Nostrand Reinhold.
Chandra, S., Godefroid, P., & Palm, C. (2002). Software
model checking in practice: An industrial case study.
Proceedings of the International Conference on Soft-
ware Engineering (pp. 431-441), USA.
Clarke, E. M., Grumberg, O., & Peled, D. A. (1999). Model
checking. MIR Press.
Deng, Y. B., & Kothari, S. (2002). Recovering conceptual
roles of data in a program. Proceedings of the Interna-
tional Conference on Software Maintenance (pp. 342-
350), Canada.
Embury, S. M., & Shao, J. (2001). Assisting the compre-
hension of legacy transactions. Proceedings of the
Working Conference on Reverse Engineering (pp. 345-
354), Germany.
Ferrante, J., Ottenstein, K. J., & Warren, J. O. (1987). The
program dependence graph and its use in optimisation.
ACM Transaction on Programming Languages and Sys-
tems, 9(3), 319-349.
116
Gravetter, F. J., & Wallnau, L. B. (2000). Statistics for the
behavioral sciences. Belmont, CA: Wadsworth.
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones,
P. W., Hoaglin, D. C., Emam, K. E., & Rosenberg, J. (2002).
Preliminary guidelines for empirical research in software
engineering. IEEE Transactions on Software Engineer-
ing, 28(8), 721-734.
Kozaczynski, W., Ning, J., & Engberts, A. (1992). Program
concept recognition and transformation. IEEE Transac-
tions on Software Engineering, 18(12), 1065-1075.
Tan, H. B. K., Ling, T. W., & Goh, C. H. (2002). Exploring
into programs for the recovery of data dependencies
designed. IEEE Transactions on Knowledge and Data
Engineering, 14(4), 825-835.
Tan, H. B. K., & Thein, N. L. (in press). Recovery of PTUIE
handling from source codes through recognizing its prob-
able properties. IEEE Transactions on Knowledge and
Data Engineering, 16(10), 1217-1231.
Ullman, J. D. (1982). Principles of database systems (2nd
ed.). Rockville, MD: Computer Science Press.
Wong, K. (2001). Research challenges in reverse engi-
neering community. Proceedings of the International
Workshop on Program Comprehension (pp. 323-332),
Canada.
KEY TERMS
Control Flow Graph: An abstract data structure
used in compilers. It is an abstract representation of a
procedure or program, maintained internally by a com-
piler. Each node in the graph represents a basic block.
Directed edges are used to represent jumps in the con-
trol flow.
Design Recovery: Recreates design abstractions
from a combination of code, existing design documen-
tation (if available), personal experience, and general
knowledge about problem and application domains.
Functional Dependency: For any record r in a
record type, its sequence of values of the attributes in X
is referred to as the X-value of r. Let R be a record type,
and X and Y be sequences of attributes of R. We say that
the functional dependency, X Y of R, holds at time t,
if at time t, for any two R records r and s, the X-values
of r and s are identical, then the U-values of r and s are
also identical.
Hypothesis Testing: Hypothesis testing refers to
the process of using statistical analysis to determine if the
observed differences between two or more samples are
TEAM LinG
Building Empirical-Based Knowledge for Design Recovery
due to random chance (as stated in the null hypothesis)
or to true differences in the samples (as stated in the
alternate hypothesis).
Model Checking: A method for formally verifying
finite-state concurrent systems. Specifications about
the system are expressed as temporal logic formulas,
and efficient symbolic algorithms are used to traverse the
model defined by the system and check whether the
specification holds.
Program Analysis: Offers static compile-time tech-
niques for predicting safe and computable approximations
to the set of values or behaviors arising dynamically at run-
time when executing a program on a computer. B
PTUIE: Posttransaction user-input error. An error
made by users in an input to a transaction execution and
discovered only after completion of the execution.
Transaction: An atomic set of processing steps in a
database application such that all the steps are per-
formed either fully or not at all.
117
TEAM LinG
118
Business Processes
David Sundaram
The University of Auckland, New Zealand
Victor Portougal
The University of Auckland, New Zealand
INTRODUCTION
The concept of a business process is central to many
areas of business systems, specifically to business sys-
tems based on modern information technology. In the
new era of computer-based business management the
design of business process has eclipsed the previous
functional design. Lindsay, Downs, and Lunn (2003)
suggest that business process may be divided into mate-
rial, information and business parts. Further search for
efficiency and cost reduction will be predominantly
through office automation. The information part, in-
cluding data warehousing, data mining and increasing
informatisation of the office processes will play a key
role. Data warehousing and data mining, in and of them-
selves, are business processes that are aimed at increas-
ing the intelligent density (Dhar & Stein, 1997) of the
data. But more important is the significant roles they play
within the context of larger business and decision pro-
cesses. Apart from these the creation and maintenance of
a data warehouse itself comprises a set of business
processes (see Scheer, 2000). Hence a proper under-
standing of business processes is essential to better
understand data warehousing as well as data mining.
BACKGROUND
Companies that make products or provide services have
several functional areas of operations. Each functional
area comprises a variety of business functions or busi-
ness activities. For example, functional area of financial
accounting includes the business functions of financials,
controlling, asset management, and so on. Human re-
sources functional area includes the business functions
of payroll, benefit administration, workforce planning,
and application data administration. Historically, orga-
nizational structures have separated functional areas,
and business education has been similarly organized, so
each functional area was taught as a separate course. In
materials requirement planning (MRP) systems, prede-
cessors of enterprise resource planning (ERP) systems,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
all functional areas were presented as subsystems sup-
ported by a separate functional areas information
system. However, in a real business environment func-
tional areas are interdependent, each requiring data
from the others. This fostered the development of the
concept of a business process as a multi-functional set
of activities designed to produce a specified output.
This concept has a customer focus. Suppose, a de-
fective product was delivered to a customer, it is the
business function of customer service to accept the
defective item. The actual repair and redelivery of the
item, however, is a business process that involves sev-
eral functional areas and functions within those areas.
The customer is not concerned about how the product
was made, or how its components were purchased, or
how it was repaired, or the route the delivery truck took
to get to her house. The customer wants the satisfaction
of having a working product at a reasonable price. Thus,
the customer is looking across the companys func-
tional areas in her process. Business managers are now
trying to view their business operations from the per-
spective of a satisfied customer. Thinking in terms of
business processes helps managers to look at their
organization from the customers perspective. ERP pro-
grams help to manage company wide business pro-
cesses, using a common database and shared manage-
ment reporting tools. ERP software supports the effi-
cient operation of business processes by integrating
business activities, including sales, marketing, manu-
facturing, accounting, and staffing.
MAIN THRUST
In the following sections we first look at some defini-
tions of business processes and follow this by some
representative classifications of business process. Af-
ter laying this foundation we look at business processes
that are specific to data warehousing and data mining.
The section culminates by looking at a business process
modelling methodology (namely ARIS) and briefly dis-
cussing the dark side of business process (namely mal-
processes).
TEAM LinG
Business Processes
Definitions of Business Processes
Many definitions have been put forward to define business
processes: some broad and some narrow. The broad ones
help us understand the range and scope of business pro-
cesses but the narrow ones are also valuable in that they are
actionable/pragmatic definitions that help us in defining,
modelling, and reengineering business processes.
Ould (1995) lists a few key features of Business
Processes; it contains purposeful activity, it is carried
out collaboratively by a group, it often crosses func-
tional boundaries, it is invariably driven by outside
agents or customers. Jacobson (1995) on the other hand
succinctly describes a business process as: The set of
internal activities performed to serve a customer. Bider
(2000) suggests that the business process re-engineer-
ing (BPR) community feel there is no great mystery
about what a process is - they follow the most general
definition of business processes proposed by Hammer
and Champy (1993) that a process is a set of partially
ordered activities intended to reach a goal.
Davenport (1993) defines process broadly as a
structured, measured set of activities designed to pro-
duce a specified output for a particular customer or
market and more specifically as a specific order of
work activities across time and place, with a beginning,
an end, and clearly identified inputs and outputs: a
structure for action.
While these definitions are useful they are not ad-
equate. Sharp and McDermott (2001) provide an excel-
lent working definition of a business process:
A business process is a collection of interrelated work
tasks, initiated in response to an event that achieves
a specific result for the customer of the process.
It is worth exploring each of the phrases within this
definition.
achieves a particular result
The result might be Goods and/or Services.
It should be possible to identify and count the
result eg. Fulfilment of Orders, Resolution of
Complaints, Raising of Purchase Orders, etc.
for the customer of the process
Every process has a customer. The customer maybe
internal (employee) or external (organisation).
A key requirement is that customer should be able
to give feedback on the process
initiated in response to a specific event
B
Every process is initiated by an event.
The event is a request for the result produced by
the process.
work tasks
The business process is a collection of clearly
identifiable tasks executed by one or more actors
(person or organisation or machine or depart-
ment).
A task could potentially be divided up into more
and finer steps.
a collection of interrelated
Such steps and tasks are not necessarily sequential
but could have parallel flows connected with com-
plex logic.
The steps are interconnected through their dealing
with or processing one (or more) common work
item(s) or business object(s)
Due to the importance of the business process
concept to the developing of the computerized enter-
prise management the work on refining the definition is
going on. Lindsay, Downs, and Lunn (2003) argue that
the definitions of business process given in much of the
literature on Business Process Management (BPM) are
limited in depth and their related models of business
processes are too constrained. Because they are too
limited to express the true nature of business processes,
they need to be further developed and adapted to todays
challenging environment.
Business Process Classification
Over the years many classifications of processes have
been suggested. The American Productivity & Quality
Center (Process Classification Framework, 1996) dis-
tinguishes two types of processes 1) operating pro-
cesses and 2) management and support processes.
Operating processes include processes such as:
Understanding Markets and Customers
Development of Vision and Strategy
Design of Product and Services
Marketing and Selling of Products and Services
Production and Delivery of Products and Services
Invoicing and Servicing of Customers
119
TEAM LinG

In contrast management and support processes in-
clude processes such as:
Development and Management of Human Resources
Management of Information
Management of Financial and Physical Resources
Management of External Relationships
Management of Improvement and Change
Execution of Environmental Management Program
Most organisations would be able to fit their pro-
cesses within this broad classification. However the
Gartner group (Genovese, Bond, Zrimsek, & Frey, 2001)
has come up with a slightly different set of processes
that has a lifecycle orientation:
Prospect to Cash and Care
Requisition to Payment
Planning and Execution
Plan to Performance
Design to Retirement
Human Capital Management
Each and every one of these processes could be
potentially supported by data warehousing and data min-
ing processes and technologies.
Data Warehousing Business Processes
The information movement and use come under the
category of a business process when it involves human-
computer interaction. Data warehousing includes a set of
business processes, classified above as management of
information. The set includes:
Data input
Data export
Special query and reporting (regular reporting is a
fully computer process)
Security and safety procedures
The bulk of the processes are of course in the data
input category. It is here is that many errors originate
that can disrupt the smooth operations of the manage-
ment system. Strict formalisation of the activities and
their sequence brings the major benefits. However, the
other categories are important as well, and their formal
description and enforcement help to avoid mal-func-
tioning of the system.
Data Mining Business Processes
Data mining is a key business process that enables
organisations to identify valid, novel, useful, and under-
120
Business Processes
standable patterns in data. Some of the key steps of the
data mining business process involve:
Understanding the business
Preparation of the data this would usually in-
volve data selection, pre-processing, and poten-
tially transformation of the data into a form that
is more amenable to modelling and understanding
Modelling using data mining techniques
Interpretation and evaluation of the model and
results which will hopefully help us to better
understand the business
Deploy the solution
Obviously the above steps are iterative at all stages
of the process and could feed backward. Tools such as
Clementine from SPSS support all the above steps.
Business Processes Modelling
The business processes frequently have to be recorded.
According to Scheer (2000) the components of a pro-
cess description are events, statuses, users, organiza-
tional units, and information technology resources.
This multitude of elements would severely complicate
the business process model. In order to reduce this
complexity, many researchers suggest that the model
is divided into individual views that can be handled
(largely) independent from one another. This is done
for example in ARIS (see Scheer, 2000). The relation-
ships between the components within a view are tightly
coupled while the individual views are rather loosely
linked.
ARIS includes four basic views, represented by:
Data view: statuses and events represented as
data and the other data structures of the process
Function view: the description of functions used
in business processes
Organization view: the users and the organiza-
tional units as well as their relationships and the
relevant structures
Control view: presenting the relationships be-
tween the other three views.
The integration of these three views through a sepa-
rate view is the main difference between ARIS and
other descriptive methodologies, like Object Oriented
Design (OOD) (Jacobson, 1995).
Breaking down the initial business process into
individual views reduces the complexity of its descrip-
tion. Figure 1 illustrates these four views.
TEAM LinG
Business Processes
Figure 1. ARIS views/house of business engineering
(Scheer, 2000)
Organisation View
Data Control Function
View View View
The control view can be depicted using event-driven
process chains (EPC). The four key components of the
EPC are events (statuses), functions (transformations),
organization, and information (data). Events (initial) trig-
ger functions that then result in events (final). The func-
tions are carried out by one or more organizational units
using appropriate data. A simplified depiction, of the
interconnection between the four basic components of
the EPC is illustrated in Figure 2. This was modeled using
the ARIS software (IDS Scheer, 2000).
One or more events can be connected to two or more
functions (and vice versa) using the three logical opera-
tors OR, AND, and XOR (exclusive OR). These basic
elements and operators enable us to model even the
most complex process.
The SAP Reference Model (Curran, Keller, & Ladd,
1998) is also based on the four ARIS views, it employs
several other views, which are not so essential, but
nevertheless helpful in depicting, storing and retrieving
business processes.
For example, information technology resources can
constitute a separate descriptive object, the resource
Figure 2. Basic components of the EPC
Initial Event
Transforming Organizational
Data
Function unit
Final Event
view. This view, however, is significant for the subject-
related view of business processes only when it gives an B
opportunity for describing in full the other components
that are more directly linked toward the business.
Mal-Processes
Most definitions of business processes suggest that
they have a goal of accomplishing a given task for the
customer of the process. This positive orientation has
been reflected in business process modelling method-
ologies, tools, techniques and even in the reference
models and implementation approaches. They do not
explicitly consider the negative business process sce-
narios that could result in the accomplishment of unde-
sirable outcomes.
Consider a production or service organisation that
implements an ERP system. Let us assume the design
stage is typical and business process oriented. We use
requirements elicitation techniques to define the as-
is business process and engineer to-be processes
(reference) using as much as possible, fragments of the
reference model of a corresponding ERP system.
As a result of the design, a set of business processes
will be created, defining a certain workflow for manag-
ers (employees) of the company. These processes will
consist of functions that should be executed in a certain
order defined by events. The employee responsible for
execution of a business process or a part of it will be
called its user. Every business process starts from a
pre-defined set of events, and performs a pre-defined
set of functions which involve either data input or data
modification (when a definite data structure is taken
from the data-base, modified and put back, or trans-
ferred to another destination).
When executing a process, a user may intentionally
put wrong data into the database, or modify the data in the
wrong way: e.g. execute an incomplete business process,
or execute a wrong branch of the business process, or
even create an undesirable branch of a business process.
Such processes are called mal-processes.
Hence mal-processes are behaviour to be avoided
by the system. It is a sequence of actions that a system
can perform, interacting with a legal user of the system,
resulting in harm for the organization or stakeholder if
the sequence is allowed to complete. The focus is on the
actions, which may be done intentionally or through
neglect. Similar effects produced accidentally are a
subject of interest for data safety. Just as we have best
business practices, mal-processes illustrate the wrong
business practices. So, it would be natural, by analogy
with the reference model (that represents a repository
of business models reflecting best business practices),
to create a repository of business models reflecting
121
TEAM LinG

wrong business practices. Thus a designer could avoid
wrong design solutions.
A repository of such mal-processes would enable
organisations to avoid typical mistakes in enterprise
system design. Such a repository can be a valuable asset
in education of ERP designers and it can be useful in
general management education as well. Another appli-
cation of this repository might be troubleshooting. For
example, sources of some nasty errors in the sales and
distribution system might be found in the mal-pro-
cesses of data entry in finished goods handling.
FUTURE TRENDS
There are three key trends that characterise business
processes: digitisation (automation), integration (intra
and inter organisational), and lifecycle management
(Kalakota & Robinson, 2003). Digitisation involves the
attempts by many organisations to completely automate
as many of their processes as possible. Another equally
important initiative is the seamless integration and co-
ordination of processes within and without the
organisation: backward to the supplier, forward to the
customers, and vertically of operational, tactical, and
strategic business processes. The management of both
these initiatives/trends depends to a large extent on the
proper management of processes throughout their
lifecycle: from process identification, process model-
ling, process analysis, process improvement, process
implementation, process execution, to process monitor-
ing/controlling (Rosemann, 2001). Implementing such a
lifecycle orientation enables organizations to move in
benign cycles of improvement and sense, respond, and
adapt to the changing environment (internal and external).
All these trends require not only the use of Enterprise
Systems as a foundation but also data warehousing and
data mining solutions. These trends will continue to be
major drivers of the enterprise of the future.
CONCLUSION
In this modern landscape, business processes and tech-
niques and tools for data warehousing and data mining
are intricately linked together. The impacts are not just
one way. Concepts from business processes could be
and are used to make the data mining and data warehous-
ing processes an integral part of organizational pro-
cesses. Data warehousing and data mining processes are
a regular part of organizational business processes,
enabling the conversion of operational information into
tactical and strategic level information. An example of
this is the Cross Industry Standard Process (CRISP) for
122
Business Processes
Data Mining (CRISP-DM, 2004) that provides a lifecycle
process oriented approach to the mining of data in
organizations. Apart from this data warehousing and data
mining techniques are a key element in various steps of
the process lifecycle alluded to in the trends. Data
warehousing and data mining techniques help us not only
in process identification and analysis (identifying and
analyzing candidates for improvement) but also in ex-
ecution and monitoring and control of processes. Data
warehousing technologies enable us in collecting, ag-
gregating, slicing and dicing of process information
while data mining technologies enable us in looking for
patterns in process information enabling us to monitor
and control the organizational process more efficiently.
Thus there is a symbiotic mutually enhancing relation-
ship both at the conceptual level as well as at the tech-
nology level between business processes and data ware-
housing and data mining.
REFERENCES
APQC (1996). Process classification framework (pp. 1-6).
APQCs International Benchmark Clearinghouse & Arthur
Andersen & Co.
Bider, I., Johannesson, P., & Perjons, E. (2002). Goal-
oriented patterns for business processes. Workshop on
Goal-Oriented Business Process Modelling, London.
CRISP-DM (2004). Retrieved from http://www.crisp-dm.org
Curran, T., Keller, G., & Ladd, A. (1998). SAP R/3 business
blueprint: Understanding business process reference
model. Upper Saddle River, NJ: Prentice Hall.
Davenport, T.H. (1993). Process innovation. Boston, MA:
Harvard Business School Press.
Davis, R., (2001) Business process modelling with
ARIS: A practical guide. UK: Springer-Verlag.
Genovese, Y., Bond, B., Zrimsek, B., & Frey, N. (2001).
The transition to ERP II: Meeting the challenges.
Gartner Group.
Hammer, M., & Champy, J. (1993). Re-engineering the
Corporation: A manifesto for business revolution. New
York: Harper Business.
Jacobson. (1995). The object advantage. Addison-
Wesley.
Kalakota, R., & Robinson, M. (2003). Services blue-
print: Roadmap for execution. Boston: Addison-
Wesley.
TEAM LinG
Business Processes
Lindsay, D., Downs, K., & Lunn. (2003). Business pro-
cessesattempts to find a definition. Information and
Software Technology, 45, 1015-1019.
Ould, A. M. (1995). Business processes: Modelling and
analysis for reengineering. Wiley.
Rosemann, M. (2001, March). Business process lifecycle
management. Queensland University of Technology.
Scheer, (2000). ARIS methods. IDS.
Scheer, A.-W., & Habermann, F. (2000). Making ERP a
success. Communications of the Association of Com-
puting Machinery, 43(4) 57-61.
Sharp, A., & McDermott, P. (2001). Just what are pro-
cesses anyway? Workflow modeling: tools for process
improvement and application development (pp. 53-69).
KEY TERMS
ARIS: Architecture of Integrated Information Sys-
tems, a modeling and design tool for business pro-
cesses.
as-is Business Process: Current business pro-
cess.
Business Process: A business process is a collection
of interrelated work tasks, initiated in response to an B
event that achieves a specific result for the customer of
the process.
Digitisation: Measures that automate processes.
ERP: Enterprise resource planning system, a software
system for enterprise management. It is also referred to as
Enterprise Systems (ES).
Functional Areas: Companies that make products
to sell have several functional areas of operations.
Each functional area comprises a variety of business
functions or business activities.
Integration of Processes: The coordination and
integration of processes seamlessly within and without
the organization.
Mal-Processes: A sequence of actions that a sys-
tem can perform, interacting with a legal user of the
system, resulting in harm for the organization or stake-
holder.
Process Lifecycle Management: Activities under-
taken for the proper management of processes such as
identification, analysis, improvement, implementation,
execution, and monitoring.
to-be Business Process: Re-engineered busi-
ness process.
123
TEAM LinG
124
Case-Based Recommender Systems
Fabiana Lorenzi
Universidade Luterana do Brasil, Brazil
Francesco Ricci
eCommerce and Tourism Research Laboratory, ITC-irst, Italy
INTRODUCTION
Recommender systems are being used in e-commerce
web sites to help the customers in selecting products
more suitable to their needs. The growth of Internet and
the business to consumer e-Commerce has brought the
need for such a new technology (Schafer, Konstan, &
Riedl., 2001).
BACKGROUND
In the past years, a number of research projects have
focused on recommender systems. These systems imple-
ment various learning strategies to collect and induce
user preferences over time and automatically suggest
products that fit the learned user model.
The most popular recommendation methodology is
collaborative filtering (Resnick, Iacovou, Suchak,
Bergstrom, & Riedl, 1994) that aggregates data about
customers preferences (ratings) to recommend new
products to the customers. Content-based filtering
(Burke, 2000) is another approach that builds a model of
user interests, one for each user, by analyzing the spe-
cific customer behavior. In collaborative filtering the
recommendation depends on the previous customers
information, and a large number of previous user/sys-
tem interactions are required to build reliable recom-
mendations. In content-based systems only the data of
the current user are exploited and it requires either
explicit information about user interest, or a record of
implicit feedback to build a model of user interests.
Content-based systems are usually implemented as clas-
sifier systems based on machine learning research
(Witten & Frank, 2000). In general, both approaches do
not exploit specific knowledge of the domain. For in-
stance, if the domain is computer recommendation, the
two above approaches, in building the recommendation
for a specific customer, will not exploit knowledge
about how a computer works and what is the function of
a computer component.
Conversely, in a third approach called knowledge-
based, specific domain knowledge is used to reason
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
about what products fit the customers preferences (Burke,
2000). The most important advantage is that knowledge
can be expressed as a detailed user model, a model of the
selection process or a description of the items that will be
suggested. Knowledge-based recommenders can exploit
the knowledge contained in case or encoded in a similarity
metric. Case-Based Reasoning (CBR) is one of the meth-
odologies used in the knowledge-based approach. CBR is
a problem solving methodology that faces a new problem
by first retrieving a past, already solved similar case, and
then reusing that case for solving the current problem
(Aaamodt & Plaza, 1994). In a CBR recommender system
(CBR-RS) a set of suggested products is retrieved from
the case base by searching for cases similar to a case
described by the user (Burke, 2000). In the simplest
application of CBR to recommendation problem solving,
the user is supposed to look for some product to pur-
chase. He/she inputs some requirements about the prod-
uct and the system searches in the case base for similar
products (by means of a similarity metric) that match the
user requirements. A set of cases is retrieved from the case
base and these cases can be recommender to the user. If
the user is not satisfied with the recommendation he/she
can modify the requirements, i.e. build another query, and
a new cycle of the recommendation process is started.
In a CBR-RS the effectiveness of the recommendation
is based on: the ability to match user preferences with
product description; the tools used to explain the match
and to enforce the validity of the suggestion; the function
provided for navigating the information space. CBR can
support the recommendation process in a number of
ways. In the simplest approach the CBR retrieval is called
taking in input a partial case defined by a set of user
preferences (attribute-value pairs) and a set of products
matching these preferences are returned to the user.
MAIN THRUST
CBR systems implement a problem solving cycle very
similar to the recommendation process. It starts with a
new problem, retrieves similar cases from the case base
and shows to the user an old solution or adapts it to better
TEAM LinG
Case-Based Recommender Systems
Figure 1. CBR-RS framework (Lorenzi & Ricci, 2004)
solve the new problem and finishes retaining the new case
in the case base. Considering the classic CBR cycle (see
Aamodt & Plaza, 1994) we specialized this general frame-
work to the specific tasks of product recommendation. In
Figure 1 the boxes, corresponding to the classical CBR
steps (retrieve, reuse, revise, review, and retain), contain
references to systems or functionalities (acronyms) that
will be described in the next sections.
We now provide a general description of the frame-
work by making some references to systems that will be
better described in the rest of the paper. The first user-
system interaction in the recommendation cycle occurs
in the input stage. According to Bergmann, Richter,
Schmitt, Stahl, and Vollrath (2001), there are different
strategies to interact with the user, depending on the level
of customer assistance offered during the input. The most
popular strategy is the dialog-based, where the system
offers guidance to the user by asking questions and
presenting products alternatives, to help the user to
decide. Several CBR recommender systems ask the user
for input requirements to have an idea of what the user
is looking for. In the First Case system (McSherry,
2003a), for instance, the user provides the features of a
personal computer that he/she is looking for, such as,
type, price, processor or speed. Expertclerk (Shimazu,
2002) asks the user to answer some questions instead of
provide requirements. And with the set of the answered
questions the system creates the query.
In CBR-RSs, the knowledge is stored in the case
base. A case is a piece of knowledge related to a particular
context and representing an experience that teaches an
essential lesson to reach the goal of the problem-solving
activity. Case modeling deals with the problem of deter-
mining which information should be represented and
which formalism of representation would be suitable. In
CBR-RSs a case should represent a real experience of
solving a user recommendation problem. In a CBR-RS, our
analysis has identified a general internal structure of the
case base: CB = [X U S E].
This means that a case c = (x, u, s, e) CB, generally
consists of four (optional) sub-elements x, u, s, e, which C
are elements of the spaces X, U, S, E respectively. Each
CBR-RS adopts a particular model for the spaces X, U,
S, E. These spaces could be empty, vector, set of docu-
ment (textual), labeled graphs, etc.
Content model (X): the content model describes
the attributes of the product.
User profile (U): the user profile models personal
user information, such as, name, address, and age
or also past information about the user, such as her
preferred products.
Session model (S): the session model is intro-
duced to collect information about the recom-
mendation session (problem solving loop). In
DieToRecs, for instance, a case includes a tree-
based model of the user interaction with the sys-
tem and it is built incrementally during the recom-
mendation session.
Evaluation model (E): the evaluation model de-
scribe the outcome of the recommendation, i.e., if
the suggestion was appropriate or not. This could
be a user a-posteriori evaluation, or, as in
(Montaner, Lopez, & la Rosa, 2002), the outcome
of an evaluation algorithm that guesses the good-
ness of the recommendation (exploiting the case
base of previous recommendations).
Actually, in CBR-RSs there is a large variability in
what a case really models and therefore what compo-
nents are really implemented. There are systems that
use only the content model, i.e., they consider a case as
a product, and other systems that focus on the perspec-
tive of cases as recommendation sessions.
The first step of the recommendation cycle is the
retrieval phase. This is typically the main phase of the
CBR cycle and the majority of CBR-RSs can be de-
scribed as sophisticated retrieval engines. For example,
in the Compromise-Driven Retrieval (McSherry, 2003b)
the system retrieves similar cases from the case base
but also groups the cases, putting together those offer-
ing to the user the same compromise, and presents to the
user just a representative case for each group.
After the retrieval, the reuse stage decides if the case
solution can be reused in the current problem. In the
simplest CBR-RSs, the system reuses the retrieved
cases showing them to the user. In more advanced solu-
tions, such as (Montaner, Lopez, & la Rosa, 2002) or
(Ricci et al., 2003), the retrieved cases are not recom-
mended but used to rank candidate products identified
with other approaches (e.g. Ricci et al., 2003) with an
interactive query management component.
125
TEAM LinG

In the next stage the reused case component is adapted
to better fit in the new problem. Mostly, the adaptation in
CBR-RSs is implemented by allowing the user to customize
the retrieved set of products. This can also be implemented
as a query refinement task. For example, in Comparison-
based Retrieval (McGinty & Smyth, 2003) the system asks
user feedbacks (positive or negative) about the retrieved
product and with this information it updates the user
query.
The last step of the CBR recommendation cycle is the
retain phase (or learning), where the new case is retained
in the case base. In DieToRecs (Fesenmaier, Ricci,
Schaumlechner, Wober, & Zanella, 2003), for instance,
all the user/system recommendation sessions are stored
as cases in the case base.
The next subsections describe very shortly some repre-
sentative CBR-RSs, focusing on their peculiar characteris-
tic (see Lorenzi & Ricci, 2004) for the complete report.
Interest Confidence Value ICV
Montaner, Lopez, and la Rosa (2002) assume that the
users interest in a new product is similar to the users
interest in similar past products. This means that when a
new product comes up, the recommender system pre-
dicts the users interest in it based on interest attributes
of similar experiences. A case is modeled by objective
attributes describing the product (content model) and
subjective attributes describing implicit or explicit in-
terests of the user in this product (evaluation model),
i.e., c X E. In the evaluation model, the authors intro-
duced the drift attribute, which models a decaying im-
portance of the case as time goes and the case is not used.
The system can recommend in two different ways:
prompted or proactive. In prompted mode, the user pro-
vides some preferences (weights in the similarity metric)
and the system retrieves similar cases. In the proactive
recommendation, the system does not have the user prefer-
ences, so it estimates the weights using past interactions.
In the reuse phase the system extracts the interest
values of retrieved cases and in the revise phase it calcu-
lates the interest confidence value of a restaurant to
decide if this should be recommender to the user or not.
The adaptation is done asking to the user the correct
evaluation of the product and after that a new case (the
product and the evaluation) is retained in the case base.
It is worth noting that in this approach the recommended
product is not retrieved from the case base, but the
retrieved cases are used to estimate the user interest in
this new product. This approach is similar to that used in
DieToRecs in the single item recommendation function.
126
Case-Based Recommender Systems
DieToRecs DTR
DieToRecs helps the user to plan a leisure travel
(Fesenmaier, Ricci, Schaumlechner, Wober, & Zanella
2003). We present here two different approaches (de-
cision styles) implemented in DieToRecs: the single
item recommendation (SIR), and the travel completion
(TC). A case represents a user interaction with the sys-
tem, and it is built incrementally during the recommenda-
tion session. A case comprises all the quoted models:
content, user profile, session and evaluation model.
SIR starts with the user providing some prefer-
ences. The system searches in the catalog for products
that (logically) match with these preferences and re-
turns a result set. This is not to be confused with the
retrieval set that contains a set of similar past recom-
mendation session. The products in the result set are
then ranked with a double similarity function (Ricci et
al., 2003) in the revise stage, after a set of relevant
recommendation sessions are retrieved.
In the TC function the cycle starts with the users
preferences too but the system retrieves from the case
base cases matching users preferences. Before rec-
ommending the retrieved cases to the user, the system
in the revise stage updates, or replaces, the travel
products contained in the case exploiting up-to-date
information taken from the catalogues. In the review
phase the system allows the user to reconfigure the
recommended travel plan. The system allows the user
to replace, add or remove items in the recommended
travel. When the user accepts the outcome (the final
version of the recommendation shown to the user), the
system retains this new case in the case base.
Compromise-Driven Retrieval CDR
CDR models a case only by the content component
(McSherry, 2003a). In CDR, if a given case c1 is more
similar to the target query than another case c2, and
differs from the target query in a subset of the at-
tributes in which c2 differs from the target query, then
c 1 is more acceptable than c2.
In the CDR retrieval algorithm the system sorts all
the cases in the case-base according to the similarity to
a given query. In a second stage, it groups together the
cases making the same compromise (do not match a
user preferred attribute value) and builds a reference
set with just one case for each compromise group. The
cases in the reference set are recommended to the user.
The user can also refine (review) the original query,
accepting one compromise, and adding some preference
on a different attribute (not that already specified). The
system will further decompose the set of cases corre-
TEAM LinG
Case-Based Recommender Systems

sponding to the selected compromise. The revise and Table 1. Comparison of the CBR-RSs
retain phases do not appear in this approach.
Approach
ICV
Retrieval
Similarity
Reuse
IC value
Revise
IC computation
Review
Feedback
Retain
Default
C
SIR Similarity Selective Rank User edit Default
ExpertClerk EC TC
OBR
Similarity
Similarity +
Default
Default
Logical query
None
User edit
Tweak
Default
None
Ordering
CDR Similarity + Default None Tweak None
Expertclerk is a tool for developing a virtual salesclerk Grouping
EC Similarity Default None Feedback None
system as a front-end of e-commerce websites (Shimazu,
2002). The system implements a question selection method
(decision tree with information gain). Using navigation- area have already delivered, how the existing CBR-RSs
by-asking, the system starts the recommendation session behave and which are the topics that could be better
asking questions to user. The questions are nodes in a exploit in future systems.
decision tree. A question node subdivides the set of
answer nodes and each one of these represents a different
answer to the question posed by the question node. The CONCLUSION
system concatenates all the answer nodes chosen by the
user and then constitutes the SQL retrieval condition In the previous sections we have briefly analyzed eight
expression. different CBR recommendations. Table 1 shows the
This query is applied to the case base to retrieve the main features of these approaches.
set of cases that best match the user query. Then, the Some observations are in order. The majority of the
system shows three samples products to the user and CBR-RSs stress the importance of the retrieval phase.
explains their characteristics (positive and negative). Some systems perform retrieval in two steps. First,
In the review phase, the system switches to the cases are retrieved by similarity, and then the cases are
navigation-by-proposing conversation mode and allows grouped or filtered. The use of pure similarity does not
the user to refine the query. After refinement, the sys- seem to be enough to retrieve a set of cases that satisfy
tem applies the new query to the case base and retrieves the user. This seems to be true especially in those
new cases. These cases are ranked and shown to the user. application domains that require a complex case struc-
The cycle continues until the user finds a preferred ture (e.g. travel plans) and therefore require the devel-
product. In this approach the revise and the retain phases opment of hybrid solutions for case retrieval.
are not implemented. The default reuse phase is used in the majority of the
CBR-RSs, i.e., all the retrieved cases are recommended
to the user. ICV and SIR have implemented the reuse
FUTURE TRENDS case in different way. In SIR, for instance, the system
can retrieve just part of the case. The same systems that
This paper presented a review of the literature on CBR implemented non-trivial reuse approaches, have also
recommender systems. We have found that it is often implemented both the revise phase, where the cases are
unclear how and why the proposed recommendation adapted, and the retain phase, where the new case (adapted
methodology can be defined as case-based and there- case) is stored.
fore we have introduced a general framework that can All the CBR-RSs analyzed implement the review phase,
illustrate similarities and differences of various ap- allowing the user to refine the query. Normally the sys-
proaches. Moreover, we have found that the classical tem expects some feedback from the user (positive or
CBR problem-solving loop is implemented only par- negative), new requirements or a product selection.
tially and sometime is not clear whether a CBR stage
(retrieve, reuse, revise, review, retain) is implemented
or not. For this reason, the proposed unifying frame- REFERENCES
work makes possible a coherent description of different
CBR-RSs. In addition an extensive usage of this frame- Aamodt, A., & Plaza, E. (1994). Case-based reasoning:
work can help describing in which sense a recommender Foundational issues, methodological variations, and
system exploits the classical CBR cycle, and can point system approaches. AI Communications, 7(1), 39-59.
out new interesting issues to be investigated in this area.
For instance, the possible ways to adapt retrieved cases Bergmann, R., Richter, M., Schmitt, S., Stahl, A., &
to improve the recommendation and how to learn these Vollrath, I. (2001). Utility-oriented matching: New re-
adapted cases. search direction for case-based reasoning. 9th German
We believe, that with such a common view it will be Workshop on Case-Based Reasoning, GWCBR01 (pp.
easier to understand what the research projects in the 14-16), Baden-Baden, Germany.

127

TEAM LinG

Burke, R. (2000). Knowledge-based recommender sys-
tems. Encyclopedia of Library and Information Science,
Vol. 69.
Fesenmaier, D, Ricci, F, Schaumlechner, E, Wober, K.,
& Zanella, C. (2003). DIETORECS: Travel advisory for
multiple decision styles. Information and Communi-
cation Technologies in Tourism, 232-241.
Lorenzi, F., & Ricci, F. (2004). A unifying framework
for case-base reasoning recommender systems. Tech-
nical Report, IRST.
McGinty, L., & Smyth, B. (2002). Comparison-based
recommendation. Advances in Case-Based Reason-
ing, 6th European Conference on Case Based Reason-
ing, ECCBR 2002 (pp. 575-589), Aberdeen, Scotland.
McGinty, L., & Smyth, B. (2003). The power of sugges-
tion. 18th International Joint Conference on Artificial
Intelligence, IJCAI-03 (pp. 276-290), Acapulco, Mexico.
McSherry, D. (2003a). Increasing dialogue efficiency in
case-based reasoning without loss of solution quality.
18 th International Joint Conference on Artificial Intelli-
gence, IJCAI-03 (pp. 121-126), Acapulco, Mexico.
McSherry, D. (2003b). Similarity and compromise. 5th In-
ternational Conference on Case-Based Reasoning,
ICCBR 2003 (pp. 291-305), Trondheim, Norway.
Montaner, M., Lopez, B., & la Rosa, J.D. (2002). Improv-
ing case representation and case base maintenance in
recommender systems. Advances in Case-Based Rea-
soning, 6th European Conference on Case Based Reason-
ing, ECCBR 2002 (pp. 234-248),Aberdeen, Scotland.
Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., &
Riedl, J. (1994). Grouplens: An open architecture for
collaborative filtering of Netnews. ACM Conference
on Computer-Supported Cooperative Work (pp. 175-
186).
Ricci, F., Venturini, A., Cavada, D., Mirzadeh, N., Blaas, D.,
& Nones, M. (2003). Product recommendation with inter-
128
Case-Based Recommender Systems
active query management and twofold similarity. 5th Inter-
national Conference on Case-Based Reasoning, ICCBR
2003 (pp. 479-493), Trondheim, Norway.
Schafer, J.B, Konstan, J. A., & Riedl, J. (2001). E-
commerce recommendation applications. Data Mining
and Knowledge Discovery, 5(1/2), 115-153.
Shimazu, H. (2002). Expertclerk: A conversational case-
based reasoning tool for developing salesclerk agents in
e-commerce webshops. Artificial Intelligence Review,
18, 223-244.
Witten, I. H., & Frank, E. (2000). Data mining. Morgan
Kaufmann Publisher.
KEY TERMS
Case-Based Reasoning: It is an Artificial Intelli-
gence approach that solves new problems using the
solutions of past cases.
Collaborative Filtering: Approach that collects
user ratings on currently proposed products to infer the
similarity between users.
Content-Based Filtering: Approach where the user
expresses needs and preferences on a set of attributes
and the system retrieves the items that match the de-
scription.
Conversational Systems: Systems that can com-
municate with users through a conversational paradigm
Machine Learning: The study of computer algo-
rithms that improve automatically through experience.
Recommender Systems: Systems that help the user to
choose products, taking into account his/her preferences.
Web Site Personalization: Web sites that are person-
alized to each user, knowing the user interests and needs.
TEAM LinG

Categorization Process and Data Mining
Maria Suzana Marc Amoretti
Federal University of Rio Grande do Sul (UFRGS), Brazil
INTRODUCTION
For some time, the fields of computer science and cogni-
tion have diverged. Researchers in these two areas know
ever less about each others work, and their important
discoveries have had diminishing influence on each other.
In many universities, researchers in these two areas are in
different laboratories and programs, and sometimes in
different buildings. One might conclude from this lack of
contact that computer science and semiotics functions,
such as perception, language, memory, representation,
and categorization, reflect our independent systems. But
for the last several decades, the divergence between
cognition and computer science tends to disappear. These
areas need to be studied together, and the cognitive
science approach can afford this interdisciplinary view.
This article refers to the possibility of circulation be-
tween the self-organization of the concepts and the rel-
evance of each conceptual property of the data-mining
process and especially discusses categorization in terms of
a prototypical theory, based on the notion of prototype and
basic level categories. Categorization is a basic means of
organizing the world around us and offers a simple way to
process the mass of stimuli that one perceives every day.
The ability to categorize appears early in infancy and has an
important role for the acquisition of concepts in a prototypi-
cal approach. Prototype structures have cognitive represen-
tations as representations of real-world categories.
The senses of the English words cat or table are
involved in a conceptual inclusion in which the extension
of the superordinated (animal/furniture) concept includes
the extension of the subordinated (Persian cat/dining
room table) concept, while the intention of the more
general concept is included by the intention of the more
specific concept. This study is included in the categori-
zation process. Categorization is a fundamental process
of mental representation used daily for any person or for
any science, and it is also a central problem in semiotics,
linguistics, and data mining.
Data mining also has been defined as a cognitive
strategy for searching automatically new information
from large datasets or selecting a document, which is
possible with computer science and semiotics tools. Data
mining is an analytic process to explore data in order to
find interesting pattern motifs and/or variables in the
great quantity of data; it depends mostly on the catego-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
129
C
rization process. The computational techniques from sta-
tistics and pattern recognition are used to do this data-
mining practice.
BACKGROUND
Semiotics is a theory of the signification (representations,
symbols, categories) and meaning extraction. It is a strongly
multi-disciplinary field of study, and mathematical tools
of semiotics include those used in pattern recognition.
Semiotics is also an inclusive methodology that incorpo-
rates all aspects of dealing with symbolic systems of
signs. Signification join all the concepts in an elementary
structure of signification. This structure is a related net
that allows the construction of a stock of formal defini-
tions such as semantic category. Hjelmeslev considers
the category as a paradigm, where elements can be intro-
duced only in some positions.
Text categorization process, also known as text clas-
sification or topic spotting, is the task of automatically
classifying a set of documents into categories from a
predefined set. This task has several applications, includ-
ing automated indexing of scientific articles according to
predefined thesauri of technical terms, filing patents into
patent directories, and selective dissemination of infor-
mation to information users. There are many new catego-
rization methods to realize the categorization task, includ-
ing, among others, (1) the language model based classi-
fication; the maximum entropy classification, which is a
probability distribution estimation technique used for a
variety of natural language tasks, such as language mod-
eling, part-of-speech tagging, and text segmentation (the
theory underlying maximum entropy is that without exter-
nal knowledge, one should prefer distributions that are
uniform); (2) the Nave Bayes classification; (3) the Near-
est Neighbor (the approach clusters words into groups
based on the distribution of class labels associated with
each word); (4) distributional clustering of words to
document classification; (5) the Latent Semantic Index-
ing (LSI), in which we are able to compress the feature
space more aggressively, while still maintaining high
document classification accuracy (this information re-
trieval method improves the users ability to find relevant
information, the text categorization method based on a
combination of distributional features with a Support
TEAM LinG

Vector Machine (SVM) classifier, and the feature se-
lection approach uses distributional clustering of words
via the recently introduced information bottleneck
method, which generates a more efficient representa-
tion of the documents); (6) the taxonomy method, based
on hierarchical text categorization that documents are
assigned to leaf-level categories of a category tree (the
taxonomy is a recently emerged subfield of the seman-
tic networks and conceptual maps). After the previous
work in hierarchical classification focused on docu-
ments category, the tree classify method internal cat-
egories with a top down level-based classification that
can classify concepts in the document.
The networks of semantic values thus created and
stabilized constitute the cultural-metaphorical worlds
which are discursively real for the speakers of particular
languages. The elements of these networks, though
ultimately rooted in the physical-biological realm can
and do operate independently of the latter, and form the
stuff of our everyday discourses (Manjali, 1997, p. 1).
The prototype has given way to a true revolution (the
Roschian revolution) regarding classic lexical semantics.
If we observe the conceptual map for chair, for instance,
we will realize that the choice of most representative chair
types; that is, our prototype of chair, supposes a double
adequacy: referential because the sign (concept of chair)
must integrate the features retained from the real or
imaginary world, and structural, because the sign must be
pertinent (ideological criterion) and distinctive concern-
ing the other neighbor concepts of chair. When I say that
this object is a chair, it is supposed that I have an idea of
the chair sign, forming the use of a lexical or visual image
competence coming from my referential experience, and
that my prototypical concept of chair is more adequate
than its neighbors bench or couch, because I perceive
that there is a back part and there are no arms. Then, it is
useless to try to explain the creation of a prototype inside
a language, because it is formed from context interactions.
The double origin of a prototype is bound, then, to
shared knowledge relation between the subjects and
their communities (Amoretti, 2003).
MAIN THRUST
Hypertext poses new challenges for a data-mining pro-
cess, especially for text categorization research, because
metadata extracted from Web sites provide rich informa-
tion for classifying hypertext documents, and it is a new
kind of problem to solve, to know how to appropriately
represent that information and automatically learn statis-
tical patterns for hypertext categorization. The use of
130
Categorization Process and Data Mining
technologies in the categorization process through the
making of conceptual maps, especially the possibility
of creating a collaborative map made by different users,
points out the cultural aspects of the concept represen-
tation in terms of existing coincidences as to the choice
of the prototypical element by the same cultural group.
Thus, the technologies of information, focused on the
study of individual maps, demand revisited discussions
on the popular perceptions concerning concepts used
daily (folk psychology). It aims to identify ideological
similarity and cognitive deviation, both based on the
prototypes and on the levels of categorization developed
in the maps, with an emphasis on the cultural and semiotic
aspects of the investigated groups.
It attempted to show how the semiotic and linguistic
analysis of the categorization process can help in the
identification of the ideological similarity and cognitive
deviations, favoring the involvement of subjects in the
map production, exploring and valuing the relation be-
tween the categorization process and the cultural experi-
ence of the subject in the world, both parts of the cogni-
tive process of conceptual map construction.
The concept maps, or the semantic nets, are space graphic
representations of the concepts and their relationships.
The concept maps represent, simultaneously, the
organization process of the knowledge, by the
relationships (links) and the final product, through the
concepts (nodes). This way, besides the relationship
between linguistic and visual factors is the interaction
among their objects and their codes (Amoretti, 2001, p.
49).
The building of a map involves collaboration when the
subjects/students/users share information, still without
modifying the data, and involves cooperation, when users
not only share their knowledge but also may interfere and
modify the information received from the other users, acting
in a asynchronized way to build a collective map. Both
cooperation and collaboration attest the autonomy of the
ongoing cognitive process, the direction given by the users
themselves when trying to adequate their knowledge.
When people do a conceptual map, they usually privi-
lege the level where the prototype is. The basic concept
map starts with a general concept at the top of the map and
then works its way down through a hierarchical structure
to more specific concepts. The empirical concept (Kant)
of cat and chair has been studied by users with map
software. They make an initial map at the beginning of the
semester and another about the same subject at the end
of the semester. I first discussed how cats and chairs
appear, what could be called the structure of cat and chair
appearance. Second, I discussed how cat and chair are
perceived and which attributes make a cat a cat and a chair
TEAM LinG
Categorization Process and Data Mining
a chair. Finally, I will consider cat and chair as an experi-
ential category, so the point of departure is our experience
in the world about cat and chair. The acquisition of the
concept cat and chair is mediated by concrete experiences.
Thus, the learner must possess relevant prior knowledge
and a mental scheme to acquire a prototypical concept.
The expertise changes the conceptual level organiza-
tion competences. In the first maps, the novice chair map
privileged the basic level, the most important exemplar of
a class, the chair prototype. This level has a high coher-
ence and distinctiveness. After thinking about this con-
cept, students (now chair experts) repeated the experiment
and carried out again the expert chair map with much more
details in the superordinate level, showing eight different
kinds of chairs: dining room chair, kitchen chair, garden
chair, and so forth. This level has high coherence and low
distinctiveness (Rosh, 2000). So, users learn by doing the
categorization process.
Language system arbitrarily cuts up the concepts into
discrete categories (Hjelmeslev, 1968), and all categories
have equal status. Human language is both natural and
cultural. According to the prototype theory, the role played
by non-linguistic factors like perception and environment
is demonstrated throughout the concept as the prototype
from the subjects of each community.
A concept is a sort of scheme. An effective way of
representing a concept is to retain only its most important
properties. This group of most important properties of a
concept is called prototype. The idea of prototype makes
it possible for the subject to have a mental construction,
identifying the typical features of several categories, and,
when the subject finds a new object, he or she may compare
it to the prototype in his or her memory. Thus, the proto-
type of chair, for instance, allows new objects to be
identified and labeled as chairs. In individual conceptual
maps creation, one may confirm the presence of variables
for the same concept.
The notion of prototype originated in the 1970s, greatly
due to Eleanor Roschs (2000) psychological research on
the organization of conceptual categories. Its revolution-
ary character marked a new era for the discussions on
categorization and brought existing theories, such as the
classical view of the prototype question. On the basis of
Roschs results, it is argued that members of the so-called
Aristotelian (or classical) categories share all the same
properties and showed that categories are structured in an
entirely different way; members that constitute them are
assigned in terms of gradual participation, and the cat-
egorical attribution is made by human beings according to
the more or less centrality/marginality of collocation within
the categorical structure. Elements recognized as central
members of the category represent the prototype. For
instance, a chair is a very good example of the category
furniture, while a television is a less typical example of the
same category. A chair is a more central member than a
television, which, in turn, is a rather marginal member. C
Rosch (2000) claims that prototypes can only constrain,
but do not determine, models of representations.
The main thrust of my argument is that it is very
important to data mining to know, besides the cognitive
categorization process, what is the prototypical concept
in a dataset. This basic level of concept organization
reflects the social representation in a better way than the
other levels (i.e., superordinate and subordinate levels).
The prototype knowledge affords a variety of culture
representations. The conceptual mapping system con-
tains prototype data on the hierarchical way of concept
levels. Using different software (three measuring lev-
elssuperordinate, basic, and subordinate), I suggest
the construction of different maps for each concept to
analyze the categorization cognitive process with maps
and to show how the categorization performance of
individual and collective or organizational team overtime
is important in a data-mining work.
Categorization is a part of Jakobsons (2000) commu-
nication model (also appropriated from information
theory) with cultural aspects (context). This principle
allows to locate on a definite gradient objects and rela-
tions that are observed, based on similarity and contigu-
ity associations (frame/script semantic) (Schank, 1999)
and based on hierarchically relations (prototypic seman-
tic) (Kleiber, 1990; Rosch, 2000), in terms of perceived
family resemblance among category members.
FUTURE TRENDS
Data-mining language technology systems typically have
focused on the factual aspect of content analysis. How-
ever, there are other categorization aspects, including
pragmatics, point of view, and style, which must receive
more attention like types and models of subjective clas-
sification information and categorization characteristics
such as centrality, polarity, intensity, and different lev-
els of granularity (i.e., expression, clause, sentence,
discourse segment, document, hypertext).
It is also important to define properties heritage
among different category levels, viewed throughout hi-
erarchical relations as one that allowed to virtually add
certain pairs of value (attributes from a unit to another).
We should also think of concepts managing that, in a
given category, are considered as an exception. It would
be necessary to allow the heritage blockage of certain
attributes. I will be opening new perspectives to the data-
mining research of categorization and prototype study,
which shows the ideological similarity perception medi-
ated by collaborative conceptual maps.
131
TEAM LinG

Much is still unknowable about the future of data
mining in higher education and in the business intelli-
gence process. The categorization process is a factor that
will affect this future and can be identified with the crucial
role played by the prototypes. Linguistics have not yet
paid this principle due attention. However, some conse-
quences should already necessarily follow from its proto-
type recognition. The extremely powerful explanation of
prototype categorization constitutes the most salient
feature in data mining. So, a very important application in
the data-mining methodology is the results of the proto-
type categorization research like a form of retrieval of
unexpected information.
CONCLUSION
Categorization explains aspects of peoples cultural logic.
In this chapter, statistical pattern recognition approaches
are used to classify concepts which are present in a given
dataset and expressed by conceptual maps. Prototypes
whose basis is the concepts representation from the
heritage par dfaut that allows a great economy in the
acquisition and managing of the information. The objec-
tive of this study was to investigate the potential of
founding the categories in the text with conceptual maps,
to use these maps as data-mining tools. Based on user
cognitive characteristics of knowledge organization and
based on the prevalence of the basic level (the prototypi-
cal level), a case is of the necessity of the categorization
cognitive process like as a step of data mining.
REFERENCES
Amoretti, M.S.M. (2001). Prottipos e esteretipos:
aprendizagem de conceitos: Mapas conceituais: Uma
experincia em Educao Distncia. Proceedings of the categorization. ACM Computing Surveys (CSUR).
Revista Informtica na Educao. Teoria e Prtica,
Porto Alegre, Brazil.
Amoretti, M.S.M. (2003). Conceptual maps: A metacognitive
strategy to learn concepts. Proceedings of the 25th Annual
Meeting of the Cognitive Science Society, Boston, Mas-
sachusetts.
Amoretti, M.S.M. (2004a). Categorization process and
conceptual maps. Proceedings of the First International
Conference on Concept Mapping, Pamplona, Spain.
Amoretti, M.S.M. (2004b). Collaborative learning con-
cepts in distance learning: Conceptual map: Analysis of
prototypes and categorization levels. Proceedings of the
CCM Digital Government Symposium, Tuscaloosa, Ala-
bama.
132
Categorization Process and Data Mining
Andler, D. (1987). Introduction aux sciences cognitives.
Paris: Gallimard.
Cordier, F. (1989). Les notions de typicalit et niveau
dabstracion: Analyse des proprits des representa-
tions [thse de doctorat ddtat]. Paris: Sud University.
das, J. (1966). Smantique structurale. Recherche de
mthode. Paris: PUF.
Frawley, W., Piatetsky-Shapiro, G., & Matheus, C. (1992).
Knowledge discovery in databases: An overwiev. AI
Magazine, 13(2), 57-70.
Greimas, A.J. (1966). Smantique structurale. Recherche
de mthode. Paris: PUF.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of
data mining. Boston: MIT Press.
Hjelmeslev, L. (1968). Prolgomnes une thorie du
langage. Paris: ditions Minuit.
Jakobson, R. (2000). Linguistics and poetics. Londres:
Lodge and Wood.
Kleiber, G. (1990). La smantique du prototype: Catgories
et sens lexical. Paris: PUF.
Manjali, F.D. (1997). Dynamical models in semiotics/
semantics. Retrieved from http://www.chass.utoronto.ca/
epc/srb/cyber/manout.html
Minksy, M.L. (1977). A framework for representing knowl-
edge. In P.H. Winston (Ed.), The psychology of computer
vision (pp. 211-277). New York: McGraw-Hill.
Rosch, E. et al. (2000). The embodied mind. London: MIT.
Schank, R. (1999). Dynamic memory revisited. Cambridge,
MA: Cambridge University Press.
Sebastiani, F. (2002). Machine learning in automated text
Yiming, Y. (1999). An evaluation of statistical approaches
to text categorization. Information Retrieval.
KEY TERMS
Categorization: A cognitive process based on simi-
larity of mental schemes and concepts that subjects
establish conditions that are both necessary and suffi-
cient (properties) to capture meaning and/or the hierarchy
inclusion (as part of a set) by family ressemblences shared
by their members. Every category has a prototypical
internal structure, depending on the context.
TEAM LinG
Categorization Process and Data Mining
Concept: A sort of scheme produced by repeated
experiences. Concepts are essentially each little idea that
we have in our heads about anything. This includes not
only everything, but every attribute of everything.
Conceptual Maps: Semiotic representation (linguistic
and visual) of the concepts (nodes) and their relation-
ships (links); represent the organization process of the
knowledge When people do a conceptual map, they
usually privilege the level where the prototype is. They
prefer to categorize at an intermediate level; this basic
level is the first level learned, the most common level
named, and the most general level where visual shape and
attributes are maintained.
Prototype: An effective way of representing a concept
is to retain only its most important properties or the most C
typical element of a category, which serves as a cognitive
reference point with respect to a cultural community. This
group of most important properties or most typical ele-
ments of a concept is called prototype. The idea of
prototype makes possible that the subject has a mental
construction, identifying the typical features of several
categories. Prototype is defined as the object that is a
categorys best model.
133
TEAM LinG
134
Center-Based Clustering and Regression
Clustering
Bin Zhang
Hewlett-Packard Research Laboratories, USA
INTRODUCTION
Center-based clustering algorithms are generalized to
more complex model-based, especially regression-
model-based, clustering algorithms. This article briefly
reviews three center-based clustering algorithmsK-
Means, EM, and K-Harmonic Meansand their gener-
alizations to regression clustering algorithms. More
details can be found in the referenced publications.
BACKGROUND
Center-based clustering is a family of techniques with
applications in data mining, statistical data analysis
(Kaufman et al., 1990), data compression (vector quanti-
zation) (Gersho & Gray, 1992), and many others. K-
means (KM) (MacQueen, 1967; Selim & Ismail, 1984),
and the Expectation Maximization (EM) (Dempster et al.,
1977; McLachlan & Krishnan, 1997; Rendner & Walker,
1984) with linear mixing of Gaussian density functions
are two of the most popular clustering algorithms.
K-Means is the simplest among the three. It starts
with initializing a set of centers M = {mk | k = 1,..., K } and
iteratively refines the location of these centers to find
the clusters in a dataset. Here are the steps:
K-Means Algorithm
Step 1: Initialize all centers (randomly or based
on any heuristic).
Step 2: Associate each data point with the nearest
center. This step partitions the data set into K
disjoint subsets (Voronoi Partition).
Step 3: Calculate the best center locations (i.e., the
centroids of the partitions) to maximize a perfor-
mance function (2), which is the total squared dis-
tance from each data point to the nearest center.
Step 4: Repeat Steps 2 and 3 until there are no
more changes on the membership of the data points
(proven to converge).
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
With guarantee of convergence to only a local opti-
mum, the quality of the converged results, measured by
the performance function of the algorithm, could be far
from its global optimum. Several researchers explored
alternative initializations to achieve the convergence to
a better local optimum (Bradley & Fayyad, 1998; Meila
& Heckerman, 1998; Pena et al., 1999).
K-Harmonic Means (KHM) (Zhang, 2001; Zhang et
al., 2000) is a recent addition to the family of center-
based clustering algorithms. KHM takes a very different
approach from improving the initializations. It tries to
address directly the source of the problema single
cluster is capable of trapping far more centers than its
fair share. This is the main reason for the existence of a
very large number of local optima under K-Means and
EM when K>10. With the introduction of a dynamic
weighting function of data, KHM is much less sensitive
to initialization, demonstrated through a large number
of experiments in Zhang (2003). The dynamic weighting
function reduces the ability of a single data cluster,
trapping many centers.
Replacing the point-centers by more complex data
model centers, especially regression models, in the
second part of this article, a family of model-based
clustering algorithms is created. Regression clustering
has been studied under a number of different names:
Clusterwise Linear Regression by Spath (1979, 1981,
1983, 1985), DeSarbo and Cron (1988), Hennig (1999,
2000) and others; Trajectory clustering using mixtures
of regression models by Gaffney and Smith (1999);
Fitting Regression Model to Finite Mixtures by Will-
iams (2000); Clustering Using Regression by Gawrysiak,
et. al. (2000); Clustered Partial Linear Regression by
Torgo, et. al. (2000). Regression clustering is a better
name for the family, because it is not limited to linear or
piecewise regressions.
Spath (1979, 1981, 1982) used linear regression and
partition of the dataset, similar to K-means, in his
algorithm that locally minimizes the total mean square
error over all K-regressions. He also developed an
incremental version of his algorithm. He visualized his
piecewise linear regression concept in his book (Spath,
TEAM LinG
Center-Based Clustering and Regression Clustering

1985) exactly as he named his algorithm. DeSarbo (1988) where Sk X is the subset of x that are closer to mk than
used a maximum likelihood method for performing to all other centers (the Voronoi partition). C
clusterwise linear regression. Hennig (1999) studied clus-
tered linear regression, as he named it, using the same linear
K
mixing of Gaussian density functions. 1
(b) EM: d ( x, M ) = log pk * EXP( || xi ml ||2 ) , where
( )
D
k =1

MAIN THRUST { pk }1K is a set of mixing probabilities.

For K-Means, EM, and K-Harmonic means, both their A linear mixture of K identical spherical (Gaussian
performance functions and their iterative algorithms density) functions, which is still a probability density
are treated uniformly in this section for comparison. function, is used here.
This uniform treatment is carried over to the three
regression clustering algorithms, RC-KM, RC-EM and (c) K-Harmonic Means: d ( x, M ) = 1HA (|| x mk || p ) , the
RC-KHM, in the second part. k K

harmonic average of the K distances,

Performance Functions of the Center-
Based Clustering
Among many clustering algorithms, center-based clus-
tering algorithms stand out in two important aspects
a clearly defined objective function that the algorithm
minimizes, compared with agglomerative clustering al-
gorithms that do not have a predefined objective; and a
low runtime cost, compared with many other types of
clustering algorithms. The time complexity per itera-
tion for all three algorithms is linear in the size of the
dataset N, the number of clusters K, and the dimension-
ality of data D. The number of iterations it takes to
converge is very insensitive to N.
Let X = {xi | i = 1,..., N } be a dataset with K clusters,
iid sampled from a hidden distribution, and
M = {mk | k = 1,..., K } be a set of K centers. K-Means, EM,
and K-Harmonic Means find the clustersthe (local)
optimal locations of the centersby minimizing a func-
tion of the following form over the K centers,
Perf ( X , M ) = d ( x, M )
xX
where d(x,M) measures the distance from a data point to
the set of centers. Each algorithm uses a different
distance function:
(a) K-Means: d ( x, M ) = MIN (|| x mk ||) , which makes
1 k K
(1) the same as the more popular form
K
Perf KM ( X , M ) = || x mk || ,
2
k =1 xSk
K
Perf KHM ( X , M ) =
1 , where p > 2.
xX
|| x m ||
mM
2
(3)
i l
K-Means and K-Harmonic Means performance func-
tions also can be written similarly to the EM, except that
only a positive function takes the place where this
probability function is (Zhang, 2001).
Center-Based Clustering Algorithms
K-Means algorithms are shown in the Introduction. We
list EM and K-Harmonic Means algorithms here to
show their similarity.
EM (with Linear Mixing of Spherical
Gausian Densities) Algorithm
Step 1: Initialize the centers and the mixing prob-
abilities { pk }1K .
(1) Step 2: Calculate the expected membership prob-
abilities (see item <B>).
Step 3: Maximize the likelihood to the current
membership by finding the best centers.
Step 4: Repeat Steps 2 and 3 until a chosen con-
vergence criterion is satisfied.
K-Harmonic Means Algorithm
Step 1: Initialize the centers.
(2)
Step 2: Calculate the membership probabilities
and the dynamic weighting (see item <C>).

135

TEAM LinG
 Center-Based Clustering and Regression Clustering

Step 3: Find the best centers to maximize its perfor- N

mk(u ) =
1 N
1
xi , di ,k =|| xi mk(u 1) ||
mance function.
2 2
K 1 K 1
i =1
dip,k+ 2 p
i =1
d p+2
p (7)

i ,k
Step 4: Repeat Steps 2 and 3 until a chosen conver- l =1 d i,l l =1 d i,l

gence criterion is satisfied.

as (Zhang, 2001)
All three iterative algorithms also can be written uni-
formly as: K
1
d p+2
1
p +2
a ( xi ) =
k =1 i ,k d
p(m p(mk( u 1) | xi ) = K ,k , i = 1,..., N
i
( u 1)
k | x) a ( u 1) ( x) x K 1
2
and 1 . (8)
m (u )
= xX
, p p +2
l =1 d i ,l
k
p(m ( u 1)
k | x) a (u 1) ( x) k =1 di ,k
xX

The dynamic weighting function a ( x) 0 approaches

a ( u 1)
( x) > 0, p(m ( u 1)
k | x ) 0 and
(We dropped the iteration index u-1 on p() for shorter
notations). Function a ( u 1) ( x) is a weight on the data point
x in the current iteration. It is called a dynamic weighting
function, because it changes in each iteration. Functions
p (mk(u 1) | x) are soft-membership functions, or the prob-
ability of x being associated to the center mk( u 1) . For each
algorithm, the details on a() and p(,) are:
K
zero when x approaches one of the centers. Intuitively,
p(m
l =1
( u 1)
l | x) = 1 (4)
the closer a data point is to a center, the smaller weight
it gets in the next iteration. This weighting reduces the
ability of a cluster trapping more than one center. The
effect is clearly observed in the visualization of hundreds
of experiments conducted (see next section). Compared
to the KHM, both KM and EM have all data points fully
participate in all iterations (weighting function is a con-
stant 1). They do not have a dynamic weighting function
as K-Harmonic Means does. EM and KHM both have
soft-membership functions, but K-Means has 0/1 mem-
bership function.

A. K-Means: a ( u 1) ( x) =1 for all x in all iterations, and Empirical Comparisons of Center-

p (m ( u 1)
k | x ) =1 if m ( u 1)
k is the closest center to x, Based Clustering Algorithms
otherwise p (mk( u 1) | x) =0. Intuitively, each x is 100%
Empirical comparisons of K-Means, EM, and K-Har-
associated with the closest center, and there is no monic Means on 1,200 randomly generated data sets
weighting on the data points. can be found in the paper by Zhang (2003). Each data set
B. EM: a ( u 1) ( x) =1 for all x in all iterations, and has 50 clusters ranging from well-separated to signifi-
cantly overlapping. The dimensionality of data ranges
p (mk( u 1) | x )
from 2 to 8. All three algorithms are run on each
dataset, starting from the same initialization of the
p ( x | mk(u 1) ) * p (mk(u 1) ) centers, and the converged results are measured by a
p (mk( u 1) | x ) = , common quality measurethe K-Meansfor com-
p( x | mk(u 1) ) * p(mk(u 1) )
xX
(5)
parison. Sensitivity to initialization is studied by a
rerun of all the experiments on different types of
initializations. Major conclusions from the empirical
1
p (mk(u 1) ) = p(mk(u 2) | x),
| X | xX
(6) study are as follows:

1. For low dimensional datasets, the performance

and p ( x | m ( u 1)
k ) is the spherical Gausian density ranking of three algorithms is KHM > KM > EM
(> means better). For low dimensional datasets
function centered at mk( u 1) . (up to 8), the difference is significant.
2. KHMs performance has the smallest variation
C. K-Harmonic Means: a ( x) and p (m ( u 1) ( u 1)
| x ) are under different datasets and different
k
initializations. EMs performance has the biggest
extracted from the KHM algorithm:
variation. Its results are most sensitive to
initializations.

136

TEAM LinG
Center-Based Clustering and Regression Clustering
3. Reproducible results become even more important
when we use these algorithms to different datasets
that are sampled from the same hidden distribution.
The results from KHM better represent the proper-
ties of the distribution and are less dependent on a
particular sample set. EMs results are more depen-
dent on the sample set.
The details on the setup of the experiments, quantita-
tive comparisons of the results, and the Matlab source
code of K-Harmonic Means can be found in the paper.
Generalization to Complex Model-
Based ClusteringRegression
Clustering
Clustering applies to datasets without response infor-
mation (unsupervised); regression applies to datasets
with response variables chosen. Given a dataset with
responses, Z = ( X , Y ) = {( xi , yi ) | i = 1,..., N } , a family of
functions = { f } (a function class making the optimi-
zation problem well defined, such as polynomials of up
to a certain degree) and a loss function e() 0 , regres-
sion solves the following minimization problem (Mont-
gomery et al., 2001):
N Means (RC-KHM).
f opt = arg min e( f ( xi ), yi )
f i =1
m
Commonly, = { l h( x, al ) | l R, al R n } , linear
l =1
expansion of simple parametric functions, such as poly-
nomials of degree up to m, Fourier series of bounded
frequency, neural networks. Usually,
e( f ( x), y ) =|| f ( x) y || , with p=1,2 most widely used
p N
(Friedman, 1999).
Figure 1. (a) Left: a single function is regressed on all
training data, which is a mixture of three different
distributions; (b) Right: three regression functions,
each regressed on a subset found by RC. The residue
errors are much smaller.
Regression in (9) is not effective when the dataset
contains a mixture of very different response character- C
istics, as shown in Figure 1a; it is much better to find the
partitions in the data and to learn a separate function on
each partition, as shown in Figure 1b. This is the idea of
Regression-Clustering (RC). Regression provides a model
for the clusters; clustering partitions the data to best fit
the models. The linkage between the two algorithms is a
common objective function shared between the regres-
sions and the clustering.
RC algorithms can be viewed as replacing the K
geometric-point centers in center-based clustering al-
gorithms by a set of model-based centers, particularly a
set of regression functions M = { f1 ,..., f K } . With
the same performance function as defined in (1), but the
distance from a data point to the set of centers replaced
by the following, ( e( f ( x), y) =|| f ( x) y ||2 )
a) d (( x, y ), M ) = MIN (e( f ( x ), y )) for RC with K-Means
f M
(RC-KM)
K
1
b) d (( x, y ), M ) = log pk * EXP( e( f ( x), y )) for EM
( )
D
k =1
and
c) d (( x, y ), M ) = HA(e( f ( x ), y )) for RC K-Harmonic
f M
(9)
The three iterative algorithmsRC-KM, RC-EM, and
RC-KHMminimizing their corresponding performance
function, take the following common form (10). Regres-
sion with weighting takes the places of weighted averag-
ing in (4). The regression function centers in the uth
iteration are the solution of the minimization,
f k( u ) = arg min a ( zi ) p ( Z k | zi ) || f ( xi ) yi ||2 (10)
f i =1
where the weighting a ( zi ) and the probability p( Z k | zi )
of data point zi in cluster Z k are both calculated from
the (u-1)-iterations centers { f k( u 1) } as follows:
(a) For RC-K-Means, a( zi ) = 1 and p( Z k | zi ) =1 if
e( f k(u 1) ( xi ), yi ) < e( f k('u 1) ( xi ), yi ) k ' k , otherwise
p ( Z k | zi ) =0. Intuitively, RC-K-Means has the fol-
lowing steps:
137
TEAM LinG
 Center-Based Clustering and Regression Clustering

Step 1: Initialize the regression functions. Empirical Comparison of Center-Based

Step 2: Associate each data point (x,y) with the
regression function that provides the best ap-
proximation ( arg kmin{e( f k ( x ), y ) | k = 1,..., K } .
Step 3: Recalculate the regression function on
each partition that maximizes the performance
function.
Step 4: Repeat Steps 2 and 3 until no more data
points change their membership.
Comparing these steps with the steps of K-Means, the
only differences are that point-centers are replaced by
regression-functions; distance from a point to a center is
replaced by the residue error of a pair (x,y) approximated
by a regression function.
(b) For RC-EM, a( zi ) = 1 and
1
pk(u 1) EXP ( e( f k(u 1) ( xi ), yi ))
2
p ( u ) ( Z k | zi ) =
Clustering Algorithms
Comparison of the three RC-algorithms on randomly gen-
erated datasets can be found in the paper by Zhang
(2003a). RC-KHM is shown to be less sensitive to
initialization than RC-KM and RC-EM. Details on imple-
menting the RC algorithms with extended linear regres-
sion models are also available in the same paper.
FUTURE TRENDS
Improving the understanding of dynamic weighting to
the convergence behavior of clustering algorithms and
finding systemic design methods to develop better per-
forming clustering algorithms require more research.
Some of the work in this direction is appearing. Nock
and Nielsen (2004) took the dynamic weighting idea and
developed a general framework similar to boosting
1 N

K
1
pk(u 1) EXP( e( f k(u 1) ( xi ), yi ))
and pk(u 1) =
N
p( Z
i =1
( u 1)
k | zi ) . theory in supervised learning.
k =1 2 Regression clustering will find many applications in
analyzing real-word data. Single-function regression
The same parallel structure can be observed between has been used very widely for data analysis and forecast-
the center-based EM clustering algorithm and the RC- ing. Data collected in an uncontrolled environment, like
EM algorithm. in stocks, marketing, economy, government census, and
many other real-world situations, are very likely to
(c) For RC-K-Harmonic Means, with contain a mixture of different response characters. Re-
e( f ( x), y ) =|| f ( xi ) yi || , p' gression clustering is a natural extension to the classi-
cal single-function regression.
K K K
a p ( zi ) = dip,l' + 2 d p'
i ,l
p '+ 2
and p ( Z k | zi ) = di ,k d p '+ 2
i ,l
l =1 l =1 l =1 CONCLUSION

where d i ,l =|| f (u 1) ( xi ) yi || . ( p > 2 is used.) Replacing the simple geometric-point centers in cen-
The same parallel structure can be observed between ter-based clustering algorithms by more complex data
the center-based KHM clustering algorithm and the RC- models provides a general scheme for deriving other
KHM algorithm. model-based clustering algorithms. Regression models
Sensitivity to initialization in center-based cluster- are used in this presentation to demonstrate the process.
ing carries over to regression clustering. In addition, a The key step in the generalization is defining the dis-
new form of local optimum is illustrated in Figure 2. tance function from a data point to the set of models
It happens to all three RC algorithms, RC-KM, RC- the regression functions in this special case.
KHM, and RC-EM. Among the three algorithms, EM has a strong foun-
dation in probability theory. It is the convergence to
only a local optimum and the existence of a very large
Figure 2. A new kind of local optimum occurs in number of optima when the number of clusters is more
regression clustering. than a few (>5, for example) that keeps practitioners
from the benefits of its theory. K-Means is the simplest
and its objective function the most intuitive. But it has
the similar problem as the EMs sensitivity to initializa-
tion of the centers. K-Harmonic Means was developed
with close attention to the dynamics of its convergence;
it is much more robust than the other two on low dimen-

138

TEAM LinG
Center-Based Clustering and Regression Clustering
sional data. Improving the convergence of center-based
clustering algorithms on higher dimensional data (dim >
10) still needs more research.
REFERENCES
Bradley, P., & Fayyad, U.M. (1998). Refining initial points
for KM clustering. MS Technical Report MSR-TR-98-36.
Dempster, A.P., Laird, N.M., & Rubin, D.B. (1977). Maxi-
mum likelihood from incomplete data via the EM algo-
rithm. Journal of the Royal Statistical Society, 39(1), 1-
38.
DeSarbo, W.S., & Corn, L.W. (1988). A maximum likeli-
hood methodology for clusterwise linear regression. Jour-
nal of Classification, 5, 249-282.
Duda, R., & Hart, P. (1972). Pattern classification and
scene analysis. John Wiley & Sons.
Friedman, J., Hastie, T., & Tibshirani. R. (1998). Additive
logistic regression: A statistical view of boosting [tech-
nical report]. Department of Statistics, Stanford Univer-
sity.
Gersho, A., & Gray, R.M. (1992). Vector quantization and
signal compression. Kluwer Academic Publishers.
Hamerly, G., & Elkan, C. (2002). Alternatives to the k-
means algorithm that find better clusterings. Proceed-
ings of the ACM conference on information and knowl-
edge management (CIKM).
Hamerly, G., & Elkan, C. (2003). Learning the k in k-means.
Proceedings of the Seventeenth Annual Conference on
Neural Information Processing Systems.
Hennig, C. (1997). Datenanalyse mit modellen fur cluster
linear regression [Dissertation]. Hamburg, Germany:
Institut Fur Mathmatsche Stochastik, Universitat Ham-
burg.
Kaufman, L., & Rousseeuw, P.J. (1990). Finding groups in
data: An introduction to cluster analysis. John Wiley &
Sons
MacQueen, J. (1967). Some methods for classification and
analysis of multivariate observations. Proceedings of the
Fifth Berkeley Symposium on Mathematical Statistics
and Probability, Berkeley, California.
McLachlan, G. J., & Krishnan, T. (1997). EM algorithm
and extensions. John Wiley & Sons.
Meila, M., & Heckerman, D. (1998). An experimental com-
parison of several clustering and initialization methods. In
Proceedings of the Fourteenth Conference on Intelli-
gence Artificial in Uncertainty (pp. 386-395). Morgan
Kaufman. C
Montgomery, D.C., Peck, E.A., & Vining, G.G. (2001).
Introduction to linear regression analysis. John Wiley
& Sons.
Nock, R., & Nielsen, F. (2004). An abstract weighting
framework for clustering algorithms. Proceedings of
the Fourth International SIAM Conference on Data
Mining. Orlando, Florida.
Pena, J., Lozano, J., & Larranaga, P. (1999). An empirical
comparison of four initialization methods for the K-
means algorithm. Pattern Recognition Letters, 20,
1027-1040.
Rendner, R.A., & Walker, H.F. (1984). Mixture densi-
ties, maximum likelihood and the EM algorithm. SIAM
Review, 26(2).
Schapire, R.E. (1999). Theoretical views of boosting and
applications. Proceedings of the Tenth International
Conference on Algorithmic Learning Theory.
Selim, S.Z., & Ismail, M.A (1984). K-means type algo-
rithms: A generalized convergence theorem and charac-
terization of local optimality. IEEE Transactions on
PAMI-6, 1.
Silverman, B.W. (1998). Density estimation for statis-
tics and data analysis. Chapman & Hall/CRC.
Spath, H. (1981). Correction to algorithm 39:
Clusterwise linear regression. Computing, 26, 275.
Spath, H. (1982). Algorithm 48: A fast algorithm for
clusterwise linear regression. Computing, 29, 175-181.
Spath, H. (1985). Cluster dissection and analysis. New
York: Wiley.
Tibshirani, R., Walther, G., & Hastie, T. (2000). Estimating
the number of clusters in a dataset via the gap statistic.
Retrieved from http://www-stat.stanford.edu/~tibs /
research.html
Zhang, B. (2001). Generalized K-harmonic meansDy-
namic weighting of data in unsupervised learning. Pro-
ceedings of the First SIAM International Conference on
Data Mining (SDM2001), Chicago, Illinois.
Zhang, B. (2003). Comparison of the performance of cen-
ter-based clustering algorithms. Proceedings of PAKDD-
03, Seoul, South Korea.
Zhang, B. (2003a). Regression clustering. Proceedings of
the IEEE International Conference on Data Mining,
Melbourne, Florida.
139
TEAM LinG

Zhang, B., Hsu, M., & Dayal, U. (2000). K-harmonic means.
Proceedings of the International Workshop on Tempo-
ral, Spatial and Spatio-Temporal Data Mining, Lyon,
France.
KEY TERMS
Boosting: Assigning and updating weights on data
points according to a particular formula in the process of
refining classification models.
Center-Based Clustering: Similarity among the data
points is defined through a set of centers. The distance
from each data point to a center determined the data points
association with that center. The clusters are represented
by the centers.
Clustering: Grouping data according to similarity
among them. Each clustering algorithm has its own defi-
nition of similarity. Such grouping can be hierarchical.
Dynamic Weighting: Reassigning weights on the
data points in each iteration of an iterative algorithm.
140
Center-Based Clustering and Regression Clustering
Model-Based Clustering: A mixture of simpler distri-
butions is used to fit the data, which defines the clusters
of the data. EM with linear mixing of Gaussian density
functions is the best example, but K-Means and K-Har-
monic Means are the same type. Regression clustering
algorithms are also model-based clustering algorithms
with mixing of more complex distributions as its model.
Regression: A statistical method of learning the rela-
tionship between two sets of variables from data. One set
is the independent variables or the predictors, and the
other set is the response variables.
Regression Clustering: Combining the regression
methods with center-based clustering methods. The
simple geometric-point centers in the center-based clus-
tering algorithms are replaced by regression models.
Sensitivity to Initialization: Center-based clus-
tering algorithms are iterative algorithms that minimiz-
ing the value of its performance function. Such algo-
rithms converge to only a local optimum of its perfor-
mance function. The converged positions of the centers
depend on the initial positions of the centers where the
algorithm start with.
TEAM LinG

Classification and Regression Trees
Johannes Gehrke
Cornell University, USA
INTRODUCTION
It is the goal of classification and regression to build a
data-mining model that can be used for prediction. To
construct such a model, we are given a set of training
records, each having several attributes. These attributes
either can be numerical (e.g., age or salary) or categorical
(e.g., profession or gender). There is one distinguished
attributethe dependent attribute; the other attributes
are called predictor attributes. If the dependent attribute
is categorical, the problem is a classification problem. If
the dependent attribute is numerical, the problem is a
regression problem. It is the goal of classification and
regression to construct a data-mining model that predicts
the (unknown) value for a record, where the value of the
dependent attribute is unknown. (We call such a record an
unlabeled record.) Classification and regression have a
wide range of applications, including scientific experi-
ments, medical diagnosis, fraud detection, credit approval,
and target marketing (Hand, 1997).
Many classification and regression models have been
proposed in the literature; among the more popular mod-
els are neural networks, genetic algorithms, Bayesian
methods, linear and log-linear models and other statistical
methods, decision tables, and tree-structured models,
which is the focus of this article (Breiman, Friedman,
Olshen & Stone, 1984). Tree-structured models, so-called
decision trees, are easy to understand; they are non-
parametric and, thus, do not rely on assumptions about
the data distribution; and they have fast construction
methods even for large training datasets (Lim, Loh & Shih,
2000). Most data-mining suites include tools for classifi-
cation and regression tree construction (Goebel &
Gruenwald, 1999).
BACKGROUND
Let us start by introducing decision trees. For the ease of
explanation, we are going to focus on binary decision
trees. In binary decision trees, each internal node has two
children nodes. Each internal node is associated with a
predicate, called the splitting predicate, which involves
only the predictor attributes. Each leaf node is associated
with a unique value for the dependent attribute. A deci-
sion encodes a data-mining model as follows. For an
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
141
C
unlabeled record, we start at the root node. If the record
satisfies the predicate associated with the root node, we
follow the tree to the left child of the root, and we go to the
right child otherwise. We continue this pattern through a
unique path from the root of the tree to a leaf node, where
we predict the value of the dependent attribute associated
with this leaf node. An example decision tree for a classi-
fication problem, a classification tree, is shown in Figure
1. Note that a decision tree automatically captures inter-
actions between variables, but it only includes interac-
tions that help in the prediction of the dependent at-
tribute. For example, the rightmost leaf node in the example
shown in Figure 1 is associated with the classification
rule: If (Age >= 40) and (Gender=male), then YES; as
classification rule that involves an interaction between
the two predictor attributes age and salary.
Decision trees can be mined automatically from a
training database of records, where the value of the
dependent attribute is known: A decision tree construc-
tion algorithm selects which attribute(s) to involve in the
splitting predicates, and the algorithm decides also on the
shape and depth of the tree (Murthy, 1998).
MAIN THRUST
Let us discuss how decision trees are mined from a
training database. A decision tree usually is constructed
in two phases. In the first phase, the growth phase, an
overly large and deep tree is constructed from the training
data. In the second phase, the pruning phase, the final size
of the tree is determined with the goal to minimize the
expected misprediction error (Quinlan, 1993).
Figure 1. An example classification tree
Age <
40
No Gender=M
No Yes
TEAM LinG

There are two problems that make decision tree con-
struction a hard problem. First, construction of the opti-
mal tree for several measures of optimality is an NP-hard
problem. Thus, all decision tree construction algorithms
grow the tree top-down according to the following greedy
heuristic: At the root node, the training database is
examined, and a splitting predicate is selected. Then the
training database is partitioned according to the splitting
predicate, and the same method is applied recursively at
each child node. The second problem is that the training
database is only a sample from a much larger population
of records. The decision tree has to perform well on
records drawn from the population, not on the training
database. (For the records in the training database, we
already know the value of the dependent attribute.)
Three different algorithmic issues need to be ad-
dressed during the tree construction phase. The first
issue is to devise a split selection algorithm, such that the
resulting tree models the underlying dependency rela-
tionship between the predictor attributes and the depen-
dent attribute well. During split selection, we have to make
two decisions. First, we need to decide which attribute we
will select as the splitting attribute. Second, given the
splitting attribute, we have to decide on the actual split-
ting predicate. For a numerical attribute X, splitting predi-
cates are usually of the form X c, where c is a constant.
For example, in the tree shown in Figure 1, the splitting
predicate of the root node is of this form. For a categorical
attribute X, splits are usually of the form X in C, where C
is a set of values in the domain of X. For example, in the
tree shown in Figure 1, the splitting predicate of the right
child node of the root is of this form. There exist decision
trees that have a larger class of possible splitting predi-
cates; for example, there exist decision trees with linear
combinations of numerical attribute values as splitting
predicates a iXi+c0, where i ranges over all attributes)
(Loh & Shih, 1997). Such splits, also called oblique splits,
result in shorter trees; however, the resulting trees are no
longer easy to interpret.
The second issue is to devise a pruning algorithm that
selects the tree of the right size. If the tree is too large, then
the tree models the training database too closely instead
of modeling the underlying population. One possible
choice of pruning a tree is to hold out part of the training
set as a test set and to use the test set to estimate the
misprediction error of trees of different size. We then
simply select the tree that minimizes the misprediction
error.
The third issue is to devise an algorithm for intelligent
management of the training database in case the training
database is very large (Ramakrishnan & Gehrke, 2002).
This issue has only received attention in the last decade,
but there exist now many algorithms that can construct
decision trees over extremely large, disk-resident training
142
Classification and Regression Trees
databases (Gehrke, Ramakrishnan & Ganti, 2000; Shafer,
Agrawal & Mehta, 1996).
In most classification and regression scenarios, we
also have costs associated with misclassifying a record,
or with being far off in our prediction of a numerical
dependent value. Existing decision tree algorithms can
take costs into account, and they will bias the model
toward minimizing the expected misprediction cost in-
stead of the expected misclassification rate, or the ex-
pected difference between the predicted and true value of
the dependent attribute.
FUTURE TRENDS
Recent developments have expanded the types of models
that a decision tree can have in its leaf nodes. So far, we
assumed that each leaf node just predicts a constant value
for the dependent attribute. Recent work, however, has
shown how to construct decision trees with linear models
in the leaf nodes (Dobra & Gehrke, 2002). Another recent
development in the general area of data mining is the use
of ensembles of models, and decision trees are a popular
model for use as a base model in ensemble learning
(Caruana, Niculescu-Mizil, Crew & Ksikes, 2004). An-
other recent trend is the construction of data-mining
models of high-speed data streams, and there have been
adaptations of decision tree construction algorithms to
such environments (Domingos & Hulten, 2002). A last
recent trend is to take adversarial behavior into account
(e.g., in classifying spam). In this case, an adversary who
produces the records to be classified actively changes his
or her behavior over time to outsmart a static classifier
(Dalvi, Domingos, Mausam, Sanghai & Verma, 2004).
CONCLUSION
Decision trees are one of the most popular data-mining
models. Decision trees are important, since they can result
in powerful predictive models, while, at the same time,
they allow users to get insight into the phenomenon that
is being modeled.
REFERENCES
Breiman, L., Friedman, J.H., Olshen, R.A., & Stone, C.J.
(1984). Classification and regression trees. Kluwer Aca-
demic Publishers.
Caruana, R., Niculescu-Mizil, A., Crew, R., & Ksikes, A.
(2004). Ensemble selection from libraries of models. Pro-
TEAM LinG
Classification and Regression Trees
ceedings of the Twenty-First International Conference,
Banff, Alberta, Canada.
Dalvi, N., Domingos, P., Mausam, S.S., & Verma, D. (2004).
Adversarial classification. Proceedings of the Tenth Inter-
national Conference on Knowledge Discovery and Data
Mining, Seattle, Washington.
Dobra, A., & Gehrke, J. (2002). SECRET: A scalable linear
regression tree algorithm. Proceedings of the Eighth ACM
SIGKDD International Conference on Knowledge Dis-
covery and Data Mining, Edmonton, Alberta, Canada.
Domingos, P., & Hulten, G. (2002). Learning from infinite
data in finite time. Advances in Neural Information Pro-
cessing Systems, 14, 673-680.
Gehrke, J., Ramakrishnan, R., & Ganti, V. (2000).
RainforestA framework for fast decision tree construc-
tion of large datasets. Data Mining and Knowledge Dis-
covery, 4(2/3), 127-162.
Goebel, M., & Gruenwald, L. (1999). A survey of data
mining software tools. SIGKDD Explorations, 1(1), 20-33.
Hand, D. (1997). Construction and assessment of classifi-
cation rules. Chichester, England: John Wiley & Sons.
Lim, T.-S., Loh, W.-Y., & Shih, Y.-S. (2000). A comparison
of prediction accuracy, complexity, and training time of
thirty-three old and new classification algorithms. Ma-
chine Learning, 48, 203-228.
Loh, W.-Y., & Shih, Y.-S. (1997). Split selection methods
for classification trees. Statistica Sinica, 7(4), 815-840.
Murthy, S.K. (1998). Automatic construction of decision
trees from data: A multi-disciplinary survey. Data Mining
and Knowledge Discovery, 2(4), 345-389.
Quinlan, J.R. (1993). C4.5: Programs for machine learn-
ing. Morgan Kaufman. C
Ramakrishnan, R. & Gehrke, J. (2002). Database manage-
ment systems (3rd ed.). McGrawHill.
Shafer, J., Agrawal, R., & Mehta, M. (1996). SPRINT: A
scalable parallel classifier for data mining. Proceedings
of the 22nd International Conference on Very Large
Databases, Bombay, India.
KEY TERMS
Attribute: Column of a dataset.
Categorical Attribute: Attribute that takes values
from a discrete domain.
Classification Tree: A decision tree where the de-
pendent attribute is categorical.
Decision Tree: Tree-structured data mining model
used for prediction, where internal nodes are labeled with
predicates (decisions), and leaf nodes are labeled with
data-mining models.
Numerical Attribute: Attribute that takes values
from a continuous domain.
Regression Tree: A decision tree where the depen-
dent attribute is numerical.
Splitting Predicate: Predicate at an internal node of
the tree; it decides which branch a record traverses on its
way from the root to a leaf node.
143
TEAM LinG
144
Classification Methods
Aijun An
York University, Canada
INTRODUCTION
Generally speaking, classification is the action of as-
signing an object to a category according to the charac-
teristics of the object. In data mining, classification
refers to the task of analyzing a set of pre-classified data
objects to learn a model (or a function) that can be used
to classify an unseen data object into one of several
predefined classes. A data object, referred to as an
example, is described by a set of attributes or variables.
One of the attributes describes the class that an example
belongs to and is thus called the class attribute or class
variable. Other attributes are often called independent
or predictor attributes (or variables). The set of ex-
amples used to learn the classification model is called
the training data set. Tasks related to classification
include regression, which builds a model from training
data to predict numerical values, and clustering, which
groups examples to form categories. Classification be-
longs to the category of supervised learning, distin-
guished from unsupervised learning. In supervised learn-
ing, the training data consists of pairs of input data
(typically vectors), and desired outputs, while in unsu-
pervised learning there is no a priori output.
Classification has various applications, such as learn-
ing from a patient database to diagnose a disease based
on the symptoms of a patient, analyzing credit card
transactions to identify fraudulent transactions, auto-
matic recognition of letters or digits based on handwrit-
ing samples, and distinguishing highly active compounds
from inactive ones based on the structures of com-
pounds for drug discovery.
BACKGROUND
Classification has been studied in statistics and machine
learning. In statistics, classification is also referred to
as discrimination. Early work on classification focused
on discriminant analysis, which constructs a set of
discriminant functions, such as linear functions of the
predictor variables, based on a set of training examples
to discriminate among the groups defined by the class
variable. Modern studies explore more flexible classes
of models, such as providing an estimate of the join
distribution of the features within each class (e.g. Baye-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
sian classification), classifying an example based on dis-
tances in the feature space (e.g. the k-nearest neighbor
method), and constructing a classification tree that clas-
sifies examples based on tests on one or more predictor
variables (i.e., classification tree analysis).
In the field of machine learning, attention has more
focused on generating classification expressions that
are easily understood by humans. The most popular
machine learning technique is decision tree learning,
which learns the same tree structure as classification
trees but uses different criteria during the learning
process. The technique was developed in parallel with
the classification tree analysis in statistics. Other ma-
chine learning techniques include classification rule
learning, neural networks, Bayesian classification, in-
stance-based learning, genetic algorithms, the rough set
approach and support vector machines. These tech-
niques mimic human reasoning in different aspects to
provide insight into the learning process.
The data mining community inherits the classifica-
tion techniques developed in statistics and machine
learning, and applies them to various real world prob-
lems. Most statistical and machine learning algorithms
are memory-based, in which the whole training data set
is loaded into the main memory before learning starts.
In data mining, much effort has been spent on scaling up
the classification algorithms to deal with large data sets.
There is also a new classification technique, called
association-based classification, which is based on as-
sociation rule learning.
MAIN THRUST
Major classification techniques are described below.
The techniques differ in the learning mechanism and in
the representation of the learned model.
Decision Tree Learning
Decision tree learning is one of the most popular clas-
sification algorithms. It induces a decision tree from
data. A decision tree is a tree structured prediction
model where each internal node denotes a test on an
attribute, each outgoing branch represents an outcome
of the test, and each leaf node is labeled with a class or
TEAM LinG
Classification Methods
Figure 1. A decision tree with tests on attributes X and Y
X=?
X<1 X 1
Y =? Class 2
Y=C
Y=A Y=B
Class 1 Class 2 Class 1
class distribution. A simple decision tree is shown in Figure
1. With a decision tree, an object is classified by following
a path from the root to a leaf, taking the edges correspond-
ing to the values of the attributes in the object.
A typical decision tree learning algorithm adopts a
top-down recursive divide-and-conquer strategy to con-
struct a decision tree. Starting from a root node repre-
senting the whole training data, the data is split into two
or more subsets based on the values of an attribute
chosen according to a splitting criterion. For each sub-
set a child node is created and the subset is associated
with the child. The process is then separately repeated
on the data in each of the child nodes, and so on, until a
termination criterion is satisfied. Many decision tree
learning algorithms exist. They differ mainly in at-
tribute-selection criteria, such as information gain, gain
ratio (Quinlan, 1993), gini index (Breiman, Friedman,
Olshen, & Stone, 1984), etc., termination criteria and
post-pruning strategies. Post-pruning is a technique that
removes some branches of the tree after the tree is con-
structed to prevent the tree from over-fitting the training
data. Representative decision tree algorithms include CART
(Breiman et al., 1984) and C4.5 (Quinlan, 1993). There are
also studies on fast and scalable construction of decision
trees. Representative algorithms of such kind include
RainForest (Gehrke, Ramakrishnan, & Ganti, 1998) and
SPRINT (Shafer, Agrawal, & Mehta., 1996).
Decision Rule Learning
Decision rules are a set of if-then rules. They are the
most expressive and human readable representation of
classification models (Mitchell, 1997). An example of
decision rules is if X<1 and Y=B, then the example
belongs to Class 2. This type of rules is referred to as
propositional rules. Rules can be generated by translat-
ing a decision tree into a set of rules one rule for each
leaf node in the tree. A second way to generate rules is
to learn rules directly from the training data. There is a
variety of rule induction algorithms. The algorithms
induce rules by searching in a hypothesis space for a
hypothesis that best matches the training data. The algo-
rithms differ in the search method (e.g. general-to-
specific, specific-to-general, or two-way search), the
search heuristics that control the search, and the pruning
method used. The most widespread approach to rule C
induction is sequential covering, in which a greedy
general-to-specific search is conducted to learn a dis-
junctive set of conjunctive rules. It is called sequential
covering because it sequentially learns a set of rules that
together cover the set of positive examples for a class.
Algorithms belonging to this category include CN2
(Clark & Boswell, 1991), RIPPER (Cohen, 1995) and
ELEM2 (An & Cercone, 1998).
Naive Bayesian Classifier
The naive Bayesian classifier is based on Bayes theo-
rem. Suppose that there are m classes, C1, C2, , Cm. The
classifier predicts an unseen example X as belonging to
the class having the highest posterior probability condi-
tioned on X. In other words, X is assigned to class Ci if
and only if
P(Ci|X) > P(Cj|X) for 1j m, j i.
By Bayes theorem, we have
P( X | Ci ) P (Ci )
P (Ci | X ) = .
P( X )
As P(X) is constant for all classes, only
P( X | C i ) P(C i ) needs to be maximized. Given a set of
training data, P(Ci) can be estimated by counting how
often each class occurs in the training data. To reduce
the computational expense in estimating P(X|Ci) for all
possible Xs, the classifier makes a nave assumption that
the attributes used in describing X are conditionally
independent of each other given the class of X. Thus,
given the attribute values (x1, x2, xn) that describe X,
we have
n
P( X | C i ) = P( x j | C i ) .
j =1
The probabilities P(x1|Ci), P(x2|Ci), , P(xn|Ci) can be
estimated from the training data.
The nave Bayesian classifier is simple to use and
efficient to learn. It requires only one scan of the
training data. Despite the fact that the independence
assumption is often violated in practice, nave Bayes
often competes well with more sophisticated classifi-
ers. Recent theoretical analysis has shown why the naive
145
TEAM LinG

Bayesian classifier is so robust (Domingos & Pazzani,
1997; Rish, 2001).
Bayesian Belief Networks
A Bayesian belief network, also known as Bayesian net-
work and belief network, is a directed acyclic graph whose
nodes represent variables and whose arcs represent de-
pendence relations among the variables. If there is an arc
from node A to another node B, then we say that A is a
parent of B and B is a descendent of A. Each variable is
conditionally independent of its nondescendents in the
graph, given its parents. The variables may correspond to
actual attributes given in the data or to hidden variables
believed to form a relationship. A variable in the network
can be selected as the class attribute. The classification
process can return a probability distribution for the class
attribute based on the network structure and some condi-
tional probabilities estimated from the training data, which
predicts the probability of each class.
The Bayesian network provides an intermediate ap-
proach between the nave Bayesian classification and the
Bayesian classification without any independence as-
sumptions. It describes dependencies among attributes,
but allows conditional independence among subsets of
attributes.
The training of a belief network depends on the
senario. If the network structure is known and the vari-
ables are observable, training the network only consists
of estimating some conditional probabilities from the
training data, which is straightforward. If the network
structure is given and some of the variables are hidden, a
method of gradient decent can be used to train the net-
work (Russell, Binder, Koller, & Kanazawa, 1995). Al-
gorithms also exist for learning the netword structure
from training data given observable variables (Buntime,
1994; Cooper & Herskovits, 1992; Heckerman, Geiger,
& Chickering, 1995).
The k-Nearest Neighbour Classifier
The k-nearest neighbour classifier classifies an unknown
example to the most common class among its k nearest
neighbors in the training data. It assumes all the ex-
amples correspond to points in a n-dimensional space. A
neighbour is deemed nearest if it has the smallest dis-
tance, in the Euclidian sense, in the n-dimensional fea-
ture space. When k = 1, the unknown example is classi-
fied into the class of its closest neighbour in the training
set. The k-nearest neighbour method stores all the train-
ing examples and postpones learning until a new example
needs to be classified. This type of learning is called
instance-based or lazy learning.
146
Classification Methods
The k-nearest neighbour classifier is intuitive, easy
to implement and effective in practice. It can construct a
different approximation to the target function for each
new example to be classified, which is advantageous
when the target function is very complex, but can be
discribed by a collection of less complex local approxima-
tions (Mitchell, 1997). However, its cost of classifying
new examples can be high due to the fact that almost all
the computation is done at the classification time. Some
refinements to the k-nearest neighbor method include
weighting the attributes in the distance computation and
weighting the contribution of each of the k neighbors
during classification according to their distance to the
example to be classified.
Neural Networks
Neural networks, also referred to as artificial neural
networks, are studied to simulate the human brain
although brains are much more complex than any arti-
ficial neural network developed so far. A neural net-
work is composed of a few layers of interconnected
computing units (neurons or nodes). Each unit com-
putes a simple function. The input of the units in one
layer are the outputs of the units in the previous layer.
Each connection between units is associated with a
weight. Parallel computing can be performed among
the units in each layer. The units in the first layer take
input and are called the input units. The units in the last
layer produces the output of the networks and are called
the output units. When the network is in operation, a
value is applied to each input unit, which then passes its
given value to the connections leading out from it, and
on each connection the value is multiplied by the weight
associated with that connection. Each unit in the next
layer then receives a value which is the sum of the
values produced by the connections leading into it, and
in each unit a simple computation is performed on the
value - a sigmoid function is typical. This process is
then repeated, with the results being passed through
subsequent layers of nodes until the output nodes are
reached. Neural networks can be used for both regres-
sion and classification. To model a classification func-
tion, we can use one output unit per class. An example
can be classified into the class corresponding to the
output unit with the largest output value.
Neural networks differ in the way in which the
neurons are connected, in the way the neurons process
their input, and in the propogation and learning methods
used (Nurnberger, Pedrycz, & Kruse, 2002). Learning
a neural network is usually restricted to modifying the
weights based on the training data; the structure of the
initial network is usually left unchanged during the
learning process. A typical network structure is the
TEAM LinG
Classification Methods
multilayer feed-forward neural network, in which none
of the connections cycles back to a unit of a previous
layer. The most widely used method for training a feed-
forward neural network is backpropagation (Rumelhart,
Hinton, & Williams, 1986).
Support Vector Machines
The support vector machine (SVM) is a recently devel-
oped technique for multidimensional function approxi-
mation. The objective of support vector machines is to
determine a classifier or regression function which
minimizes the empirical risk (that is, the training set
error) and the confidence interval (which corresponds
to the generalization or test set error) (Vapnik, 1998).
Given a set of N linearly separable training ex-
amples S = {x i R n | i = 1,2,..., N } , where each example
belongs to one of the two classes, represented
by yi {+1,1} , the SVM learning method seeks the op-
timal hyperplane w x + b = 0 , as the decision surface,
which separates the positive and negative examples with
the largest margin. The decision function for classify-
ing linearly separable data is:
f (x) = sign(w x + b) ,
where w and b are found from the training set by solving
a constrained quadratic optimization problem. The final
decision function is
N
f (x) = sign i y i (x i x) + b .
i =1
The function depends on the training examples for
which i is non-zero. These examples are called sup-
port vectors. Often the number of support vectors is
only a small fraction of the original dataset. The basic
SVM formulation can be extended to the nonlinear case
by using nonlinear kernels that map the input space to a
high dimensional feature space. In this high dimensional
feature space, linear classification can be performed.
The SVM classifier has become very popular due to its
high performances in practical applications such as text
classification and pattern recognition.
FUTURE TRENDS
Classification is a major data mining task. As data mining
becomes more popular, classification techniques are in-
creasingly applied to provide decision support in busi-
ness, biomedicine, financial analysis, telecommunications C
and so on. For example, there are recent applications of
classification techniques to identify fraudulent usage of
credit cards based on credit card transaction databases;
and various classification techniques have been explored
to identify highly active compounds for drug discovery.
To better solve application-specific problems, there has
been a trend toward the development of more application-
specific data mining systems (Han & Kamber, 2001).
Traditional classification algorithms assume that the
whole training data can fit into the main memory. As
automatic data collection becomes a daily practice in many
businesses, large volumes of data that exceed the memory
capacity become available to the learning systems. Scal-
able classification algorithms become essential. Although
some scalable algorithms for decision tree learning have
been proposed, there is still a need to develop scalable and
efficient algorithms for other types of classification tech-
niques, such as decision rule learning.
Previously, the study of classification techniques
focused on exploring various learning mechanisms to
improve the classification accuracy on unseen examples.
However, recent study on imbalanced data sets has
shown that classification accuracy is not an appropriate
measure to evaluate the classification performance when
the data set is extremely unbalanced, in which almost all
the examples belong to one or more, larger classes and
far fewer examples belong to a smaller, usually more
interesting class. Since many real world data sets are
unbalanced, there has been a trend toward adjusting
existing classification algorithms to better identify ex-
amples in the rare class.
Another issue that has become more and more impor-
tant in data mining is privacy protection. As data mining
tools are applied to large databases of personal records,
privacy concerns are rising. Privacy-preserving data min-
ing is currently one of the hottest research topics in data
mining and will remain so in the near future.
CONCLUSION
Classification is a form of data analysis that extracts a
model from data to classify future data. It has been
studied in parallel in statistics and machine learning, and
is currently a major technique in data mining with a
broad application spectrum. Since many application
problems can be formulated as a classification problem
and the volume of the available data has become over-
whelming, developing scalable, efficient, domain-spe-
cific, and privacy-preserving classification algorithms is
essential.
147
TEAM LinG

REFERENCES
An, A., & Cercone, N. (1998). ELEM2: A learning
system for more accurate classifications. Proceedings
of the 12th Canadian Conference on Artificial Intelli-
gence (pp. 426-441).
Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984).
Classification and regression trees. Wadsworth Interna-
tional Group.
Buntine, W.L. (1994). Operations for learning with graphi-
cal models. Journal of Artificial Intelligence Research, 2,
159-225.
Castillo, E., Gutirrez, J.M., & Hadi, A.S. (1997). Expert
systems and probabilistic network models. New York:
Springer-Verlag.
Clark P., & Boswell, R. (1991). Rule induction with CN2:
Some recent improvements. Proceedings of the 5th Euro-
pean Working Session on Learning (pp. 151-163).
Cohen, W.W. (1995). Fast effective rule induction. Pro-
ceedings of the 11th International Conference on Ma-
chine Learning (pp. 115-123), Morgan Kaufmann.
Cooper, G., & Herskovits, E. (1992). A Bayesian method
for the induction of probabilistic networks from data.
Machine Learning, 9, 309-347.
Domingos, P., & Pazzani, M. (1997). On the optimality
of the simple Bayesian classifier under zero-one loss.
Machine Learning, 29, 103-130.
Gehrke, J., Ramakrishnan, R., & Ganti, V. (1998). RainForest
- A framework for fast decision tree construction of large
datasets. Proceedings of the 24th International Confer-
ence on Very Large Data Bases (pp. 416-427).
Han, J., & Kamber, M. (2001). Data mining Concepts
and techniques. Morgan Kaufmann.
Heckerman, D., Geiger, D., & Chickering, D.M. (1995) Learn-
ing bayesian networks: The combination of knowledge and
statistical data. Machine Learning, 20, 197-243.
Mitchell, T.M. (1997). Machine learning. McGraw-Hill.
Nurnberger, A., Pedrycz, W., & Kruse, R. (2002). Neural
network approaches. In Klosgen & Zytkow (Eds.), Hand-
book of data mining and knowledge discovery (pp. 304-
317). Oxford University Press.
Pearl, J. (1986). Fusion, propagation, and structuring in
belief networks. Artificial Intelligence, 29(3), 241-288.
Quinlan, J.R. (1993). C4.5: Programs for machine learn-
ing. Morgan Kaufmann.
148
Classification Methods
Rish, I. (2001). An empirical study of the naive Bayes
classifier. Proceedings of IJCAI 2001 Workshop on
Empirical Methods in Artificial Intelligence.
Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986).
Learning representations by back-propagating errors.
Nature, 323, 533-536.
Russell, S., Binder, J., Koller, D., & Kanazawa, K. (1995).
Local learning in probabilistic networks with hidden
variables. Proceedings of the 14th Joint International
Conference on Artificial Intelligence, 2 (pp. 1146-1152).
Shafer, J., Agrawal, R., & Mehta, M. (1996). SPRINT: A
scalable parallel classifier for data mining. Proceed-
ings of the 22 th International Conference on Very
Large Data Bases (pp. 544-555).
Vapnik, V. (1998). Statistical learning theory. New York:
John Wiley & Sons.
KEY TERMS
Backpropagation: A neural network training algo-
rithm for feedforward networks where the errors at the
output layer are propagated back to the previous layer to
update connection weights in learning. If the previous
layer is not the input layer, then the errors at this hidden
layer are propagated back to the layer before.
Disjunctive Set of Conjunctive Rules: A conjunc-
tive rule is a propositional rule whose antecedent con-
sists of a conjunction of attribute-value pairs. A dis-
junctive set of conjunctive rules consists of a set of
conjunctive rules with the same consequent. It is called
disjunctive because the rules in the set can be combined
into a single disjunctive rule whose antecedent consists
of a disjunction of conjunctions.
Generic Algorithm: An algorithm for optimizing a
binary string based on an evolutionary mechanism that
uses replication, deletion, and mutation operators car-
ried out over many generations.
Information Gain: Given a set E of classified ex-
amples and a partition P = {E1, ..., En} of E, the informa-
tion gain is defined as
n
| Ei |
entropy( E ) entropy( Ei ) * ,
i =1 |E|
where |X| is the number of examples in X, and
m
entropy( X ) = p j log 2 ( p j ) (assuming there are m classes
j =1
TEAM LinG
Classification Methods

in X and pj denotes the probability of the jth class in X).
Intuitively, the information gain measures the decrease of
the weighted average impurity of the partitions E1, ..., E n,
compared with the impurity of the complete set of ex-
amples E.
approximations of a class. It can be used to reduce the
feature set and to generate decision rules. C
Sigmod Function: A mathematical function defined
by the formula

Machine Learning: The study of computer algorithms 1

P (t ) =
that develop new knowledge and improve its performance 1 + e t
automatically through past experience.
Its name is due to the sigmoid shape of its graph. This
Rough Set Data Analysis: A method for modeling function is also called the standard logistic function.
uncertain information in data by forming lower and upper

149

TEAM LinG
150
Closed-Itemset Incremental-Mining Problem
Luminita Dumitriu
Dunarea de Jos University, Romania
INTRODUCTION
Association rules, introduced by Agrawal, Imielinski
and Swami (1993), provide useful means to discover
associations in data. The problem of mining association
rules in a database is defined as finding all the associa-
tion rules that hold with more than a user-given mini-
mum support threshold and a user-given minimum con-
fidence threshold. According to Agrawal, Imielinski and
Swami, this problem is solved in two steps:
1. Find all frequent itemsets in the database.
2. For each frequent itemset I, generate all the associa-
tion rules I'I\I', where I'I.
The second problem can be solved in a straightfor-
ward manner after the first step is completed. Hence,
the problem of mining association rules is reduced to
the problem of finding all frequent itemsets. This is not
a trivial problem, because the number of possible fre-
quent itemsets is equal to the size of the power set of I,
2 |I| .
Many algorithms are proposed in the literature, most
of them based on the Apriori mining method (Agrawal &
Srikant, 1994), which relies on a basic property of
frequent itemsets: All subsets of a frequent itemset are
frequent. This property also says that all supersets of an
infrequent itemset are infrequent. This approach works
well on weakly correlated data, such as market-basket
data. For overcorrelated data, such as census data, there
are other approaches, including Close (Pasquier, Bastide,
Taouil, & Lakhal, 1999), CHARM (Zaki & Hsiao, 1999)
and Closet (Pei, Han, & Mao, 2000), which are more
appropriate.
An interesting study of specific approaches is per-
formed in Zheng, Kohavi and Mason (2001), qualifying
CHARM as the most adjusted algorithm to real-world
data. Some later improvements of Closet are mentioned
in Wang, Han, and Pei (2003) concerning speed and
memory usage, while a different support-counting
method is proposed in Zaki and Gouda (2001). These
approaches search for closed itemsets structured in
lattices that are closely related with the concept lattice
in formal concept analysis (Ganter & Wille, 1999). The
main advantage of a closed itemset approach is the
smaller size of the resulting concept lattice versus the
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
number of frequent itemsets, that is, search space reduc-
tion.
In this article, I describe the closed-itemset ap-
proaches, considering the fact that an association rule
mining process leads to a large amount of results (most
of the time, that is) difficult to understand by the user.
I take into account the interactivity of the data-mining
process proposed by Ankerst (2001).
BACKGROUND
In this section I first describe the use of closed-itemset
lattices as the theoretical framework for the closed-
itemset approach. The application of Formal Concept
Analysis to the association rule problem was first men-
tioned in Zaki and Ogihara (1998). For more details on
lattice theory, see Ganter and Wille (1999).
The closed-itemset approach is described below. I
define a context (T, I, D), the Galois connection of a
context ((T, I, D), s, t), a concept of the context (X, Y),
and the set of concepts in the context, denoted (T, I, D).
The main result in the Formal Concept Analysis
theory is as follows:
Fundamental theorem of Formal Concept
Analysis (FCA): Let (T, I, D) be a context. Then
(T, I, D) is a complete lattice with join and meet
operators given by closed set intersection and
reunion operators.
How does FCA apply to the association rule prob-
lem? First, T is the set of transaction ids, I is the set of
items in the database, and D is the database itself.
Second, the mapping s associates to a transaction set X
the maximal itemset Y present in all transactions in X.
The mapping t associates to any itemset Y the maximal
transaction set X, where each transaction comprises all
the items in the itemset Y. The resulting frequent con-
cepts are considered in mining application only for their
itemset side, the transaction set side being ignored.
What is the advantage of FCA application? Among
the results in Apriori, there are itemsets I will call
them Y and Y, where Y is included in Y, and they have the
same support. These two itemsets are two distinct re-
sults of Apriori, even if they characterize differently the
TEAM LinG
Closed-Itemset Incremental-Mining Problem
same transaction set. In fact, the longest itemset is the
most precise characterization of that transaction set, all
the others being partial and redundant definitions. Due
to the observation that s t and ts are closure operators,
the concepts in the lattice of concepts eliminate the
presence of any unclosed itemsets. Under these cir-
cumstances, the FCA-based approaches have only
subunitary confidence association rules as results, due
to the fact that all unitary confidence association rules
are considered redundant behaviors. All unitary confi-
dence association rules can be expressed through a base,
the pseudo-intent set.
If I consider the data-mining process in the vision of
Ankerst (2001) the resulting data model in Apriori is a
long list of frequent itemsets, while FCA is a conceptual
structure, namely a lattice of concepts, free of any
redundancy.
MAIN THRUST
Many interesting algorithms are issued from the FCA
approach to the association rule problem. Although
many differences exist between the support counting,
memory usage and performance of all these algorithms,
the general lines are almost the same.
In the following sections, I take into account some
of the main characteristics of these algorithms.
Resulting Data Model
Most of the FCA approaches do not generate a lattice of
concepts but a spanning tree on that lattice. The argu-
ment is that some of the pairs of adjacent concepts in
lattice can be inferred later. The main algorithms that
follow this principle are CHARM and Closet.
One approach builds the entire lattice (Dumitriu,
2002), called ERA. The argument for building the entire
lattice resides in the fact that missing pairs of adjacent
concepts in the spanning tree-based approaches can
have an identical support count, thus transforming one
of the concepts in the pair in a nonconcept. In fact,
CHARM has a later stage of results checking from this
point of view. Another strong point of the ERA algo-
rithm is that it offers the pseudo-intents as results as
well, thus completely characterizing the data.
Itemset-Building Strategy
While Apriori is a breadth-first result-building algo-
rithm, most of the FCA-based algorithms are depth-first.
Only ERA has a different strategy: Each item in the
database is used to enlarge an already existing data
model built upon the previously selected items, thus
generating at all times a new and extended data model. C
This strategy generates results layer by layer, just like
an onion.
The main difference between the depth-first strategy
and the layer-based strategy is that interactivity is of-
fered to the user. Just like peeling an onion, one can take
a previously found data model, reduce it or enlarge it
with some items, and reach the data view that is the most
revealing to the individual.
In breadth-first as well as depth-first strategies, it is
impossible to provide interactivity to the user due to the
fact that all items of interest for the mining process have
to be available from the start.
FUTURE TRENDS
I am considering that the most challenging trends would
manifest in quantitative attribute mapping as well as in
online mining. The first case has a well-known problem
in what concerns the quality of numerical-to-Boolean
attribute mapping. Solutions to this problem are already
considered in Aumann and Lindell (1999), Hong, Kuo,
Chi, and Wang (2000), Imberman and Domanski (2001),
and Webb (2001), but the problem is far from resolu-
tion. An interactive association rule approach may help
in a generate-and-test paradigm to find the most suitable
mappings in a particular database context.
Online mining is very alike cognitive processes: A
data model is flooded with facts; either they are consis-
tent with the data model, contradict it, or have a neutral
character. When contradicting facts become important
(in number, frequency, or any other way), the model has
to change. The real problem of the data-mining process
is that the model is too large in number of concepts or
the concepts are too rigidly related to support change.
CONCLUSION
I have introduced the idea of data model, expressed as a
frequent closed-itemset lattice, with a base for global
implications, if needed. In the closed-itemset incre-
mental-mining solution, two new and important opera-
tions are applicable to data models: extension and re-
duction with several items. The main advantages of this
approach are:
The construction of small models of data, which
makes them more understandable for the user;
also, the response time is small
151
TEAM LinG

The extension of data models with a set of new items
returns to the user only the supplementary results,
hence a smaller amount of results; the response time
is considerably smaller than when building the model
from scratch
Whenever data models are incomprehensible, some
of the items can be removed, thus obtaining and
easy-to-understand data model
The extension or reduction of a model spares the
time spent building it, thus reusing knowledge
After many successful attempts to make it faster, the
mining process becomes more interactive and flexible
due to the increased number of human interventions.
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993, May).
Mining association rules between sets of items in large
databases. Proceedings of the ACM SIGMOD Confer-
ence on Management of Data (pp. 207-216), USA.
Agrawal, R., & Srikant, R. (1994, September). Fast algo-
rithms for mining association rules. Proceedings of the
20th International Conference on Very Large Data Bases
(pp. 487-499), Chile.
Ankerst, M. (2001, May). Human involvement and
interactivity of the next generations data mining tools.
Proceedings of the ACM SIGMOD Workshop on Re-
search Issues in Data Mining and Knowledge Discovery
(pp. 178-188), USA.
Aumann, Y., & Lindell, Y. (1999, August). A statistical
theory for quantitative association rules. Proceedings of
the International Conference on Knowledge Discov-
ery in Databases (pp. 261-270), USA.
Dumitriu, L. (2002). Interactive mining and knowledge
reuse for the closed-itemset incremental-mining prob-
lem. Newsletter of the ACM SIG on Knowledge Discov-
ery and Data Mining, 3(2), 28-36. Retrieved from http://
www.acm.org/sigs/sigkdd/explorations/issue3-2/
contents.htm
Ganter, B., & Wille, R. (1999). Formal concept analy-
sis Mathematical foundations. Berlin, Germany:
Springer-Verlag.
Hong, T.-P., Kuo, C.-S., Chi, S.-C., & Wang, S.-L. (2000,
March). Mining fuzzy rules from quantitative data based
152
Closed-Itemset Incremental-Mining Problem
on the AprioriTid algorithm. Proceedings of the ACM
Symposium on Applied Computing (pp. 534-536), Italy.
Imberman, S., & Domanski, B. (2001, August). Finding
association rules from quantitative data using data
Booleanization. Proceedings of the Seventh Americas
Conference on Information Systems, USA.
Pasquier, N., Bastide, Y., Taouil, R., & Lakhal, L. (1999,
January). Discovering frequent closed itemsets for asso-
ciation rules. Proceedings of the International Confer-
ence on Database Theory (pp. 398-416), Israel.
Pei, J., Han, J., & Mao, R. (2000, May). CLOSET: An
efficient algorithm for mining frequent closed itemsets.
Proceedings of the Conference on Data Mining and
Knowledge Discovery (pp. 11-20), USA.
Valtchev, P., Missaoui, R., Godin, R., & Meridji, M.
(2002). A framework for incremental generation of
frequent closed itemsets using Galois (concept) lat-
tice theory. Journal of Experimental and Theoretical
Artificial Intelligence, 14(2/3), 115-142.
Wang, J., Han, J., & Pei, J. (2003, August). CLOSET+:
Searching for the best strategies for mining frequent
closed itemsets. Proceedings of the ACM SIGKDD
International Conference on Knowledge Discovery
and Data Mining (pp. 236-245), USA.
Webb, G. I. (2001, August). Discovering associations
with numeric variables. Proceedings of the ACM
SIGKDD International Conference on Knowledge
Discovery and Data Mining (pp. 383-388), USA.
Zaki, M. J., & Gouda, K. (2001). Fast vertical mining
using diffsets (Tech. Rep. No. 01-1): Rensselaer Poly-
technic Institute, Department of Computer Science.
Zaki, M. J., & Hsiao, C. J. (1999). CHARM: An efficient
algorithm for closed association rule mining (Tech.
Rep. No. 99-10). Rensselaer Polytechnic Institute,
Department of Computer Science.
Zaki, M. J., & Ogihara, M. (1998, June). Theoretical
foundations of association rules. Proceedings of the
Third ACM SIGMOD Workshop on Research Issues in
Data Mining and Knowledge Discovery (pp. 7:1-7:8),
USA.
Zheng, Z., Kohavi, R., & Mason, L. (2001, August).
Real world performance of association rule algorithms.
Proceedings of the ACM SIGKDD International Con-
ference on Knowledge Discovery and Data Mining (pp.
401-406), USA.
TEAM LinG
Closed-Itemset Incremental-Mining Problem
KEY TERMS
Association Rule: A pair of frequent itemsets (A, B),
where the ratio between the support of AB and A itemsets
is greater than a predefined threshold, denoted minconf.
Closure Operator: Let S be a set and c: (S) (S);
c is a closure operator on S if X, Y S, c satisfies the
following properties:
1. extension, X c(X);
2. mononicity, if XY, then c(X) c(Y);
3. idempotency, c(c(X)) = c(X).
Note: st and ts are closure operators, when s and t
are the mappings in a Galois connection
Concept: In the Galois connection of the (T, I, D)
context, a concept is a pair (X, Y), X T, Y I, that satisfies
s(X)=Y and t(Y)=X. X is called the extent and Y the intent
of the concept (X,Y).
Context: A triple (T, I, D) where T and I are sets and
DTI. The elements of T are called objects, and the
elements of I are called attributes. For any t T and i I,
note tDi when t is related to i, that is, ( t, i) D.
Frequent Itemset: An itemset with support higher
than a predefined threshold, denoted minsup. C
Galois Connection: Let (T, I, D) be a context. Then the
mappings
s: (T) (I), s(X) = { i I | ( t X) tDi }
t: (I) (T), s(Y) = { t T | ( i Y) tDi }
define a Galois connection between (T) and (I), the
power sets of T and I, respectively.
Itemset: A set of items in a Boolean database D,
I={i1, i2, in}.
Itemset Support: The ratio between the number of
transactions in D comprising all the items in I and the
total number of transactions in D (support(I) = |{TiD|
( ijI) ij Ti }| / |D|).
Pseudo-Intent: The set X is a pseudo-intent if X
c(X), where c is a closure operator and for all pseudo-
intents Q X, c(Q) X.
153
TEAM LinG
154
Cluster Analysis in Fitting Mixtures of Curves
Tom Burr
Los Alamos National Laboratory, USA
INTRODUCTION
One data mining activity is cluster analysis, of which
there are several types. One type deserving special
attention is clustering that arises due to a mixture of
curves. A mixture distribution is a combination of two
or more distributions. For example, a bimodal distribu-
tion could be a mix with 30% of the values generated
from one unimodal distribution and 70% of the values
generated from a second unimodal distribution. The
special type of mixture we consider here is a mixture of
curves in a two-dimensional scatter plot. Imagine a
collection of hundreds or thousands of scatter plots,
each containing a few hundred points, including back-
ground noise, but also containing from zero to four or
five bands of points, each having a curved shape. In a
recent application (Burr et al., 2001), each curved band
of points was a potential thunderstorm event (see Figure
1), as observed from a distant satellite, and the goal was
to cluster the points into groups associated with thun-
derstorm events. Each curve has its own shape, length,
and location, with varying degrees of curve overlap,
point density, and noise magnitude. The scatter plots of
points from curves having small noise resemble a smooth
curve with very little vertical variation from the curve,
but there can be a wide range in noise magnitude so that
some events have large vertical variation from the cen-
ter of the band. In this context, each curve is a cluster and
the challenge is to use only the observations to estimate
how many curves comprise the mixture, plus their shapes
and locations. To achieve that goal, the human eye could
train a classifier by providing cluster labels to all points
in example scatter plots. Each point either would belong
to a curved region or to a catch-all noise category, and a
specialized cluster analysis would be used to develop an
approach for labeling (clustering) the points generated
according to the same mechanism in future scatter plots.
BACKGROUND
Two key features that distinguish various types of clus-
tering approaches are the assumed mechanism for how
the data is generated and the dimension of the data. The
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
data-generation mechanism includes deterministic and
stochastic components and often involves determinis-
tic mean shifts between clusters in high dimensions. But
there are other settings for cluster analysis. The particu-
lar one discussed here (see Figure 1) is a mixture of
curves, where any notion of a cluster mean would be
quite different from that in more typical clustering
applications. Furthermore, although finding clusters in
a two-dimensional scatter plot seems less challenging
than in higher-dimensions (the trained human eye is
likely to perform as well as any machine-automated
method, although the eye would be slower), complica-
tions include overlapping clusters; varying noise magni-
tude; varying feature and noise and density; varying
feature shape, locations, and length; and varying types of
noise (scene-wide and event-specific). Any one of these
complications would justify treating the fitting of curve
mixtures as an important special case of cluster analy-
sis. Although as in pattern recognition, the following
methods discussed require training scatter plots with
points labeled according to their cluster memberships,
we regard this as cluster analysis rather than pattern
recognition, because all scatter plots have from zero to
four or five clusters whose shape, length, location, and
extent of overlap with other clusters vary among scatter
plots. The training data can be used both to train cluster-
ing methods and then to judge their quality. Fitting
mixtures of curves is an important special case that has
received relatively little attention to date. Fitting mix-
tures of probability distributions dates to Titterington
et al. (1985), and several model-based clustering
schemes have been developed (Banfield & Raftery,
1993; Bensmail et al., 1997; Dasgupta & Raftery, 1998),
along with associated theory (Leroux, 1992). However,
these models assume that the mixture is a mixture of
probability distributions (often Gaussian, which can be
long and thin, ellipsoidal, or more circular) rather than
curves. More recently, methods for mixtures of curves
have been introduced, including a mixture of principal
curves model (Stanford & Raftery, 2000), a mixture of
regression models (Gaffney & Smyth 2003; Hurn, Justel,
& Robert, 2003; Turner, 2000), and mixtures of local
regression models (i.e., smooth curves obtained using
splines or nonparametric kernel smoothers for example).
TEAM LinG
Cluster Analysis in Fitting Mixtures of Curves

Figure 1. Four mixture examples containing (a) one, from another cluster. The noise points are identified
(b) one, (c) two, and (d) zero thunderstorm events plus initially as those having low local density (away from C
background noise. The label 1 is for the first the central portion of any cluster) but, during the ex-
thunderstorm in the scene, 2 for the second, and so trapolation, can be judged to be a cluster member, if they
forth., and the highest integer label is reserved for the lie near the extrapolated curve. To increase robustness,
catch-all noise class. Therefore, in (d), because the method 1 can be applied twice, each time using slightly
highest integer is 1, there is no thunderstorm present different inputs (such as the decision threshold for the
(the mixture is all noise). initial noise rejection and the criteria for accepting
points into a cluster that are close to the extrapolated
region of the clusters curve). Then, only clusters that
10

2 2 11 2
111
8

2
are identified both times are accepted.
2 222 2
2 11 1 1
2 22 1
1 111 11
8

1
1
Method 2 uses the minimized, integrated squared
2 2 2 2 11 1
6

11
2 22 2 1
1 11111
2 11111111 1 1 11 1111 1 1
2 1111111 111
6

1111 1
y

1 1 1 1
111111
error (ISE, or L 2 distance) (Scott, 2002; Scott &
11111111111111
1 1 111111
1
11 111
1111111
111 111 1
111
1 111111 11111
4

1 11111 111
111 1111 1111
111 1 1 2 2
1 11111111111 11
111
Szewczyk, 2002) and appears to be a good approach for
4

11111
111111 11111111 2
1 1 1 1111 1111111
111111
111
11111111 11
111111111 1 1 2
1 111 111111111111
111111111
1111 22
111111 11
2

1 111
11
11
111111 2 22
fitting mixture models, including mixtures of regres-
2 2
2 2 2
2

0 50 100 150 200 250 0 200 400 600 800

x
(a)
x
(b)
sion models, as is our focus here. Qualitatively, the
minimum L2 distance method tries to find the largest
portion of the data that matches the model. In our
10

33 1
11

333333
1
11
111 context, at each stage, the model is all the points belong-
6

1
8

3 3 33 1 1 1 1 11

ing to a single curve plus everything else. Therefore, we

3 1 1
3 1 1111 1
3 1 11 1 111111
1 11 1 111 1
6

1 111 1 1 1 11
4

1 1111 11111
first seek cluster 1 having the most points, regard the
y

3 1 1 111 1 1111
111 1111 1111
1 11
1111111111111
1
1 11111111 1111111111 1
4

3 3 1 1
11 1 1111
remaining points as noise, remove the cluster, then
1 11 11 1
1111111 1 1
2

2221 1 1
22 1111111111111 1 11 1
2

3 22222222222222 2

repeat the procedure in search of feature 2, and so on,

222222222222
2 1

until a stop criterion is reached. It also should be pos-

0 200 400 600 800 1000 0 200 400 600 800 1000
x x

sible to estimate the number of components in the

(c) (d)

mixture in the first evaluation of the data, but that

approach has not yet been attempted. Scott (2002) has
MAIN THRUST
shown that in the parametric setting with model f(x|q), we
^
Several methods have been proposed for fitting mix- estimate using = arg min [ f ( x | ) f ( x | 0 )]2 dx where
tures of curves. In method 1 (Burr et al., 2001), first use the true parameter O is unknown. It follows that a reason-
density estimation to reject the background noise points able estimator minimizing the parametric ISE criterion is
such as those labeled as 2 in Figure 1a. For example, 2 n
L2 E = arg min [ f ( x | )2 dx f ( xi | )] . This assumes that
each point in the scatter plot has a distance to its kth n i =1
nearest neighbor, which can be used as a local density the correct parametric family is used; the concept can be
estimate (Silverman, 1986) to reject noise points. Next, extended to include the case in which the assumed para-
use a distance measure that favors long, thin clusters metric form is incorrect in order to achieve robustness.
(e.g., let the distance between clusters be the minimum Method 3 (principal curve clustering with noise) was
distance between any a point in the first cluster and a developed by Stanford and Raftery (2000) to locate
point in the second cluster), together with standard principal curves in noisy spatial point process data.
hierarchical clustering to identify at least the central Principal curves were introduced by Hastie and Stuetzle
portion of each cluster. Alternatively, model-based clus- (1989). A principal curve is a smooth curvilinear sum-
tering favoring long, thin Gaussian shapes (Banfield & mary of p-dimensional data. It is a nonlinear generaliza-
Raftery, 1993) or the fitting straight lines method in tion of the first principal component line that uses a
Murtagh and Raftery (1984) or Campbell et al. (1997) local averaging method. Stanford and Raftery (2000)
are effective for finding the central portion of each developed an algorithm that first uses hierarchical prin-
cluster. A curve fitted to this central portion can be cipal curve clustering (HPCC, which is a hierarchical
extrapolated and then used to accept other points as and agglomerative clustering method) and next uses
members of the cluster. Because hierarchical cluster- iterative relocation (reassign points to new clusters)
ing cannot accommodate overlapping clusters, this based on the classification estimation-maximization
method assumes that the central portions of each cluster (CEM) algorithm. A probability model included the
are non-overlapping. Points away from the central por- principal curve probability model for the feature clus-
tion from one cluster that lie close to the curve fitted to ters and a homogeneous Poisson process model for the
the central portion of the cluster can overlap with points noise cluster. More specifically, let X denote the set of

155

TEAM LinG

observations, x1, x2, , xn, and C be a partition considering
of clusters C0, C 1, , C K, where the cluster Cj contains nj
points. The noise cluster is C0 and assumes feature points
are distributed uniformly along the true underlying feature
so that projections onto the features principal curve are
randomly drawn from a uniform U(0,j) distribution, where
j is the length of the jth curve. An approximation to the
probability for 0, 1, , 5 clusters is available from the
Bayesian Information Criterion (BIC), which is defined as
BIC = 2 log(L(X|) - M log(n), where L is the likelihood of the
data X, and M is the number of fitted parameters, so M =
K(DF + 2) + K + 1. For each of K features, we fit 2 parameters
( j and j defined below) and a curve having DF degrees of
freedom; there are K mixing proportions (j defined below),
and the estimate of scene area is used to estimate the noise
density. The likelihood L satisfies L(X|) =
n K
L( X | ) = L( xi | ) where L( xi | ) = j L( xi | , xi C j )
i =1 j =0
mixture likelihood (j is the probability that point i
belongs to feature j) and
|| xi f (ij ) ||2
L ( xi | , xi C j ) = (1/ j )(1/ 2 j ) exp(
2 2j
clusters ( || xi f (ij ) || is the Euclidean distance from xi to its
projection point f(ij) on curve j) and L( xi | , xi C j ) = 1/ Area
for the noise cluster. Space will not permit a complete
description of the HPCC-CEM method, but briefly, the
HPCC steps are as follows:
(1) Make an initial estimate of noise points and re-
move;
(2) form an initial clustering with at least seven points
in each cluster;
(3) fit a principal curve to each cluster; and
(4) calculate a clustering criterion
V = VAbout + aVAlong, where VAbout measures the orthogonal
distances to the curve (residual error sum of squares), and
VAlong measures the variance in arc length distances be-
tween projection points on the curve. Minimizing V (the
sum is over all clusters) will lead to clusters with regularly
spaced points along the curve and tightly grouped around
it. Large values of a will cause the method to avoid clusters
with gaps and small values of a favor thinner clusters.
Clustering (merging clusters) continues until V stops de-
creasing. The flexibility provided by such a clustering
criterion (avoid or allow gaps and avoid or favor thin
clusters) is potentially extremely useful, and Method 2 is
currently the only published method to include it.
Method 3 (Turner, 2000) uses the well-known EM
(estimation-maximization) algorithm (Dempster, Laird &
Rubin, 1977) to handle a mixture of regression models.
156
Cluster Analysis in Fitting Mixtures of Curves
Therefore, it is similar to method 3 in that a mixture model
is specified but differs in that the curve is fit using
parametric regression rather than principal curves, so
yi xi j
L ( xi | , xi C j ) = f ij = (1/ j ) ( ) where is the stan-
j
dard Gaussian distribution. Also, Turners implementa-
tion did not introduce a clustering criterion, but it did
attempt to estimate the number of clusters as follows.
Introduce indicator variable zi of which component of the
mixture generated observation yi and iteratively maximize
n K
Q = ik ln( f ik ) with respect to where is the complete
i =1 k =1
parameter set j for each class. The ik sat-
K
isfy ik = k fik / k fik . Then Q is maximized with respect to
i =1
is the t by weighted regression of yi on x1, , xn with weights
n n
ik, and each k2 is given by k = ik ( yi xi k ) / ik . In
2 2
i =1 i =1
practice, the components of come from the value of in
n
) for the feature
the previous iteration and k = (1/ n) ik . A difficulty in
i =1
choosing the number of components in the mixture, each
with unknown mixing probability, is that the likelihood
ratio statistic has an unknown distribution. Therefore,
Turner (2000) implemented a bootstrap strategy to choose
between 1 and 2 components, between 2 and 3, and so
forth. The strategy to choose between K and K + 1
components is (a) calculate the log-likelihood ratio
statistic Q for a model having K and for a model having
K + 1 components; (b) simulate data from the fitted K
-component model; (c) fit the K and K + 1 component
n
Q*; (d) compute the p-value for Q as p = 1/ n I (Q Q ) ,
*
i =1
where the indicator I( Q Q ) = 1 if Q Q* and 0*
otherwise.
To summarize, method 1 avoids the mathematics of
mixture fitting and seeks high-density regions to use as
a basis for extrapolation. It also checks the robustness
of its solution by repeating the fitting using different
input parameters and comparing results. Method 2 uses
a likelihood for one cluster and everything else, and
then removes the highest-density cluster and repeats
the procedure. Methods 3 and 4 both include formal
mixture fitting (known to be difficult), with method 3
assuming the curve is well modeled as a principal curve
and method 4 assuming the curve is well fit using para-
metric regression. Only method 3 allows the user to favor
or penalize clusters with gaps. All methods can be tuned
to accept only thin (i.e., small residual variance) clusters.
TEAM LinG
Cluster Analysis in Fitting Mixtures of Curves
FUTURE TRENDS
Although we focus here on curves in the two dimensions,
clearly there are analogous features in higher dimensions,
such as principal curves in higher dimensions. Fitting
mixtures in two dimensions is fairly straightforward, but
performance is rarely as good as desired. More choices for
probability distributions are needed; for example, the
noise in Burr et al. (2001) was a non-homogeneous Pois-
son process, which might be better handled with adaptively
chosen regions of the scene. Also, very little has been
published regarding the measure of difficulty of curve-
mixture problem, although Hurn, Justel, and Robert (2003)
provided a numerical Bayesian approach for mixtures
involving switching regression. The term switching re-
gression describes the situation in which the likelihood is
nearly unchanged when some of the cluster labels are
switched. For example, imagine that clusters 1 and 2 were
slightly closer in Figure 1c, and then consider changes to
the likelihood if we switched some 2 and 1 labels. Clearly,
if curves are not well separated, we should not expect high
clustering and classification success. In the case where
groups are separated by mean shifts, it is clear how to
define group separation. In the case where groups are
curves that overlap to varying degrees, one could envi-
sion a few options for defining group separation. For
example, the minimum distance between a point in one
cluster and a point in another cluster could define the
cluster separation and, therefore, also the measure of
difficulty. Depending on how we define the optimum, it is
possible that performance can approach the theoretical
optimum. We define performance first by determining
whether a cluster was found, and then by reporting the
false positive and false negative rates for each found
cluster.
Another area of valuable research would be to ac-
commodate mixtures of local regression models (i.e.,
smooth curves obtained using splines or nonparametric
kernel smoothers) (Green & Silverman, 1996). Be-
cause local curve smoothers such as splines are ex-
tremely successful in the case where the mixture is
known to contain only one curve, it is likely that the
flexibility provided by local regression (parametric or
nonparametric) would be desirable in the broader con-
text of fitting a mixture of curves.
Existing software is fairly accessible for the meth-
ods described, assuming users have access to a statisti-
cal programming language such as S-PLUS (2003).
Executable code to accommodate a user-specified input
format would, of course, also be a welcome future contri-
bution, but it is now reasonable to empirically compare
each method in each application of fitting mixtures of
curves.
CONCLUSION
C
Fitting mixtures of curves with noise in two dimensions
is a specialized type of cluster analysis. It is a manage-
able but fairly challenging and unique cluster analysis
task. Four options have been described briefly. Example
performance results for these methods all applied to the
same data for one application are available (Burr et al.,
2001). Results for methods 2, 3, and 4 also are available
in their respective references (on different data sets),
and results for a numerical Bayesian approach using a
mixture of regressions with attention to the case having
nearly overlapping clusters also has been published
recently (Hurn, Justel & Robert, 2003).
REFERENCES
Banfield, J., & Raftery, A. (1993). Model-based gaussian
and non-gaussian clustering. Biometrics, 49, 803-821.
Bensmail, H., Celeux, G., Raftery, A., & Robert, C.
(1997). Inference in model-based cluster analysis, Sta-
tistics and Computing, 7, 1-10.
Burr, T., Jacobson, A., & Mielke, A. (2001). Identifying
storms in noisy radio frequency data via satellite: An
application of density estimation and cluster analysis.
Proceedings of US Army Conference on Applied Sta-
tistics, Santa Fe, New Mexico.
Campbell, J., Fraley, C., Murtagh, F., & Raftery, A.
(1997). Linear flaw detection in woven textiles using
model-based clustering. Pattern Recognition Letters,
18, 1539-1548.
Dasgupta, A., & Raftery, A. (1998). Detecting features
in spatial point processes with clutter via model-based
clustering. Journal of American Statistical Associa-
tion, 93(441), 294-302.
Dempster, A., Laird, N., & Rubin, D. (1977). Maximum
likelihood for incomplete data via the EM algorithm. Jour-
nal of the Royal Statistical Society, Series B, 39, 1-38.
Gaffney, S., & Smyth, P. (2003). Curve clustering with
random effects regression mixtures. Proceedings of the
9th International Conference on AI and Statistics, Florida.
Green, P., & Silverman, B. (1994). Nonparametric regres-
sion and generalized linear models. London: Chapman
and Hall.
Hastie, T., & Stuetzle, W. (1989). Principal curves. Journal
of American Statistical Association, 84, 502-516.
157
TEAM LinG

Hurn, M., Justel, A., & Robert, C. (2003). Estimating
mixtures of regressions. Journal of Computational and
Graphical Statistics, 12, 55-74.
Leroux, B. (1992). Consistent estimation of a mixing
distribution. The Annals of Statistics, 20, 1350-1360.
Murtagh, F., & Raftery, A. (1984). Fitting straight lines
to point patterns. Pattern Recognition, 17, 479-483.
Scott, D. (2002). Parametric statistical modeling by
minimum integrated square error. Technometrics, 43(3),
274-285.
Scott, D., & Szewczyk, W. (2002). From kernels to
mixtures. Technometrics, 43(3), 323-335.
Silverman, B. (1986). Density estimation for statistics
and data analysis. London: Chapman and Hall.
S-PLUS Statistical Programming Language. (2003).
Seattle, WA: Insightful Corp.
Stanford, D., & Raftery, A. (2000). Finding curvilinear
features in spatial point patterns: Principal curve clus-
tering with noise. IEEE Transactions on Pattern Analy-
sis and Machine Intelligence, 22(6), 601-609.
Tiggerington, D., Smith, A., & Kakov, U. (1985). Statis-
tical analysis of finite mixture distributions. New
York: Wiley.
Turner, T. (2000). Estimating the propagation rate of a
viral infection of potato plants via mixtures of regres-
sions. Applied Statistics, 49(3), 371-384.
KEY TERMS
Bayesian Information Criterion: An approxima-
tion to the Bayes Factor, which can be used to estimate
the Bayesian posterior probability of a specified model.
Bootstrap: A resampling scheme in which surrogate
data is generated by resampling the original data or
sampling from a model that was fit to the original data.
158
Cluster Analysis in Fitting Mixtures of Curves
Cluster Analysis: Dividing objects into groups using
varying assumptions regarding the number of groups,
and the deterministic and stochastic mechanisms that
generate the observed values.
Estimation-Maximization Algorithm: An algorithm
for computing maximum likelihood estimates from incom-
plete data. In the case of fitting mixtures, the group labels
are the missing data.
Mixture of Distributions: A combination of two
or more distributions in which observations are gener-
ated from distribution i with probability pi and pi =1.
Poisson Process: A stochastic process for generat-
ing observations in which the number of observations in
a region (a region in space or time, for example) is
distributed as a Poisson random variable.
Principal Curve: A smooth, curvilinear summary
of p-dimensional data. It is a nonlinear generalization of
the first principal component line that uses a local
average of p-dimensional data.
Probability Density Function: A function that can
be summed (for discrete-valued random variables) or
integrated (for interval-valued random variables) to give
the probability of observing values in a specified set.
Probability Density Function Estimate: An esti-
mate of the probability density function. One example is
the histogram for densities that depend on one variable
(or multivariate histograms for multivariate densities).
However, the histogram has known deficiencies involv-
ing the arbitrary choice of bin width and locations.
Therefore, the preferred density function estimator,
which is a smoother estimator that uses local weighted
sums with weights determined by a smoothing param-
eter, is free from bin width and location artifacts.
Scatter Plot: One of the most common types of
plots, also known as an x-y plot, in which the first
component of a two-dimensional observation is dis-
played in the horizontal dimension and the second com-
ponent is displayed in the vertical dimension.
TEAM LinG

Clustering Analysis and Algorithms
Xiangji Huang
York University, Canada
INTRODUCTION
Clustering is the process of grouping a collection of
objects (usually represented as points in a multidimen-
sional space) into classes of similar objects. Cluster
analysis is a very important tool in data analysis. It is a
set of methodologies for automatic classification of a
collection of patterns into clusters based on similarity.
Intuitively, patterns within the same cluster are more
similar to each other than patterns belonging to a differ-
ent cluster. It is important to understand the difference
between clustering (unsupervised classification) and
supervised classification.
Cluster analysis has wide applications in data min-
ing, information retrieval, biology, medicine, market-
ing, and image segmentation. With the help of clustering
algorithms, a user is able to understand natural clusters
or structures underlying a data set. For example, clus-
tering can help marketers discover distinct groups and
characterize customer groups based on purchasing pat-
terns in business. In biology, it can be used to derive
plant and animal taxonomies, categorize genes with
similar functionality, and gain insight into structures
inherent in populations.
Typical pattern clustering activity involves the fol-
lowing steps: (1) pattern representation (including fea-
ture extraction and/or selection), (2) definition of a
pattern proximity measure appropriate to the data do-
main, (3) clustering, (4) data abstraction, and (5) as-
sessment of output.
BACKGROUND
General references regarding clustering include Hartigan
(1975), Jain and Dubes (1988), Kaufman and Rousseeuw
(1990), Mirkin (1996), Jain, Murty, and Flynn (1999),
and Ghosh (2002). A good introduction to contempo-
rary data-mining clustering techniques can be found in
Han and Kamber (2001). Early clustering methods be-
fore the 90s, such as k-means (Hartigan, 1975) and
PAM and CLARA (Kaufman & Rousseeuw, 1990), are
generally suitable for small data sets. CLARANS (Ng &
Han, 1994) made improvements to CLARA in quality
and scalability based on randomized search. After
CLARANS, many algorithms were proposed to deal with
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
159
C
large data sets, such as BIRCH (Zhang, Ramakrishnan, &
Livny, 1996), CURE (Guha, Rastogi, & Shim, 1998),
Squashing (DuMouchel, Volinsky, Johnson, Cortes, &
Pregibon, 1999) and Data Bubbles (Breuning, Kriegel,
Krger, & Sander, 2001).
MAIN THRUST
There exist a large number of clustering algorithms in
the literature. In general, major clustering algorithms
can be classified into the following categories.
Hierarchical Clustering
Hierarchical clustering builds a cluster hierarchy or a
tree of clusters, also known as a dendrogram. Every
cluster node contains child clusters; sibling clusters
partition the points covered by their common parent.
Such an approach allows the exploration of data on
different levels of granularity. Hierarchical clustering
can be further classified into agglomerative (bottom-
up) and divisive (top-down) hierarchical clustering. An
agglomerative clustering starts with one-point (single-
ton) clusters and recursively merges two or more most
appropriate clusters. A divisive clustering starts with
one cluster of all data points and recursively splits the
most appropriate cluster. The process continues until a
stopping criterion (for example, the requested number
of k clusters) is achieved. Advantages of hierarchical
clustering include (a) embedded flexibility regarding
the level of granularity, (b) ease of handling of any
forms of similarity or distance, and (c) applicability to
any attribute types. Disadvantages of hierarchical clus-
tering are (a) vagueness of termination criteria, and (b)
the fact that most hierarchical algorithms do not revisit
once-constructed clusters with the purpose of their
improvement.
One of the most striking developments in hierarchi-
cal clustering is the algorithm BIRCH. BIRCH creates a
height-balanced tree of nodes that summarize its zero,
first, and second moments. Guha et al. (1998) intro-
duced the hierarchical agglomerative clustering algo-
rithm called CURE (Clustering Using Representatives).
This algorithm has a number of novel features of general
significance. It takes special care with outliers and with
TEAM LinG

label assignment. Although CURE works with numeri-
cal attributes (particularly low-dimensional spatial data),
the algorithm ROCK, developed by the same research-
ers (Guha, Rastogi, & Shim, 1999) targets hierarchical
agglomerative clustering for categorical attributes.
Partitioning Clustering
Given a database of n objects and k, the number of
clusters to form, a partitioning algorithm organizes the
objects into k partitions (k n), where each partition
represents a cluster. The clusters are formed to opti-
mize a partitioning criterion, often called a similarity
function, such as distance, so that the objects within a
cluster are similar, whereas the objects of different
clusters are dissimilar in terms of the database at-
tributes.
Partitioning clustering algorithms have advantages
in applications involving large data sets for which the
construction of a dendrogram is computationally pro-
hibitive. A problem accompanying the use of a partition-
ing algorithm is the choice of the number of desired
output clusters. A seminal paper (Dubes, 1987) pro-
vides guidance on this key design decision. The parti-
tioning techniques usually produce clusters by optimiz-
ing a criterion function defined either locally (on a
subset of the patterns) or globally (defined over all the
patterns). Combinatorial search of the set of possible
labelings for an optimum value of a criterion is clearly
computationally prohibitive. In practice, the algorithm
is typically run multiple times with different starting
states, and the best configuration obtained from all the
runs is used as the output clustering. The most well-
known and commonly used partitioning algorithms are
k-means, k-medoids, and their variations.
K-Means Method
The k-means algorithm (Hartigan, 1975) is by far the
most popular clustering tool used in scientific and
industrial applications. It proceeds as follows. First, it
randomly selects k objects, each of which initially
represents a cluster mean or centre. For each of the
remaining objects, an object is assigned to the cluster to
which it is the most similar, based on the distance
between the object and the cluster mean. It then com-
putes the new mean for each cluster. This process iter-
ates until the criterion function converges. Typically,
the squared-error criterion is used, defined as
k
E = i=1 pC | p mi |2 sity-based algorithm that computes an augmented clus-
i
160
Clustering Analysis and Algorithms
where E is the sum of square-error for all objects in
the database, p is point in space representing a given
object, and mi is the mean of cluster Ci (both p and mi
are multidimensional).
K-Medoids Method
In the k-medoids algorithm, a cluster is represented by
one of its points. Instead of taking the mean value of the
objects in a cluster as a reference point, the medoid can
be used, which is the most centrally located object in a
cluster. The basic strategy of the k-medoids clustering
algorithms is to find k clusters in n objects by first
arbitrarily finding a representative object (the medoid)
for each cluster. Each remaining object is clustered
with the medoid to which it is the most similar. The
strategy then iteratively replaces one of the medoids by
one of the nonmedoids as long as the quality of the
resulting clustering is improved. This quality is esti-
mated by using a cost function that measures the average
dissimilarity between an object and the medoid of its
cluster. It is important to understand that k-means is a
greedy algorithm, but k-medoids is not.
Density-Based Clustering
Heuristic clustering algorithms (such as partitioning
methods) work well for finding spherical-shaped clus-
ters in databases that are not very large. To find clusters
with complex shapes and for clustering very large data
sets, partitioning-based algorithms need to be extended.
Most partitioning-based algorithms cluster objects based
on the distance between objects. Such methods can find
only spherical shaped clusters and encounter difficulty
at discovering clusters of arbitrary shapes. To discover
clusters with arbitrary shape, density-based clustering
algorithms have been developed. These algorithms typi-
cally regard clusters as dense regions of objects in the
data space that are separated by regions of low density.
The general idea is to continue growing the given
cluster as long as the density (number of objects or data
points) in the neighborhood exceeds some threshold.
That is, for each data point within a given cluster, the
neighborhood of a given radius has to contain at least a
minimum number of points. Such a method can be used
to filter out noise (outliers) and discover clusters of
arbitrary shape. DBSCAN (EsterKriegel, Sander, & Xu,
1996) is a typical density-based algorithm that grows
clusters according to a density threshold. OPTICS
(Ankerst Breuning, Kriegel, & Sander, 1999) is a den-
TEAM LinG
Clustering Analysis and Algorithms
ter ordering for automatic and interactive cluster analy-
sis. DENCLUE (Hinneburg & Keim, 1998) is another
clustering algorithm based on a set of density distribu-
tion functions. It differs from partition-based algorithms
not only by accepting arbitrary shape clusters but also by
how it handles noise.
Grid-Based Clustering
Grid-based algorithms quantize the object space into a
finite number of cells that form a grid structure on which
all the operations for clustering are performed. To some
extent, the grid-based methodology reflects a technical
point of view. The category is eclectic: It contains both
partitioning and hierarchical algorithms. The main ad-
vantage of this method is its fast processing time, which
is typically independent of the number of data objects,
yet dependent on only the number of cells in each dimen-
sion in the quantized space.
Some typical examples of the grid-based algorithms
include STING, which explores statistical information
stored in the grid cells; WaveCluster, which clusters
objects using a wavelet transform method; and CLIQUE,
which represents a grid and density-based approach for
clustering in a high-dimensional data space. The algo-
rithm STING (Wang, W., Wang, J., & Munta, 1997)
works with numerical attributes (spatial data) and is
designed to facilitate region-oriented queries. In doing
so, STING constructs data summaries in a way similar to
BIRCH. However, it assembles statistics in a hierarchi-
cal tree of nodes that are grid-cells. The algorithm
WaveCluster (Sheikholeslami, Chatterjee, & Zhang,
1998) works with numerical attributes and has an ad-
vanced multiresolution. It is also known for some out-
standing properties such as (a) a high quality of clusters,
(b) the ability to work well in relatively high-dimen-
sional spatial data, (c) the successful handling of outli-
ers, and (d) O(N) complexity. WaveCluster, which ap-
plies wavelet transforms to filter the data, is based on
ideas of signal processing. The algorithm CLIQUE
(Agrawal, Gehrke, Gunopulos, & Raghavan, 1998) for
numerical attributes is fundamental in subspace cluster-
ing. It combines the ideas of density-based clustering,
grid-based clustering, and the induction through dimen-
sions similar to the Apriori algorithm in association rule
learning.
Model-Based Clustering
Model-based clustering algorithms attempt to optimize
the fit between the given data and some mathematical
model. For example, clustering algorithms based on
probability models offer a principled alternative to heu-
ristic-based algorithms. In particular, the model-based
approach assumes that the data are generated by a finite
mixture of underlying probability distributions such as C
multivariate normal distributions. The Gaussian mix-
ture model has been shown to be a powerful tool for
many applications (Banfield & Raftery, 1993). With
the underlying probability model, the problems of de-
termining the number of clusters and of choosing an
appropriate clustering algorithm become probabilistic
model choice problems (Dasgupta & Raftery, 1998).
This provides a great advantage over heuristic cluster-
ing algorithms, for which no established method to
determine the number of clusters or the best clustering
algorithm exists. Model-based clustering follows two
major approaches: a probabilistic approach or a neural
network approach.
Probabilistic Approach
In the probabilistic approach, data are considered to be
samples independently drawn from a mixture model of
several probability distributions (McLachlan & Basford,
1988). The main assumption is that data points are
generated by (1) randomly picking a model j with
probability j and (2) drawing a point x from a corre-
sponding distribution. The area around the mean of
each distribution constitutes a natural cluster. We as-
sociate the cluster with the corresponding distributions
parameters such as mean, variance, and so forth. Each
data point carries not only its observable attributes, but
also a hidden cluster ID. Each point x is assumed to
belong to one and only one cluster.
Probabilistic clustering has some important fea-
tures. For example, it (a) can be modified to handle
records of complex structure, (b) can be stopped and
resumed with consecutive batches of data, and (c) re-
sults in easily interpretable cluster system. Because
the mixture model has a clear probabilistic foundation,
the determination of the most suitable number of clus-
ters k becomes a more tractable task. From a data-
mining perspective, an excessive parameter set causes
overfitting, but from a probabilistic perspective, the
number of parameters can be addressed within the
Bayesian framework. An important property of proba-
bilistic clustering is that the mixture model can be
naturally generalized to clustering heterogeneous data.
However, statistical mixture models often require a
quadratic space, and the EM algorithm converges rela-
tively slowly, making scalability an issue.
Artificial Neural Networks Approach
Artificial neural networks (ANNs) (Hertz, Krogh, &
Palmer, 1999) are inspired by biological neural net-
161
TEAM LinG

works. ANNs have been used extensively over the past
four decades for both classification and clustering (Jain
& Mao, 1994). The neural ANNs approach to clustering
has two prominent methods: competitive learning and
self-organizing feature maps. Both involve competing
neural units. Some of the features of the ANNs that are
important in pattern clustering are that they (a) process
numerical vectors and so require patterns to be repre-
sented with quantitative features only, (b) are inherently
parallel and distributed processing architectures, and
(c) may learn their interconnection weights adaptively.
The neural network approach to clustering tries to
emulate actual brain processing. Further research is
needed to make it readily applicable to very large data-
bases, due to long processing times and the intricacies
of complex data.
Other Clustering Techniques
Traditionally, each pattern belongs to one and only one
cluster. Hence, the clusters resulting from this kind of
clustering are disjoint. Fuzzy clustering extends this
notion to associate each object with every cluster using
a membership function (Zadeh, 1965). Another approach
is constrained-based clustering, introduced in (Tung, Ng,
Lakshmanan & Han, 2001). This approach has important
applications in clustering two-dimensional spatial data in
the presence of obstacles. Another approach used in
clustering analysis is the Genetic Algorithm (Goldbery,
1989). An example is the GGA (Genetically Guided Algo-
rithm) for fuzzy and hard k-means (Hall, Ozyurt, & Bezdek,
1999).
FUTURE TRENDS
Choosing a clustering algorithm for a particular prob-
lem can be a daunting task. One major challenge in using
a clustering algorithm on a specific problem lies not in
performing the clustering itself, but rather in choosing
the algorithm and the values of the associated param-
eters. Clustering algorithms also face problems of
scalability, both in terms of computing time and memory
requirements. Despite the ongoing exponential increases
in the power of computers, scalability remains still a
major issue in many clustering applications. In com-
mercial data-mining applications, the quantity of the
data to be clustered can far exceed the main memory
capacity of the computer, making both time and space
efficiency critical; this issue is addressed by clustering
systems in the database community such as BIRCH.
This leads to the following set of continuing re-
search in clustering, and in particular data mining. (1)
Extend clustering algorithms to handle very large data-
162
Clustering Analysis and Algorithms
bases, such as real-life terabyte data sets. (2) Gracefully
eliminate the need for a priori assumptions about the
data. (3) Use good sampling and data compression meth-
ods to improve efficiency and speed up clustering algo-
rithms. (4) Cluster extremely large and high-dimen-
sional data.
CONCLUSION
This article describes five major approaches to cluster-
ing, in addition to some other clustering algorithms.
Each has both positive and negative aspects, and each is
suitable for different types of data and different as-
sumptions about the cluster structure of the input. .
Clustering is a process of grouping data items based on
a measure of similarity. Clustering is also a subjective
process; the same set of data items often needs to be
partitioned differently for different applications. This
subjectivity makes the process of clustering difficult,
because a single algorithm or approach is not adequate
to solve every clustering problem. However, clustering
is an interesting, useful, and challenging problem. It has
great potential in applications such as object represen-
tation, image segmentation, information filtering and
retrieval, and analyzing gene expression data.
REFERENCES
Agrawal, R., Gehrke, J., Gunopulos, D., & Raghavan, P.
(1998). Automatic subspace clustering of high dimen-
sional data for data mining applications. Proceedings of
the ACM SIGMOD Conference (pp. 94-105), USA.
Ankerst, M., Breuning, M., Kriegel, H., & Sander, J. (1999).
OPTICS: Ordering points to identify clustering structure.
Proceedings of the ACM SIGMOD Conference (pp. 49-
60), USA.
Banfield, J. D., & Raftery, A. E. (1993). Model-based
Gaussian and non-Gaussian clustering. Biometrics, 49,
803-821.
Breuning, M., Kriegel, H., Krger, H., & Sander, J. (2001).
Data bubbles: Quality preserving performance boosting
for hierarchical clustering. Proceedings of the ACM
SIGMOD Conference, USA.
Dasgupta, A., & Raftery, A. E. (1998). Detecting features
in spatial point processes with clutter via model-based
clustering. Journal of the American Statistical Associa-
tion, 93, 294-302.
Dubes, R. C. (1987). How many clusters are best? An
experiment. Pattern Recognition, 20, 645-663.
TEAM LinG
Clustering Analysis and Algorithms
DuMouchel, W., Volinsky, C., Johnson, T., Cortes, C., &
Pregibon, D. (1999). Squashing flat files flatter. Proceed-
ings of the ACM SIGKDD Conference (pp. 6-15), USA.
Ester, M., Kriegel, H., Sander, J., & Xu, X. (1996). A
density-based algorithm for discovering clustering in
large spatial databases with noise. Proceedings of the
the ACM SIGKDD Conference (pp. 226-231), USA.
Ghosh, J. (2002). Scalable clustering methods for data
mining. In N. Ye (Ed.), Handbook of data mining
Lawrence Erlbaum.
Goldbery, D. (1989). Genetic algorithm in search,
optimization and machine learning. Addison-Wesley.
Guha, S., Rastogi, R., & Shim, K. (1998). CURE: An
efficient clustering algorithm for large databases. Pro-
ceedings of the ACM SIGMOD Conference (pp. 73-84),
USA.
Guha, S., Rastogi, R., & Shim, K. (1999). ROCK: A robust
clustering algorithm for categorical attributes. Proceed-
ings of the 15th International Conference on Data Engi-
neering (pp. 512-521), Australia.
Hall, L. O., Ozyurt, B., & Bezdek, J. C. (1999). Clustering
with a genetically optimized approach. IEEE Transac-
tions on Evolutionary Computation, 3(2), 103-112.
Han, J., & Kamber, M. (2001). Data mining: Concepts and
techniques. Morgan Kaufmann.
Hartigan, J. (1975). Clustering algorithms. New York:
Wiley.
Hertz, J., Krogh, A., & Palmer, R. G. (1991). Introduc-
tion to the theory of neural computation. Reading,
MA: Addison Wesley Longman.
Hinneburg, A., & Keim, D. (1998). An efficient ap-
proach to clustering large multimedia databases with
noise. Proceedings of the ACM SIGMOD Conference (pp.
58-65), USA.
Jain, A., & Dubes, R. (1988). Algorithms for clustering
data. Englewood Cliffs, NJ: Prentice-Hall.
Jain, A., & Mao, J. (1994). Neural networks and pattern
recognition. In J. M. Zurada, R. J. Marks, & C. J.
Robinson (Eds.), Computational intelligence: Imitat-
ing life (pp.194-212).
Jain, A., Murty, M., & Flynn, P. (1999). Data clustering: A
review. ACM Computing Surveys, 31(3), 264-323.
Kaufman, L., & Rousseeuw, P. (1990). Finding groups in
data: An introduction to cluster analysis. New York:
Wiley.
Lu, S. Y., & Fu, K. S. (1978). A sentence-to-sentence
clustering procedure for pattern analysis. IEEE Transac- C
tions on System Man Cybern., 8, 381-389.
McLachlan, G. J., & Basford, K. D. (1998). Mixture models:
Inference and application to clustering. New York:
Dekker.
Mirkin, B. (1996). Mathematic classification and cluster-
ing. Kluwer Academic.
Ng, R., & Han, J. (1994). Efficient and effective clustering
method for spatial data mining. Proceedings of the 20th
Conference on Very Large Data Bases (pp. 144-155),
Chile.
Sheikholeslami, G., Chatterjee, S., & Zhang, A. (1998).
WaveCluster: A multi-resolution clustering approach
for very large spatial databases. Proceedings of the
24th Conference on Very Large Data Bases (pp. 428-439),
USA.
Tung, A. K. H., Hou, J., & Han, J. (2001). Spatial clustering
in the presence of obstacles. Proceedings of the 17th
International Conference on Data Engineering (pp. 359-
367), Germany.
Tung, A. K. H., Ng, R. T., Lakshmanan, L. V. S., & Han, J.
(2001). Constraint-based clustering in large databases.
Proceedings of the Eighth ICDT, London.
Wang, W., Wang, J., & Munta, R. (1997). STING: A
statistical information grid approach to spatial data
mining. Proceedings of the 23rd Conference on Very
Large Data Bases (pp. 186-195), Greece.
Zadeh, L. H. (1965). Fuzzy sets. Information Control, 8,
338-353.
Zhang, T., Ramakrishnan, R., & Livny, M. (1996). BIRCH:
An efficient data clustering method for very large data-
bases. Proceedings of the ACM SIGMOD Conference
(pp. 103-114), Canada.
KEY TERMS
Apriori Algorithm: An efficient association rule
mining algorithm developed by Agrawal, in 1993. Apriori
employs a breadth-first search and uses a hash tree
structure to count candidate item sets efficiently. The
algorithm generates candidate item sets of length k
from k1 length item sets. Then, the patterns that have an
infrequent subpattern are pruned. Following that, the
whole transaction database is scanned to determine
frequent item sets among the candidates. For determining
163
TEAM LinG

frequent items in a fast manner, the algorithm uses a hash
tree to store candidate item sets.
Association Rule: A rule in the form of if this, then
that. It states a statistical correlation between the occur-
rence of certain attributes in a database.
Customer Relationship Management: The process
by which companies manage their interactions with cus-
tomers.
Data Mining: The process of efficient discovery of
actionable and valuable patterns from large databases.
164
Clustering Analysis and Algorithms
Feature Selection: The process of identifying the
most effective subset of the original features to use in
data analysis, such as clustering.
Overfitting: The effect on data analysis, data min-
ing, and biological learning of training too closely on
limited available data and building models that do not
generalize well to new unseen data.
Supervised Classification: Given a collection of
labeled patterns, the problem in supervised classifica-
tion is to label a newly encountered but unlabeled pat-
tern. Typically, the given labeled patterns are used to
learn the descriptions of classes that in turn are used to
label a new pattern.
TEAM LinG

Clustering in the Identification of Space Models
Maribel Yasmina Santos
University of Minho, Portugal
Adriano Moreira
University of Minho, Portugal
Sofia Carneiro
University of Minho, Portugal
INTRODUCTION
Clustering is the process of grouping a set of objects into
clusters so that objects within a cluster have high similar-
ity with each other, but are as dissimilar as possible to
objects in other clusters. Dissimilarities are measured
based on the attribute values describing the objects (Han
& Kamber, 2001).
Clustering, as a data mining technique (Cios, Pedrycz,
& Swiniarski, 1998; Groth, 2000), has been widely used to
find groups of customers with similar behavior or groups
of items that are bought together, allowing the identifica-
tion of the clients profile (Berry & Linoff, 2000). This
article presents another use of clustering, namely in the
creation of Space Models.
Space Models represent divisions of the geographic
space in which the several geographic regions are grouped
accordingly to their similarities with respect to a specific
indicator (values of an attribute). Space Models represent
natural divisions of the geographic space based on some
geo-referenced data.
This article addresses the development of a clustering
algorithm for the creation of Space Models STICH
(Space Models Identification Through Hierarchical Clus-
tering). The Space Models identified, integrating several
clusters, point out particularities of the analyzed data,
namely the exhibition of clusters with outliers, regions
which behavior is strongly different from the other ana-
lyzed regions.
In the work described in this article, some assumptions
were adopted to the creation of Space Models through a
clustering process, namely:
Space Models must be created by looking at the data
values available and no constraints must be im-
posed for their identification.
Space Models can include clusters of different
shapes and sizes.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
165
C
Space Models are independent of specific domain
knowledge, like the specification of the final number
of clusters.
The following sections, in outline, include: (i) an
overview of clustering, its methods and techniques; (ii)
the STICH algorithm, its assumptions, its implementation
and the results for a sample dataset; (iii) future trends; and
(iv) a conclusion with some comments about the pro-
posed algorithm.
BACKGROUND
Clustering (Grabmeier, 2002; Jain, Murty, & Flynn, 1999;
Zat & Messatfa, 1997) is a discovering process (Fayyad
& Stolorz, 1997) that identifies homogeneous groups of
segments in a dataset.
Han & Kamber (2001) state that clustering is a chal-
lenging field of research integrating a set of special
requirements. Some of the typical requirements of cluster-
ing in data mining are:
Scalability in order to allow the analysis of large
datasets, since clustering on a sample of a database
may lead to biased results.
Discovery of clusters with arbitrary shapes since
clusters can be of any shape. Some existing cluster-
ing algorithms identify clusters that tend to be
spherical, with similar size and density.
Minimal domain knowledge since the clustering
results can be quite sensitive to the input param-
eters, like the number of clusters required by many
clustering algorithms.
Ability to deal with noisy data avoiding the iden-
tification of clusters that were negatively influ-
enced by outliers or erroneous data.
Insensitivity to the order of input records since
there are clustering algorithms that are influenced
TEAM LinG

by the order in which the available records are
analyzed.
Two of the well-known types of clustering algorithms
are based on partitioning and hierarchical methods. These
methods and some of the corresponding algorithms are
presented in the following subsections.
Clustering Methods
Partitioning Methods
A partitioning method constructs a partition of n objects
into k clusters where k n. Given k, the partitioning
method creates an initial partitioning and then, using an
iterative relocation technique, it attempts to improve the
partitioning by moving objects from one cluster to an-
other. The clusters are formed to optimize an objective-
partitioning criterion, such as the distance between the
objects. A good partitioning must aggregate objects such
that objects in the same cluster are similar to each other,
whereas objects in different clusters are very different
(Han & Kamber, 2001).
Two of the well-known partitioning clustering algo-
rithms are the k-means algorithm, where each cluster is
represented by the mean value of the objects in the
cluster, and the k-medoid algorithm, where each cluster is
represented by one of the objects located near the centre
of the cluster.
Hierarchical Methods
Hierarchical methods perform a hierarchical composition
of a given set of data objects. It can be done bottom-up
(Agglomerative Hierarchical methods) or top-down (Divi-
sive Hierarchical methods). Agglomerative methods start
with each object forming a separate cluster. Then they
perform repeated merges of clusters or groups close to
one another until all the groups are merged into one
cluster or until some pre-defined threshold is reached
(Han & Kamber, 2001). These algorithms are based on the
inter-object distances and on finding the nearest neigh-
bors objects. Divisive methods start with all the objects
in a single cluster and, in each successive iteration, the
clusters are divided into smaller clusters until each cluster
contains only one object or a termination condition is
verified.
The next subsection presents two examples of cluster-
ing algorithms associated with partitioning and hierarchi-
cal methods respectively.
166
Clustering in the Identification of Space Models
Clustering Techniques
The K-Means Algorithm
Partitioning-based clustering algorithms such as k-means
attempt to break data into a set of k clusters (Karypis, Han,
& Kumar, 1999).
The k-means algorithm takes as input a parameter k
that represents the number of clusters in which the n
objects of a dataset will be partitioned. The obtained
division tries to maximize the Intracluster similarity (a
measurement of the similarity between the objects inside
a cluster) and minimize the Intercluster similarity (a mea-
surement of the similarity between different clusters).
That is to say a high similarity between the objects inside
a cluster and a low similarity between objects in different
clusters. This similarity is measured looking at the centers
of gravity (centroids) of the clusters, which are calculated
as the mean value of the objects inside them.
Given the input parameter k, the k-means algorithm
works as follows (MacQueen, 1967):
1. Randomly selects k objects, each of which initially
represents the cluster centre or the cluster mean.
2. Assign each of the remaining objects to the cluster
to which it is the most similar, based on the distance
between the object and the cluster mean.
3. Compute the new mean (centroid) of each cluster.
After the first iteration, each cluster is represented by
the mean calculated in step 3. This process is repeated
until the criterion function converges. The squared-error
criterion is often used, which is defined as:
k l
E = o j Ci o j mi
2
(1)
i =1 j =1
where E is the sum of the square-error for the objects in
the dada set, l is the number of objects in a given cluster,
oj represents an object, and mi is the mean value of the
cluster Ci. This criterion intends to make the resulting k
clusters as compact and as separate as possible (Han &
Kamber, 2001).
The k-means algorithm is applied when the mean of a
cluster can be obtained, which makes it not suitable for the
analysis of categorical attributes1. One of the disadvan-
tages that can be pointed out to this method is the
necessity for users to specify k in advance. This method
is also not suitable for discovering clusters with nonconvex
shapes or clusters of very different sizes (Han & Kamber,
TEAM LinG
Clustering in the Identification of Space Models
2001). To overcome some of the drawbacks of the k-means
algorithm, several improvements and new developments
have been done. See for example (Likas, Vlassis, & Verbeek,
2003; Estivill-Castro & Yang, 2004).
The K-Nearest Neighbor Algorithm
Hierarchical clustering algorithms generate a set of clus-
ters with a single cluster at the top (integrating all data
objects) and single-point clusters at the bottom, in which
each cluster is formed by one data object (Karypis, Han, &
Kumar, 1999).
Agglomerative hierarchical algorithms, like k-nearest
neighbor, start with each data object in a separate cluster.
Each step of the clustering algorithm merges the k records
that are most similar into a single cluster.
The k-nearest neighbor algorithm2 uses a graph to
represent the links between the several data objects. This
graph allows the identification of the several k objects
most similar to a given data item. Each node of the k-
nearest neighbor graph represents a data item. An edge
exists between two nodes p and q if q is among the k most
similar objects of p, or if p is among the k most similar
objects of q (Figure 1).
MAIN THRUST
The Space Models Identification Through Hierarchical
Clustering (STICH) algorithm presented in this article is
based on the k-nearest neighbor algorithm. However, the
principles defined for STICH try to overcome some of the
limitations of the k-nearest neighbor approach, namely
the necessity to define a value for the input parameter k.
This value imposes restrictions to the maximum number of
members that a given cluster can have. The number of
members of the obtained clusters is k+1 (except in the case
of tie in the neighbors value). In some applications there
are no objective reasons for the choice of a value for k. The
k-means algorithm was also analyzed mainly because of its
simplicity, and to allow the comparison of results with
STICH.
Figure 1. K-nearest neighbor algorithm
25
72 99
38
45
124
87
65
50
a) Dataset
The Algorithm
C
STICH uses an iterative process, in which no input
parameters are required. Another characteristic of STICH
is that it produces several usable Space Models, one at
the end of each iteration of the clustering process.
As a hierarchical clustering algorithm with an
agglomerative approach, STICH starts to assign each
object in the dataset to a different cluster. The clustering
process begins with as many clusters as objects in the
dataset, and ends with all the objects grouped into the
same cluster.
The approach undertaken in STICH is as follows:
1. For the dataset, calculate the Similarity Matrix of
the objects, which is the matrix of the Euclidean
distances between every pair of objects in the
dataset.
2. For each object, identify its minimum distance to
the dataset (the value that represents the distance
between the object and its 1-nearest neighbor).
3. Calculate the median3 of all the minimum distances
identified in the previous step.
4. For each object, identify its k-nearest neighbors,
selecting the objects that have a distance value
less or equal to the median calculated in step 3. The
number of objects selected, k, may vary from one
object to another.
5. For each object, calculate the average c of its k-
nearest neighbors.
6. For each object, verify in which clusters it appears
as one of the k-nearest neighbors and then assign
this object to the cluster in which it appears with
the minimum k-nearest neighbor average c. In case
of tie, one object that appears into two clusters with
equal average c, the object is assigned to the first
cluster in which it appears in the Similarity Matrix.
7. For each new cluster, calculate its centroid.
This process is iteratively repeated until all the ob-
jects are grouped into the same cluster. Each iteration of
STICH produces a new Space Model, with decreasing
25
99
72
38
45
124
87
65
50
b) 2-nearest neighbor
167
TEAM LinG

number of clusters, which can be used for different pur-
poses.
For each model, a quality metric is calculated after each
iteration of the STICH algorithm. This metric, the
ModelQuality, is defined as:
ModelQuality = Intraclust er Intercluster (2)
and is based on the difference between the Intracluster
and Intercluster similarities.
The Intracluster indicator is calculated as the sum of
all distances between the several objects in a given cluster
(l represents the total number of objects in a cluster) and
the mean value (mi) of the cluster (Ci) in which the object
(oj) reside. The total number of clusters identified in each
iteration is represented by t. The Intracluster indicator is
calculated as follows:
t l
Intracluster = o j Ci o j mi (3)
i =1 j =1
The Intercluster indicator is calculated as the sum of
all distances existing between the centers of all the clus-
ters identified in a given iteration. The Intercluster indi-
cator is calculated as follows:
t dissolving the borders of its members (the regions inside
t
Intercluster = mi m j (4)
i =1 j =1
j i
The ModelQuality metric can be used by the user in
the selection of the appropriate Space Model for a given
task. However, it is on the minimum value of this metric
that we found the STICH outliers model, in the case they
exist in the dataset (Figure 2). In the beginning of the
Figure 2. Intracluster similarity vs. Intercluster similarity
1 109
8 108
6 108
4 108
2 108
2 4
168
Clustering in the Identification of Space Models
clustering process the Intracluster similarity value is low,
because of the high similarity between the objects inside
the clusters, and the Intercluster similarity value is high,
because of the low similarity between the several clusters.
As the process proceeds, the Intracluster value increases
and the Intercluster value decreases. The minimum differ-
ence between these two values means that the objects are
as separate as possible, considering a low number of
clusters. After this point, the clustering process forces
the aggregation of all the objects into the same cluster (the
stop criterion of agglomerative hierarchical clustering
methods), within a few more iterations.
At this minimum point (Figure 2), reached at some
iteration of the clustering process, the resulting Space
Model is the one where the outliers are pointed out, that
is, where the model isolates in different clusters the
regions that are very different from all other regions.
The Implementation
STICH was implemented in Visual Basic for Applications
(VBA) and integrated in the Geographic Information Sys-
tem (GIS) ArcView 8.2 using ArcObjects4. For the creation
of Space Models, STICH considers two processes: the
clustering of geographic regions based on a selected
indicator, and the creation of a new space geometry where
a new polygon is created for each resulting cluster by
it). By using ArcGIS, both processes can be implemented
in the same platform.
The Results
The Space Model presented afterward was obtained ana-
lyzing an indicator available in the EPSILON database.
EPSILON (Environmental Policy via Sustainability Indi-
Intracluster similarity
Intercluster similarity
Diference
Min
Iteration
6 8 10
TEAM LinG
Clustering in the Identification of Space Models

cators on a European-wide NUTS III Level) is a research
project founded by the European Union through the
Information Society Technologies program. STICH is a
deliverable of this research project, which contributes to
the better understanding of the European Environmental
Figure 3. Space model: Analysis of the heavy metals
lead attribute
Quality and Quality of Life, by delivering a tool aimed to
generate environmental sustainability indices at NUTS5- C
III level.
In the following example, an indicator collected in the
EPSILON project is analyzed, namely the concentration in
the air of Heavy Metals - Lead (the data is available in
Table 1 for the 15 European countries that integrate the
EPSILON database).
Using the k-means and STICH algorithms, adopting
the value k=3 in order to allow the comparison between
the clusters achieved by them, the results obtained are
systematized in Table 2. Analyzing these results it is
possible to see that the clusters generated with the k-
means algorithm, using the implementation available in
the Clementine Data Mining System v8.0, are not as
homogeneous as the clusters obtained with STICH. The
k-means approach integrates the 0.0231413 value, an
outlier6 present in the dataset, with values like 0.00197921,
leaving out of this cluster (Cluster 1) values that are more
proximal to this last one. STICH obtains clusters that are
more homogeneous and that separates values that are
very different from the others.
Figure 3 shows the Space Model created by STICH as
result of the clustering process. This model, with three
clusters, shows the spatial distribution of the concentra-
tion of Lead in the air, for the 15 analyzed countries.

Table 1. Data available in the Epsilon database for the Heavy Metals - Lead attribute

Country Sweden Finland Ireland Denmark France

Indicator 0.00197921 0.00207107 0.0032313 0.0035931 0.00482136

Country England Austria Luxemburg Spain Netherlands

Indicator 0.00512893 0.00560728 0.00638133 0.00797498 0.00879874

Country Germany Belgium Greece Portugal Italy

Indicator 0.0091856 0.00931672 0.0114614 0.0124667 0.0231413

Table 2. Results obtained for the Heavy Metals - Lead attributes

Cluster 1 = {0.00197921, 0.0035931, 0.00482136, 0.00512893, 0.00560728,

0.00638133, 0.00797498, 0.00879874, 0.0091856, 0.00931672,
0.0114614, 0.0124667, 0.0231413}

k-means Cluster 2 = {0.00207107}

Cluster 3 = {0.0032313}

Cluster 1 = {0.00197921, 0.00207107, 0.0032313, 0.0035931, 0.00482136,

0.00512893, 0.00560728, 0.00638133}

STICH Cluster 2 = {0.00797498, 0.00879874, 0.0091856, 0.00931672, 0.0114614,

0.0124667}

Cluster 3 = {0.0231413}

169

TEAM LinG

As advantages for the creation of Space Models it can
be pointed out the definition of new space geometries,
which can be reused whenever needed, and the certainty
that the new divisions of the space were obtained by
characteristics present in data, and not imposed by human
constraints as in administrative subdivisions.
FUTURE TRENDS
As already mentioned, clustering has been widely used in
several domain applications to find segments in data, like
groups of clients with similar behavior. In the location-
based services domain, clustering can also be used to find
groups of mobile users with the same interests or with
similar mobility patterns. In this context, the STICH algo-
rithm can also be used to create Space Models as divi-
sions of the geographic space where each region repre-
sents an area with certain attributes useful to characterize
the situation of a mobile user. Examples of such regions
are urban or rural areas, commercial or leisure zones, or not
safety zones in a town. When a mobile user is detected to
be inside one of these regions, the attributes of that region
can be used to identify the appropriate services and
information to provide to him. These attributes can also
be combined with other contextual data, such as the time
of the day or the user preferences, to enhance the adapt-
ability of a context-aware application (Salber, Dey, &
Abowd, 1999).
CONCLUSION
This article presented STICH, a hierarchical clustering
algorithm that allows the identification of Space Models.
These models are characterized by the integration of
groups of regions that are similar to each other. Clusters
of outliers are also formed by STICH, enabling the iden-
tification of regions that are very different from the other
regions present in the dataset.
As advantages of STICH, and besides the creation of
new divisions of the space, we point out:
The discovery of clusters with arbitrary shapes and
sizes.
The avoidance of any previous domain knowledge,
as for example the definition of a value for k.
The ability to deal with outliers values, since when
they are present in the dataset, the approach under-
taken allows their identification.
170
Clustering in the Identification of Space Models
REFERENCES
Berry, M., & Linoff, G. (2000). Mastering data mining: The
art and science of customer relationship management.
New York: John Wiley and Sons, Inc.
Bhm, C., & Krebs, F. (2004). The k-nearest neighbor join:
Turbo charging the KDD process. Knowledge and Infor-
mation Systems, 6(6).
Cios, K., Pedrycz, W., & Swiniarski, R. (1998). Data mining
methods for knowledge discovery. Boston: Kluwer Aca-
demic Publishers.
Estivill-Castro, V., & Yang, J. (2004). Fast and robust
general purpose clustering algorithms. Data Mining and
Knowledge Discovery, 8(2), 127-150.
Fayyad, U., & Stolorz, P. (1997). Data mining and KDD:
Promise and challenges. Future Generation Computer
Systems, 13(2), 99-115.
Grabmeier, J. (2002). Techniques of cluster algorithms
in data mining. Data Mining and Knowledge Discovery,
6(4), 303-360.
Groth, R. (2000). Data mining: Building competitive ad-
vantage. Prentice Hall PTR.
Han, J., & Kamber, M. (2001). Data mining: Concepts and
techniques. CA: Morgan Kaufmann Publishers.
Jain, A.K., Murty, M.N., & Flynn, P.J. (1999). Data cluster-
ing: A review. ACM Computing Surveys, 31(3), 264-323.
Karypis, G., Han, E.-H., & Kumar, V. (1999). Chameleon:
Hierarchical clustering using dynamic modeling. IEEE
Computer, 32(8), 68-75.
Laurikkala, J., Juhola, M., & Kentala, E. (2000). Informal
identification of outliers in medical data. In Proceedings
of Intelligent Data Analysis in Medicine and Pharmacol-
ogy, Workshop at the 14th European Conference on
Artificial Intelligence (pp. 20-24). Berlin.
Likas, A., Vlassis, N., & Verbeek, J. J. (2003). The global
k-means clustering algorithm. Pattern Recognition, 36(2),
451-461.
MacQueen, J.B. (1967). Some methods for classification
and analysis of multivariate observations. In Proceed-
ings of 5-th Berkeley Symposium on Mathematical Sta-
tistics and Probability (pp. 281-297). Berkeley, CA: Uni-
versity of California Press.
Modha, D.S., & Spangler, W.S. (2003). Feature weighting
in k-Means clustering. Machine Learning, 52(3), 217-237.
TEAM LinG
Clustering in the Identification of Space Models
Salber, D., Dey, A.K., & Abowd, G.D. (1999). The context
toolkit: Aiding the development of context-enabled appli-
cations. In Proceedings of the 1999 Conference on Hu-
man Factors in Computing Systems (CHI 99) (pp. 434-
441). Pittsburgh.
Zat, M., & Messatfa, H. (1997). A comparative study of
clustering methods. Future Generation Computer Sys-
tems, 13(2), 149-159.
KEY TERMS
Cluster: Group of objects of a dataset that are similar
with respect to specific metrics (values of the attributes
used to identify the similarity between the objects).
Clustering: Process of identifying homogeneous
groups of clusters in a dataset.
Data Mining: A discovering process that aims the
identification of patterns hidden in the analyzed dataset.
Hierarchical Clustering: A clustering method char-
acterized by the successive aggregation (agglomerative
hierarchical methods) or desegregation (divisive hierar-
chical methods) of objects in order to find clusters in a
dataset.
Intercluster Similarity: A measurement of the simi-
larity between the clusters identified in a particular clus-
tering process. These clusters must be as dissimilar as
possible.
Intracluster Similarity: A measurement of the simi-
larity between the objects inside a cluster. Objects in a
cluster must be as similar as possible.
Partitioning Clustering: A clustering method char-
acterized by the division of the initial dataset in order to C
find clusters that maximize the similarity between the
objects inside the clusters.
Space Model: A geometry of the geographic space
obtained by the identification of geographic regions with
similar behavior looking at a specific metric.
ENDNOTES
1
For datasets with multiple, heterogeneous feature
spaces see Modha & Spangler (2003).
2
The k-nearest neighbor principles are also used for
classification, a data mining task (Bhm & Krebs,
2004).
3
The median is used instead of the average, since the
average can be negatively influenced by outliers.
4
http://arcobjectsonline.esri.com
5
Nomenclature of Territorial Units for Statistics.
6
Following the Interquartile Range definition
(Laurikkala, Juhola, & Kentala, 2000) for the identi-
fication of outliers, it can be confirmed that the value
0.0231413 is the unique value that represent a
possible outlier in the analyzed dataset (in this
definition, outliers are values exceeding by 1.5 the
Interquartile Range below the 25th percentile or above
the 75th percentile).
171
TEAM LinG
172
Clustering of Time Series Data
Anne Denton
North Dakota State University, USA
INTRODUCTION
Time series data is of interest to most science and
engineering disciplines and analysis techniques have
been developed for hundreds of years. There have, how-
ever, in recent years been new developments in data
mining techniques, such as frequent pattern mining,
which take a different perspective of data. Traditional
techniques were not meant for such pattern-oriented
approaches. There is, as a result, a significant need for
research that extends traditional time-series analysis, in
particular clustering, to the requirements of the new
data mining algorithms.
BACKGROUND
Time series clustering is an important component in the
application of data mining techniques to time series data
(Roddick & Spiliopoulou, 2002) and is founded on the
following research areas:
Data Mining: Besides the traditional topics of
classification and clustering, data mining addresses
new goals, such as frequent pattern mining, asso-
ciation rule mining, outlier analysis, and data ex-
ploration.
Time Series Data: Traditional goals include fore-
casting, trend analysis, pattern recognition, filter
design, compression, Fourier analysis, and cha-
otic time series analysis. More recently frequent
pattern techniques, indexing, clustering, classifi-
cation, and outlier analysis have gained in impor-
tance.
Clustering: Data partitioning techniques such as
k-means have the goal of identifying objects that
are representative of the entire data set. Density-
based clustering techniques rather focus on a de-
scription of clusters, and some algorithms iden-
tify the most common object. Hierarchical tech-
niques define clusters at arbitrary levels of granu-
larity.
Data Streams: Many applications, such as com-
munication networks, produce a stream of data.
For real-valued attributes such a stream is ame-
nable to time series data mining techniques.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Time series clustering draws from all of these areas.
It builds on a wide range of clustering techniques that
have been developed for other data, and adapts them
while critically assessing their limitations in the time
series setting.
MAIN THRUST
Many specialized tasks have been defined on time series
data. This chapter addresses one of the most universal data
mining tasks, clustering, and highlights the special aspects
of applying clustering to time series data. Clustering tech-
niques overlap with frequent pattern mining techniques,
since both try to identify typical representatives.
Clustering Time Series
Clustering of any kind of data requires the definition of
a similarity or distance measure. A time series of length
n can be viewed as a vector in an n-dimensional vector
space. One of the best-known distance measures, Eu-
clidean distance, is frequently used in time series clus-
tering. The Euclidean distance measure is a special case
of an Lp norm. Lp norms may fail to capture similarity
well when being applied to raw time series data because
differences in the average value and average derivative
affect the total distance. The problem is typically ad-
dressed by subtracting the mean and dividing the result-
ing vector by its L2 norm, or by working with normalized
derivatives of the data (Gavrilov et al., 2000). Several
specialized distance measures have been used for time
series clustering, such as dynamic time warping, DTW
(Berndt & Clifford 1996), and longest common subse-
quence similarity, LCSS (Vlachos, Gunopulos, & Kollios,
2002).
Time series clustering can be performed on whole
sequences or on subsequences. For clustering of whole
sequences, high dimensionality is often a problem. Di-
mensionality reduction may be achieved through Dis-
crete Fourier Transform (DFT), Discrete Wavelet Trans-
form (DWT), and Principal Component Analysis (PCA),
as some of the most commonly used techniques. DFT
(Agrawal, Falutsos, & Swami, 1993) and DWT have the
goal of eliminating high-frequency components that are
typically due to noise. Specialized models have been
TEAM LinG
Clustering of Time Series Data
introduced that ignore some information in a targeted way
(Jin, Lu, & Shi 2002). Others are based on models for
specific data such as socioeconomic data (Kalpakis, Gada,
& Puttagunta, 2001).
A large number of clustering techniques have been
developed, and for a variety of purposes (Halkidi,
Batistakis, & Vazirgiannis, 2001). Partition-based tech-
niques are among the most commonly used ones for
time series data. The k-means algorithm, which is based
on a greedy search, has recently been generalized to a
wide range of distance measures (Banerjee et al., 2004).
Clustering Subsequences of a Time
Series
A variety of data mining tasks require clustering of
subsequences of one or more time series as preprocess-
ing step, such as Association Rule Mining (Das et al.,
1998), outlier analysis and classification. Partition-
based clustering techniques have been used for this
purpose in analogy to vector quantization (Gersho &
Gray, 1992) that has been developed for signal com-
pression. It has, however, been shown that when a large
number of subsequences are clustered, the resulting
cluster centers are very similar for different time series
(Keogh, Lin, & Truppel, 2003). The problem can be
resolved by using a clustering algorithm that is robust to
noise (Denton, 2004), which adapts kernel-density-
based clustering (Hinneburg & Keim, 2003) to time
series data. Partition-based techniques aim at finding
representatives for all objects. Cluster centers in ker-
nel-density-based clustering, in contrast, are sequences
that are in the vicinity of the largest number of similar
objects. The goal of kernel-density-based techniques is
thereby similar to frequent-pattern mining techniques
such as Motif-finding algorithms (Patel et al., 2002).
Related Problems
One time series clustering problem of particular prac-
tical relevance is clustering of gene expression data
(Eisen, Spellman, Brown, & Botstein, 1998). Gene ex-
pression is typically measured at several points in time
(time course experiment) that may not be equally spaced.
Hierarchical clustering techniques are commonly used.
Density-based clustering has recently been applied to
this problem (Jiang, Pei, & Zhang, 2003)
Time series with categorical data constitute a fur-
ther related topic. Examples are log files and sequences
of operating system commands. Some clustering algo-
rithms in this setting borrow from frequent sequence
mining algorithms (Vaarandi, 2003).
FUTURE TRENDS
C
Storage grows exponentially at rates faster than Moores
law for microprocessors. A natural consequence is that
old data will be kept when new data arrives leading to a
massive increase in the availability of the time-depen-
dent data. Data mining techniques will increasingly have
to consider the time dimension as an integral part of
other techniques. Much of the current effort in the data
mining community is directed at data that has a more
complex structure than the simple tabular format ini-
tially covered in machine learning (Deroski & Lavra ,
2001). Examples, besides time series data, include data
in relational form, such as graph- and tree-structured
data, and sequences. When addressing new settings it
will be of major importance to not only generalize
existing techniques and make them more broadly appli-
cable but to also critically assess problems that may
appear in the generalization process.
CONCLUSION
Despite the maturity of both clustering and time series
analysis, time series clustering is an active and fascinat-
ing research topic. New data mining applications are
constantly being developed and require new types of
clustering results. Clustering techniques from different
areas of data mining have to be adapted to the time series
context. Noise is a particularly serious problem for
time series data, thereby adding challenges to clustering
process. Considering the general importance of time
series data, it can be expected that time series clustering
will remain an active topic for years to come.
REFERENCES
Banerjee, A., Merugu, S., Dhillon, I., & Ghosh, J. (2004,
April). Clustering with Bregman divergences. In Pro-
ceedings SIAM International Conference on Data Min-
ing, Lake Buena Vista, FL.
Berndt D.J., & Clifford, J. (1996). Finding patterns in time
series: A dynamic programming approach. In Advances in
knowledge discovery and data mining (pp. 229-248).
Menlo Park, CA AAAI Press.
Das, G., & Gunopulos, D. (2003). Time series similarity and
indexing. In N. Ye (Ed.), The handbook of data mining (pp.
279-304). Mahwah, NJ: Lawrence Erlbaum Associates.
173
TEAM LinG

Das, G., Lin, K.-I., Mannila, H., Renganathan, G., & Smyth,
P. (1998, Sept). Rule discovery from time series. In Pro-
ceedings IEEE Int. Conf. on Data Mining, Rio de Janeiro,
Brazil.
Denton, A. (2004, August). Density-based clustering of
time series subsequences. In Proceedings The Third
Workshop on Mining Temporal and Sequential Data
(TDM 04) In Conjunction with The Tenth ACM SIGKDD
International Conference on Knowledge Discovery and
Data Mining, Seattle, WA.
Deroski, & Lavra . (2001). Relational data mining. Ber-
lin: Springer.
Jiang, D., Pei, J., & Zhang, A. (2003, March). DHC: A
density-based hierarchical clustering method for time
series gene expression data. In Proceedings 3rd IEEE
Symposium on Bioinformatics and Bioengineering
(BIBE03), Washington, D.C.
Eisen, M.B., Spellman, P.T., Brown, P.O., & Botstein, D.
(1998, December). Cluster analysis and display of ge-
nome-wide expression patterns. In Proceedings of the
National Academy of Science USA, 95 (25) (pp. 14863-8).
Gavrilov, M., Anguelov, D., Indyk, P., & Motwani, R.
(2000). Mining the stock market (extended abstract):
Which measure is best? In Proceedings of the Sixth
ACM SIGKDD International Conference on Knowl-
edge Discovery and Data Mining (pp. 487-496), Boston,
MA.
Gersho, A., & Gray, R.M. (1992). Vector quantization and
signal compression. Boston, MA: Kluwer Academic Pub-
lishers.
Halkidi, M., Batistakis, Y., & Vazirgiannis, M. (2001). On
clustering validation techniques. Intelligent Information
Systems Journal, 17(2-3), 107-145.
Hinneburg, A., & Keim, D.A. (2003, November). A general
approach to clustering in large databases with noise.
Knowledge Information Systems, 5(4), 387-415.
Kalpakis, K., Gada, D., & Puttagunta, V. (2001). Distance
measures for effective clustering of ARIMA time-series. In
Proceedings IEEE International Conference on Data
Mining (pp. 273-280), San Jose, CA.
Keogh, E.J., Lin, J., & Truppel, W. (2003, December).
Clustering of time series subsequences is meaningless:
Implications for previous and future research. In Pro-
ceedings IEEE International Conference on Data Min-
ing (pp. 115-122), Melbourne, FL.
Jin, X., Lu, Y., & Shi, C. (2002). Similarity measure based on
partial information of time series. In Proceedings Eighth
174
Clustering of Time Series Data
ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (pp. 544-549). Edmonton,
AB, Canada.
Patel, P., Keogh, E., Lin, J., & Lonardi, S. (2002). Mining
motifs in massive time series databases. In Proceedings
2002 IEEE International Conference on Data Mining,
Maebashi City, Japan.
Reif, F. (1965). Fundamentals of statistical and thermal
physics. New York: McGraw-Hill.
Roddick, J.F., & Spiliopoulou, M. (2002). A survey of
temporal knowledge discovery paradigms and meth-
ods. IEEE Transactions on Knowledge and Data En-
gineering, 14(4), 750-767.
Vlachos, M., Gunopoulos, D., & Kollios, G., (2002, Feb-
ruary). Discovering similar multidimensional trajecto-
ries. In Proceedings 18th International Conference on
Data Engineering (ICDE02), San Jose, CA.
Vaarandi, R. (2003). A data clustering algorithm for min-
ing patterns from event logs. In Proceedings 2003 IEEE
Workshop on IP Operations and Management, Kansas
City, MO.
KEY TERMS
Dynamic Time Warping (DTW): Sequences are
allowed to be extended by repeating individual time
series elements, such as replacing the sequence
X={x1,x2,x3} by X={x1,x2,x2,x3}. The distance be-
tween two sequences under dynamic time warping is
the minimum distance that can be achieved in by ex-
tending both sequences independently.
Kernel-Density Estimation (KDE): Consider the
vector space in which the data points are embedded. The
influence of each data point is modeled through a
kernel function. The total density is calculated as the
sum of kernel functions for each data point.
Longest Common Subsequence Similarity
(LCSS): Sequences are compared based on the as-
sumption that elements may be dropped. For example,
a sequence X={x1,x2,x3} may be replaced by
X={x1,x3}. Similarity between two time series is
calculated as the maximum number of matching time
series elements that can be achieved if elements are
dropped independently from both sequences. Matches
in real-valued data are defined as lying within some
predefined tolerance.
Partition-Based Clustering: The data set is parti-
tioned into k clusters, and cluster centers are defined
TEAM LinG
Clustering of Time Series Data
based on the elements of each cluster. An objective
function is defined that measures the quality of clustering
based on the distance of all data points to the center of the
cluster to which they belong. The objective function is
minimized.
Principle Component Analysis (PCA): The projec-
tion of the data set to a hyper plane that preserves the
maximum amount of variation. Mathematically, PCA is
equivalent to singular value decomposition on the cova-
riance matrix of the data.
Random Walk: A sequence of random steps in an n-
dimensional space, where each step is of fixed or ran-
domly chosen length. In a random walk time series, time
is advanced for each step and the time series element is
derived using the prescription of a 1-dimensional ran-
dom walk of randomly chosen step length.
Sliding Window: A time series of length n has (n-
w+1) subsequences of length w. An algorithm that oper- C
ates on all subsequences sequentially is referred to as a
sliding window algorithm.
Time Series: Sequence of real numbers, collected
at equally spaced points in time. Each number corre-
sponds to the value of an observed quantity.
Vector Quantization: A signal compression tech-
nique in which an n-dimensional space is mapped to a
finite set of vectors. Each vector is called a codeword
and the collection of all codewords a codebook. The
codebook is typically designed using Linde-Buzo-Gray
(LBG) quantization, which is very similar to k-means
clustering.
175
TEAM LinG
176
Clustering Techniques
Sheng Ma
IBM T.J. Watson Research Center, USA
Tao Li
Florida International University, USA
INTRODUCTION
Clustering data into sensible groupings as a fundamental
and effective tool for efficient data organization, summa-
rization, understanding, and learning has been the sub-
ject of active research in several fields, such as statistics
(Hartigan, 1975; Jain & Dubes, 1988), machine learning
(Dempster, Laird & Rubin, 1977), information theory (Linde,
Buzo & Gray, 1980), databases (Guha, Rastogi & Shim,
1998; Zhang, Ramakrishnan & Livny, 1996), and
bioinformatics (Cheng & Church, 2000) from various per-
spectives and with various approaches and focuses.
From an application perspective, clustering techniques
have been employed in a wide variety of applications,
such as customer segregation, hierarchal document orga-
nization, image segmentation, microarray data analysis,
and psychology experiments.
Intuitively, the clustering problem can be described as
follows: Let W be a set of n entities, finding a partition of
W into groups, such that the entities within each group are
similar to each other, while entities belonging to different
groups are dissimilar. The entities usually are described
by a set of measurements (attributes). Clustering does not
use category information that labels the objects with prior
identifiers. The absence of label information distinguishes
cluster analysis from classification and indicates that the
goals of clustering are just finding a hidden structure or
compact representation of data instead of discriminating
future data into categories.
BACKGROUND
Generally, clustering problems are determined by five
basic components:
Data Representation: What is the (physical) repre-
sentation of the given dataset? What kind of at-
tributes (e.g., numerical, categorical or ordinal) are
there?
Data Generation: The formal model for describing
the generation of the dataset. For example, Gaussian
mixture model is a model for data generation.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Criterion/Objective Function: What are the objec-
tive functions or criteria that the clustering solu-
tions should aim to optimize? Typical examples
include entropy, maximum likelihood, and within-
class or between-class distance (Li, Ma & Ogihara, 2004a).
Optimization Procedure: What is the optimization
procedure for finding the solutions? A clustering
problem is known to be NP-complete (Brucker, 1977),
and many approximation procedures have been de-
veloped. For instance, Expectation-Maximization-
(EM) type algorithms have been used widely to find
local minima of optimization.
Cluster Validation and Interpretation: Cluster vali-
dation evaluates the clustering results and judges
the cluster structures. Interpretation often is neces-
sary for applications. Since there is no label informa-
tion, clusters are sometimes justified by ad hoc
methods (such as exploratory analysis), based on
specific application areas.
For a given clustering problem, the five components
are tightly coupled. The formal model is induced from the
physical representation of the data; the formal model,
along with the objective function, determines the cluster-
ing capability, and the optimization procedure decides
how efficiently and how effectively the clustering results
can be obtained. The choice of the optimization procedure
depends on the first three components. Validation of
cluster structures is a way of verifying assumptions on
data generation and of evaluating the optimization procedure.
MAIN THRUST
We review some of the current clustering techniques in
this section. Figure 1 gives a summary of clustering
techniques. The following further discusses traditional
clustering techniques, spectral-based analysis, model-
based clustering, and co-clustering.
Traditional clustering techniques focus on one-sided
clustering, and they can be classified as partitional, hier-
archical, density-based, and grid-based (Han & Kamber,
2000). Partitional clustering attempts to directly decom-
TEAM LinG
Clustering Techniques
Figure 1. Summary of clustering techniques
pose the dataset into disjoint classes, such that the data
points in a class are nearer to one another than the data
points in other classes. Hierarchical clustering proceeds
successively by building a tree of clusters. Density-
based clustering is grouping the neighboring points of a
dataset into classes based on density conditions. Grid-
based clustering quantizes the data space into a finite
number of cells that form a grid-structure and then per-
forms clustering on the grid structure. Most of these
algorithms use distance functions as objective criteria
and are not effective in high-dimensional spaces.
As an example, we take a closer look at K-means
algorithms. The typical K-means type algorithm is a widely-
used partition-based clustering approach. Basically, it
first chooses a set of K data points as initial cluster
representatives (e.g., centers) and then performs an itera-
tive process that alternates between assigning the data
points to clusters, based on their distances to the cluster
representatives, and updating the cluster representa-
tives, based on new cluster assignments. The iterative
optimization procedure of K-means algorithm is a special
form of EM-type procedure. The K-means type algorithm
treats each attribute equally and computes the distances
between data points and cluster representatives to deter-
mine cluster memberships.
A lot of algorithms have been developed recently to
address the efficiency and performance issues presented
in traditional clustering algorithms. Spectral analysis has
been shown to tightly relate to clustering task. Spectral
clustering (Ng, Jordan & Weiss, 2001; Weiss, 1999),
closely related to the latent semantics index (LSI), uses
selected eigenvectors of the data affinity matrix to obtain
a data representation that easily can be clustered or
C
embedded in a low-dimensional space. Model-based clus-
tering attempts to learn generative models, by which the
cluster structure is determined, from the data. Tishby,
Pereira, and Bialek (1999) and Slonim and Tishby (2000)
developed information bottleneck formulation, in which,
given the empirical joint distribution of two variables, one
variable is compressed so that the mutual information
about the other is preserved as much as possible. Other
recent developments of clustering techniques include
ensemble clustering, support vector clustering, matrix
factorization, high-dimensional data clustering, distrib-
uted clustering, and so forth.
Another interesting development is co-clustering,
which conducts simultaneous, iterative clustering of both
data points and their attributes (features) through utiliz-
ing the canonical duality contained in the point-by-at-
tribute data representation. The idea of co-clustering of
data points and attributes dates back to Anderberg (1973)
and Nishisato (1980). Govaert (1985) researches simulta-
neous block clustering of the rows and columns of the
contingency table. The idea of co-clustering also has
been applied to cluster gene expression and experiments
(Cheng & Church, 2000). Dhillon (2001) presents a co-
clustering algorithm for documents and words using
bipartite graph formulation and a spectral heuristic. Re-
cently, Dhillon, et al. (2003) proposed an information-
theoretic co-clustering method for a two-dimensional
contingency table. By viewing the non-negative contin-
gency table as a joint probability distribution between
two discrete random variables, the optimal co-clustering
then maximizes the mutual information between the clus-
tered random variables. Li and Ma (2004) recently devel-
oped Iterative Feature and Data (IFD) clustering by rep-
177
TEAM LinG

resenting the data generation with data and feature coef-
ficients. IFD enables an iterative co-clustering procedure
for both data and feature assignments. However, unlike
previous co-clustering approaches, IFD performs cluster-
ing using the mutually reinforcing optimization procedure
that has a proven convergence property. IFD only handles
data with binary features. Li, Ma, and Ogihara (2004b)
further extended the idea for general data.
FUTURE TRENDS
Although clustering has been studied for many years,
many issues, such as cluster validation, still need more
investigation. In addition, new challenges, such as
scalability, high-dimensionality, and complex data types,
have been brought by the ever-increasing growth of infor-
mation exposure and data collection.
Scalability and Efficiency: With the collection of
huge amounts of data, clustering faces the problems
of scalability in terms of both computation time and
memory requirements. To resolve the scalability is-
sues, methods such as incremental and streaming
approaches, sufficient statistics for data summary,
and sampling techniques have been developed.
Curse of Dimensionality: Another challenge is the
high dimensionality of data. It has been shown that
in a high dimensional space, the distance between
every pair of points is almost the same for a wide
variety of data distributions and distance functions
(Beyer et al., 1999). Hence, most algorithms do not
work efficiently in high-dimensional spaces, due to
the curse of dimensionality. Many feature selection
techniques have been applied to reduce the dimen-
sionality of the space. However, as demonstrated in
Aggarwal, et al. (1999), in many case, the correlations
among the dimensions often are specific to data
locality; in other words, some data points are corre-
lated with a given set of features, and others are
correlated with respect to different features. As
pointed out in Hastie, Tibshirani, and Friedman (2001)
and Domeniconi, Gunopulos, and Ma (2004), all
methods that overcome the dimensionality problems
use a metric for measuring neighborhoods, which is
often implicit and/or adaptive.
Complex Data Types: The problem of clustering
becomes more challenging when the data contains
complex types (e.g., when the attributes contain
both categorical and numerical values). There are no
inherent distance measures between data values.
This is often the case in many applications where
data are described by a set of descriptive or pres-
ence/absence attributes, many of which are not nu-
178
Clustering Techniques
merical. The presence of complex types also makes
the cluster validation and interpretation difficult.
More challenges also include clustering with multiple
criteria (where clustering problems often require optimi-
zation over more than one criterion), clustering relation
data (where data is represented with multiple relation
tables), and distributed clustering (where data sets are
geographically distributed across multiple sites).
CONCLUSION
Clustering is a classical topic to segment and group
similar data objects. Many algorithms have been devel-
oped in the past. Looking ahead, many challenges drawn
from real-world applications will drive the search for
efficient algorithms that are able to handle heteroge-
neous data in order to process a large volume and to scale
to deal with a large number of dimensions.
REFERENCES
Aggarwal, C. et al. (1999). Fast algorithms for projected
clustering. Proceedings of ACM SIGMOD Conference.
Anderberg, M.-R. (1973). Cluster analysis for applica-
tions. Academic Press Inc.
Beyer, K., Goldstein, J., Ramakrishnan, R., & Shaft, U.
(1999). When is nearest neighbor meaningful? Proceed-
ings of the International Conference on Database
Theory.
Brucker, P. (1977). On the complexity of clustering prob-
lems. In R. Henn, B. Korte, & W. Oletti (Eds.), Optimiza-
tion and operations research (pp. 45-54). New York:
Springer-Verlag.
Cheng, Y., & Church, G.M. (2000). Bi-clustering of ex-
pression data. Proceedings of the Eighth International
Conference on Intelligent Systems for Molecular Biol-
ogy (ISMB).
Dempster, A., Laird, N., & Rubin, D. (1977). Maximum
likelihood from incomplete data via the EM algorithm.
Journal of the Royal Statistical Society, 39, 1-38.
Dhillon, I. (2001). Co-clustering documents and words
using bipartite spectral graph partitioning. Technical
Report 2001-05. Austin CS Dept.
Dhillon, I.S., Mallela, S., & Modha, S.-S. (2003). Informa-
tion-theoretic co-clustering. Proceedings of ACM
SIGKDD.
TEAM LinG
Clustering Techniques
Domeniconi, C., Gunopulos, D., & Ma, S. (2004). Within-
cluster adaptive metric for clustering. Proceedings of the
SIAM International Conference on Data Mining.
Govaert, G. (1985). Simultaneous clustering of rows and
columns. Control and Cybernetics, 437-458.
Guha, S., Rastogi, R., & Shim, K. (1998). CURE: An efficient
clustering algorithm for large databases. Proceedings of
the ACM SIGMOD Conference.
Han, J., & Kamber, M. (2000). Data mining: Concepts and
techniques. Morgan Kaufmann Publishers.
Hartigan, J. (1975). Clustering algorithms. Wiley.
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The
elements of statistical learning: Data mining, inference,
prediction. Springer.
Jain, A.K., & Dubes, R.C. (1988). Algorithms for cluster-
ing data. Upper Saddle River, NJ: Prentice Hall.
Li, T., & Ma, S. (2004). IFD: Iterative feature and data
clustering. Proceedings of the SIAM International Con-
ference on Data Mining.
Li, T., Ma, S., & Ogihara, M. (2004a). Entropy-based
criterion in categorical clustering. Proceedings of the
International Conference on Machine Learning (ICML 2004).
Li, T., Ma, S., & Ogihara, M. (2004b). Document clustering via
adaptive subspace iteration. Proceedings of the ACM SIGIR.
Linde, Y., Buzo, A., & Gray, R.M. (1980). An algorithm for
vector quantization design. IEEE Transactions on Com-
munications, 28(1), 84-95.
Ng, A., Jordan, M., & Weiss, Y. (2001). On spectral
clustering: Analysis and an algorithm. Advances in Neu-
ral Information Processing Systems, 14.
Nishisato, S. (1980). Analysis of categorical data: Dual
scaling and its applications. Toronto: University of
Toronto Press.
Slonim, N., & Tishby, N. (2000). Document clustering
using word clusters via the information bottleneck method.
Proceedings of the ACM SIGIR.
Tishby, N., Pereira, F.-C., & Bialek, W. (1999). The infor-
mation bottleneck method. Proceedings of the 37th An- C
nual Allerton Conference on Communication, Control
and Computing.
Weiss, Y. (1999). Segmentation using eigenvectors: A
unifying view. Proceedings of ICCV (2).
Zhang, T., Ramakrishnan, R., & Livny, M. (1996). BIRCH:
An efficient data clustering method for very large data-
bases. Proceedings of the ACM SIGMOD Conference.
KEY TERMS
Cluster: A set of entities that are similar between
themselves and dissimilar to entities from other clusters.
Clustering: The process of dividing the data into
clusters.
Cluster Validation: Evaluates the clustering results
and judges the cluster structures.
Co-Clustering: Performs simultaneous clustering of
both points and their attributes by way of utilizing the
canonical duality contained in the point-by-attribute data
representation.
Curse of Dimensionality: This expression is due to
Bellman; in statistics, it relates to the fact that the conver-
gence of any estimator to the true value of a smooth
function defined on a space of high dimension is very
slow. It has been used in various scenarios to refer to the
fact that the complexity of learning grows significantly
with the dimensions.
Spectral Clustering: The collection of techniques
that performs clustering tasks using eigenvectors of
matrices derived from the data.
Subspace Clustering: An extension of traditional
clustering techniques that seeks to find clusters in differ-
ent subspaces within a given dataset.
179
TEAM LinG
180
Clustering Techniques for Outlier Detection
Frank Klawonn
University of Applied Sciences Braunschweig/Wolfenbuettel, Germany
Frank Rehm
German Aerospace Center, Germany
INTRODUCTION
For many applications in knowledge discovery in data-
bases, finding outliers, which are rare events, is of
importance. Outliers are observations that deviate sig-
nificantly from the rest of the data, so they seem to have
been generated by another process (Hawkins, 1980).
Such outlier objects often contain information about an
untypical behaviour of the system.
However, outliers bias the results of many data-
mining methods such as the mean value, the standard
deviation, or the positions of the prototypes of k-means
clustering (Estivill-Castro & Yang, 2004; Keller, 2000).
Therefore, before further analysis or processing of data
is carried out with more sophisticated data-mining tech-
niques, identifying outliers is a crucial step. Usually,
data objects are considered as outliers when they occur
in a region of extremely low data density.
Many clustering techniques that deal with noisy data
and can identify outliers, such as possibilistic cluster-
ing (PCM) (Krishnapuram & Keller, 1993, 1996) and
noise clustering (NC) (Dave, 1991; Dave &
Krishnapuram, 1997), need good initializations or suf-
fer from lack of adaptability to different cluster sizes.
Distance-based approaches (Knorr & Ng, 1998; Knorr,
Ng, & Tucakov, 2000) have a global view on the data set.
These algorithms can hardly treat data sets that contain
regions with different data density (Breuning, Kriegel,
Ng, & Sander, 2000).
In this work, we present an approach that combines a
fuzzy clustering algorithm (Hppner, Klawonn, Kruse,
& Runkler, 1999) or any other prototype-based cluster-
ing algorithm with statistical distribution-based outlier
detection.
BACKGROUND
Prototype-based clustering algorithms approximate a
feature space by means of an appropriate number of
prototype vectors, where each vector is located in the
centre of the group of data (the cluster) that belongs to
the respective prototype. Clustering usually aims at
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
partitioning a data set into groups or clusters of data,
where data assigned to the same cluster are similar and
data from different clusters are dissimilar. With this
partitioning concept in mind, an important aspect of
typical applications of cluster analysis is the identifica-
tion of the number of clusters in a data set. However,
when we are interested in identifying outliers, the exact
number of clusters is irrelevant (Georgieva & Klawonn).
The idea of whether one prototype covers two or more
data clusters or whether two or more prototypes com-
pete for the same data cluster is not important as long as
the actual outliers are identified and assigned to a proper
cluster. The number of prototypes used for clustering
depends, of course, on the number of expected clusters
but also on the distance measure respectively the shape
of the expected clusters. Because this information is
usually not available, the Euclidean distance measure is
often recommended with rather copious prototypes.
One of the most referred statistical tests for outlier
detection is the Grubbs test (Grubbs, 1969). This test is
used to detect outliers in a univariate data set. Grubbs
test detects one outlier at a time. This outlier is removed
from the data set, and the test is iterated until no outliers
are detected.
MAIN THRUST
The detection of outliers that we propose in this work is
a modified version of the one proposed in Santos-
Pereira and Pires (2002) and is composed of two differ-
ent techniques. In the first step we partition the data set
with the fuzzy c-means clustering algorithm so the fea-
ture space is approximated with an adequate number of
prototypes. The prototypes will be placed in the centre
of regions with a high density of feature vectors. Be-
cause outliers are far away from the typical data, they
influence the placing of the prototypes.
After partitioning the data, only the feature vectors
belonging to each single cluster are considered for the
detection of outliers. For each attribute of the feature
vectors of the considered cluster, the mean value and the
standard deviation has to be calculated. For the vector
TEAM LinG
Clustering Techniques for Outlier Detection
Figure 1. Outlier detection with different numbers of prototypes
(a) 1 prototype (b) 3 prototypes
with the largest distance1 to the mean vector, which is
assumed to be an outlier, the value of the z-transforma-
tion for each of its components is compared to a critical
value. If one of these values is higher than the respective
critical value, then this vector is declared an outlier. One
can use the Mahalanobis distance as in Santos-Pereira
and Pires (2002), but because simple clustering tech-
niques such as the fuzzy c-means algorithm tend to spheri-
cal clusters, we apply a modified version of Grubbs test,
not assuming correlated attributes within a cluster.
The critical value is a parameter that must be set for
each attribute depending on the specific definition of an
outlier. One typical criterion can be the maximum num-
ber of outliers with respect to the amount of data
(Klawonn, 2004). Eventually, large critical values lead
to smaller numbers of outliers, and small critical values
lead to very compact clusters. Note that the critical
value is set for each attribute separately. This leads to an
axes-parallel view of the data, which in cases of axes-
parallel clusters leads to a better outlier detection than
the (hyper)spherical view of the data.
If an outlier is found, the feature vector has to be
removed from the data set. With the new data set, the
mean value and the standard deviation have to be calcu-
lated again for each attribute. With the vector that has
the largest distance to the new centre vector, the outlier
test will be repeated by checking the critical values. This
procedure will be repeated until no outlier is found. The
other clusters are treated in the same way.
Figure 1 shows the results of the proposed algo-
rithm. The crosses in this figure are feature vectors,
which are recognized as outliers. As expected, only a
few points are declared as outliers when approximating
the feature space with only one prototype. The proto-
type will be placed in the centre of all feature vectors.
Hence, only points on the edges are defined as outliers.
Comparing the solutions with 3 and 10 prototypes, you
can determine that both solutions are almost identical.
Even in the border regions, were two prototypes com-
peting for some data points, the algorithm would rarely
identify these points as outliers, which they intuitively
are not.
C
(c) 10 prototypes
FUTURE TRENDS
Figure 1 shows that the algorithm can identify outliers
in multivariate data in a stable way. With only a few
parameters, the solution can be adapted to different
requirements concerning the specific definition of an
outlier. With the choice of the number of prototypes, it
is possible to influence the result in a way that with lots
of prototypes, even smaller data groups can be found. To
avoid overfitting the data, it makes sense in certain
cases to eliminate very small clusters. However, finding
out the proper number of prototypes should be of inter-
est for further investigations.
In the case of using a fuzzy clustering algorithm such
as FCM (Bezdek, 1981) to partition the data, it is
possible to assign a feature vector to different proto-
type vectors. In that way, you can consolidate whether a
certain feature vector is an outlier if the algorithm
decides for each single cluster that the corresponding
feature vector is an outlier.
FCM provides membership degrees for each feature
vector to every cluster. One approach could be to assign
a feature vector to the corresponding clusters with the
two highest membership degrees. The feature vector is
considered as an outlier if the algorithm makes the same
decision in both clusters. In cases where the algorithm
gives no definite answers, the feature vector can be
labeled and processed by further analysis.
CONCLUSION
In this article, we describe a method to detect outliers in
multivariate data. Because information about the num-
ber and shape of clusters is often not known in advance,
it is necessary to have a method that is relatively robust
with respect to these parameters. To obtain a stable
algorithm, we combined approved clustering techniques,
including the FCM or k-means, with a statistical method
to detect outliers. Because the complexity of the pre-
sented algorithm is linear in the number of points, it can
be applied to large data sets.
181
TEAM LinG

REFERENCES
Bezdek, J. C. (1981). Pattern recognition with fuzzy
objective function algorithms. New York: Plenum Press.
Breunig, M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000).
LOF: Identifying density-based local outliers. Proceed-
ings of the ACM SIGMOD International Conference on
Management of Data (pp. 93-104).
Dave, R. N. (1991). Characterization and detection of noise
in clustering. Pattern Recognition Letters, 12, 657-664.
Dave, R. N., & Krishnapuram, R. (1997). Robust clustering
methods: A unified view. IEEE Transactions on Fuzzy
Systems, 5(2), 270-293.
Estivill-Castro, V., & Yang, J. (2004). Fast and robust
general purpose clustering algorithms. Data Mining
and Knowledge Discovery, 8, 127-150.
Georgieva, O., & Klawonn, F. A cluster identification
algorithm based on noise clustering. Manuscript sub-
mitted for publication.
Grubbs, F. (1969). Procedures for detecting outlying
observations in samples. Technometrics, 11(1), 1-21.
Hawkins, D. (1980). Identification of outliers. London:
Chapman and Hall.
Hppner, F., Klawonn, F., Kruse, R., & Runkler, T. (1999).
Fuzzy cluster analysis. Chichester, UK: Wiley.
Keller, A. (2000). Fuzzy clustering with outliers. Pro-
ceedings of the 19th International Conference of the
North American Fuzzy Information Processing Society,
USA.
Klawonn, F. (2004). Noise clustering with a fixed fraction
of noise. In A. Lotfi & J. M. Garibaldi (Eds.), Applications
and science in soft computing. Berlin, Germany: Springer.
Knorr, E. M., & Ng, R. T. (1998). Algorithms for mining
distance-based outliers in large datasets. Proceedings
of the 24th International Conference on Very Large
Data Bases (pp. 392-403).
Knorr, E. M., Ng, R. T., & Tucakov, V. (2000). Distance-
based outliers: Algorithms and applications. VLDB Jour-
nal, 8(3-4), 237-253.
Krishnapuram, R., & Keller, J. M. (1993). A possibilistic
approach to clustering. IEEE Transactions on Fuzzy
Systems, 1(2), 98-110.
182
Clustering Techniques for Outlier Detection
Krishnapuram, R., & Keller, J. M. (1996). The possibilistic
C-means algorithm: Insights and recommendations. IEEE
Transactions on Fuzzy Systems, 4(3), 385-393.
Santos-Pereira, C. M., & Pires, A. M. (2002). Detection of
outliers in multivariate data: A method based on cluster-
ing and robust estimators. Proceedings of the 15th
Symposium on Computational Statistics (pp. 291-296),
Germany.
KEY TERMS
Cluster Analysis: To partition a given data set into
clusters where data assigned to the same cluster are
similar and data from different clusters are dissimilar.
Cluster Prototypes (Centres): Clusters in objec-
tive function-based clustering are represented by pro-
totypes that define how the distance of a data object to
the corresponding cluster is computed. In the simplest
case, a single vector represents the cluster, and the
distance to the cluster is the Euclidean distance be-
tween cluster centre and data object.
Fuzzy Clustering: Cluster analysis where a data
object can have membership degrees to different clus-
ters. Usually it is assumed that the membership degrees
of a data object to all clusters add up to one, so a
membership degree can also be interpreted as the prob-
ability that the data object belongs to the correspond-
ing cluster.
Noise Clustering: An additional noise cluster is
induced in objective function-based clustering to col-
lect the noise data or outliers. All data objects are
assumed to have a fixed (large) distance to the noise
cluster, so only data that is far away from all the other
clusters will be assigned to the cluster.
Outliers: Observations in a sample, so far sepa-
rated in value from the remainder as to suggest that they
are generated by another process or are the result of an
error in measurement.
Overfitting: The phenomenon that a learning algo-
rithm adapts so well to a training set that the random
disturbances in the training set are included in the
model as being meaningful. Consequently, as these
disturbances do not reflect the underlying distribution,
the performance on the test set, with its own but defini-
tively other disturbances, will suffer from techniques
that tend to fit well to the training set.
TEAM LinG
Clustering Techniques for Outlier Detection

Possibilistic Clustering: The Possibilistic c-means ENDNOTE

(PCM) family of clustering algorithms is designed to
alleviate the noise problem by relaxing the constraint on
memberships used in probabilistic fuzzy clustering.
Z-Transformation: With the z-transformation, one
can transform the values of any variable into values of the
standard normal distribution.
C
1
Different distance measures can be used to deter-
mine the vector with the largest distance to the
centre vector. For elliptical non-axes-parallel clus-
ters, the Mahalanobis distance leads to good re-
sults. If no information about the shape of the
clusters is available, the Euclidean distance is
commonly used.

183

TEAM LinG
184
Combining Induction Methods with the
Multimethod Approach
Mitja Leni
University of Maribor, FERI, Slovenia
Peter Kokol
University of Maribor, FERI, Slovenia
Petra Povalej
University of Maribor, FERI, Slovenia
Milan Zorman
University of Maribor, FERI, Slovenia
INTRODUCTION
The aggressive rate of growth of disk storage and, thus,
the ability to store enormous quantities of data have far
outpaced our ability to process and utilize that. This
challenge has produced a phenomenon called data
tombsdata is deposited to merely rest in peace, never
to be accessed again. But the growing appreciation that
data tombs represent missed opportunities in cases
supporting scientific discovering, business exploita-
tion, or complex decision making has awakened the
growing commercial interest in knowledge discovery
and data-mining techniques. That, in order, has stimu-
lated new interest in the automatic knowledge induction
from cases stored in large databasesa very important
class of techniques in the data-mining field. With the
variety of environments, it is almost impossible to
develop a single-induction method that would fit all
possible requirements. Thereafter, we constructed a
new so-called multi-method approach, trying out some
original solutions.
BACKGROUND
Through time, different approaches have evolved, such
as symbolic approaches, computational learning theory,
neural networks, and so forth. In our case, we focus on
an induction process to find a way to extract generalized
knowledge from observed cases (instances). That is
accomplished by using inductive inference that is the
process of moving from concrete instances to general
model(s), where the goal is to learn how to extract
knowledge from objects by analyzing a set of instances
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
(already solved cases) whose classes are known. In-
stances are typically represented as attribute-value vec-
tors. Learning input consists of a set of vectors/in-
stances, each belonging to a known class, and the output
consists of a mapping from attribute values to classes.
This mapping hypothesis should accurately classify both
the learning instances and the new unseen instances. The
hypothesis hopefully represents generalized knowledge
that is interesting for domain experts. The fundamental
theory of learning is presented by Valiant (1984) and
Auer (1995).
Single-Method Approaches
When comparing single approaches with different knowl-
edge representations and different learning algorithms,
there is no clear winner. Each method has its own advan-
tages and some inherent limitations. Decision trees
(Quinlan, 1993), for example, are easily understandable
by a human and can be used even without a computer, but
they have difficulties expressing complex nonlinear prob-
lems. On the other hand, connectivistic approaches that
simulate cognitive abilities of the brain can extract com-
plex relations, but solutions are not easily understandable
to humans (only numbers of weights), and, therefore, as
such, they are not directly usable for data mining. Evolu-
tionary approaches to knowledge extraction are also a
good alternative, because they are not inherently limited
to a local solution (Goldberg, 1989) but are
computationally expensive. There are many other ap-
proaches, like representation of the knowledge with rules,
rough sets, case-based reasoning, support vector ma-
chines, different fuzzy methodologies, and ensemble
methods (Dietterich, 2000).
TEAM LinG
Combining Induction Methods with the Multimethod Approach
Hybrid Approaches
Hybrid approaches rest on the assumption that only the
synergetic combination of single models can unleash
their full power. Each of the single methods has its
advantages but also inherent limitations and disadvan-
tages, which must be taken into account when using a
particular method. For example, symbolic methods usu-
ally represent the knowledge in human readable form,
and the connectivistc methods perform better in classi-
fication of unseen objects and are less affected by the
noise in data as are symbolic methods. Therefore, the
logical step is to combine both worlds to overcome the
disadvantages and limitations of a single one.
In general, the hybrids can be divided according to
the flow of knowledge into four categories (Iglesias,
1996):
Sequential Hybrid (Chain Processing): The out-
put of one method is an input to another method.
For example, the neural net is trained with the
training set to reduce noise.
Parallel Hybrid (Co-Processing): Different
methods are used to extract knowledge. In the next
phase, some arbitration mechanism should be used
to generate appropriate results.
External Hybrid (Meta Processing): One
method uses another external one. For example,
meta decision trees (Todorovski & Dzeroski, 2000)
that use neural nets in decision nodes to improve
the classification results.
Embedded Hybrid (Sub-Processing): One
method is embedded in another. That is the most
powerful hybrid, but the least modular one, be-
cause usually the methods are coupled tightly.
The hybrid systems are commonly static in structure
and cannot change the order of how single methods are
applied. To be able to use embedded hybrids of different
internal knowledge representation, it is commonly re-
quired to transform one method representation into
another. Some transformations are trivial, especially
when converting from symbolic approaches. The prob-
lem is when the knowledge is not so clearly presented,
like in a case of the neural network (McGarry, Wermter
& MacIntyre, 2001; Zorman, Kokol & Podgorelec,
2000). The knowledge representation issue is very im-
portant in the multi-method approach, and we solved it
in the original manner.
MULTI-METHOD APPROACH
C
Multi-method approach was introduced in Leni and Kokol
(2002). While studying other approaches, we were in-
spired by the idea of hybrid approaches and evolutionary
algorithms. Both approaches are very promising in
achieving the goal to improve the quality of knowledge
extraction and are not inherently limited to sub-optimal
solutions. We also noticed that almost all attempts to
combine different methods use the loose coupling ap-
proach. Of course, loose coupling is easier to imple-
ment, but methods work almost independently of each
other, and, therefore, a lot of luck is needed to make
them work as a team.
Opposed to the conventional hybrids described in
the previous section, our idea is to dynamically combine
and apply different methods in no predefined order to
the same problem or decomposition of the problem. The
main concern of the mutli-method approach is to find a
way to enable a dynamic combination of methods to the
somehow quasi-unified knowledge representation. In
multiple, equally-qualitative solutions, like evolution-
ary algorithms (EA), each solution is obtained using an
application of different methods with different param-
eters. Therefore, we introduce a population composed
of individuals/solutions that have the common goal to
improve their classification abilities on a given environ-
ment/problem. We also enable the coexistence of dif-
ferent types of knowledge representation in the same
population. The most common knowledge representa-
tion models have to be standardized and strictly typed to
support the applicability of different methods on indi-
viduals. Each induction method implementation uses its
own internal knowledge representation that is not com-
patible with other methods that use the same type of
knowledge. A typical example is WEKA, which uses at
least four different knowledge representations for de-
cision trees. Standardization, in general, brings greater
modularity and interchangeability, but it has the follow-
ing disadvantage: already existing methods cannot be
directly integrated and have to be adjusted to the stan-
dardized representation.
Initial population of extracted knowledge is gener-
ated using different methods. In each generation, differ-
ent operations appropriate for individual knowledge are
applied to improve existing and create new intelligent
systems. That enables incremental refinement of ex-
tracted knowledge, with different views on a given prob-
lem. The main problem is how to combine methods that
185
TEAM LinG
 Combining Induction Methods with the Multimethod Approach

Figure 1. An example of a decision tree induced using
multi-method approach. Each node is induced with
appropriate method (GAgenetic algorithm, ID3, Gini,
Chi-square, J-measure, SVM, neural network, etc.)
GA
Gini
GA
use different knowledge representations (e.g., neural net-
works and decision trees). In such cases, we have two
alternatives: (1) to convert one knowledge representation
to another using different already known methods or (2) to
combine both knowledge representations in a single intel-
ligent system. In both cases, knowledge transmutation is
executed (Cox & Ram, 1999). In the first case, conversion
between different knowledge representations must be
implemented, which is usually not perfect, and some parts
of the knowledge can be lost. But on the other hand, it can
provide a different view and a good starting point in
hypothesis search space.
The second approach, which is based on combining
knowledge, requires some cut-points where knowledge
representations can be merged. For example, in a deci-
sion tree, such cut-points are internal nodes, where the
condition in an internal node can be replaced by another
intelligent system (e.g., support vector machine [SVM]).
The same idea also can be applied in decision leafs
(Figure 1).
Operators
Using the idea of the multi-method approach, we de-
signed a framework that operates on a population of
extracted knowledge representationsindividuals. Since
methods usually are composed of operations that can be
Figure 2. Multi-method framework
Framework
support
Methods Framework
support
Methods
Population
Individual
operator 1
Individual
operator i+1
Population
operator 1
operator k+1
.
. . . .
. . . .
. . . .
reused in other methods, we introduced methods on the
basis of operators. Therefore, we introduced the opera-
tion on an individual as a function that transforms one or
more individuals into a single individual. Operation can
be a part of one or more methods, like a pruning operator,
a boosting operator, and so forth. An operator-based
view provides the ability simply to add new operations
ID3, GA to the framework (Figure 2).
Usually, methods are composed of operations that
GA can be reused in other methods. We introduced the
operation on an individual that is a function, which
ID3
transforms one or more individuals into a single indi-
vidual. Operation can be part of one or more methods,
like a pruning operator, a boosting operator, and so
forth. The transformation to another knowledge repre-
sentation also is introduced on the individual operator
level. Therefore, the transition from one knowledge
representation to another is presented as a method. An
operator-based view provides us with the ability simply
to add new operations to the framework (Figure 2).
Representation with individual operations facili-
tates is an effective and modular way to represent the
result as a single individual, but, in general, the result of
operation also can be a population of individuals (e.g.,
mutation operation in EA is defined on an individual
level and on the population level). The single method is
composed of population operations that use individual
operations and is introduced as a strategy in the frame-
work that improves individuals in a population. A single
method is composed out of population operations that
use individual operations and is introduced as a strategy
in the framework that improves individuals in a popula-
tion (Figure 2). Population operators can be general-
ized with higher order functions and thereby reused in
different methods.
Meta-Level Control
An important concern of the multi-method framework
is how to provide the meta-level services to manage the
available resources and the application of the methods.
We extended the quest for knowledge into another
dimension; that is, the quest for the best application
order of methods. The problem that arises is how to
control the quality of resulting individuals and how to
intervene in the case of bad results. Due to different
knowledge representations, solutions cannot be com-
Framework
support
Methods
pared trivially to each other, and the assessment of
Strategy
1
Strategy
i+1
.
which method is better is hard to imagine. Individuals in
a population cannot be evaluated explicitly, and the
. .
. .

Individual Population Population Strategy Strategy

explicit fitness function cannot be calculated easily.
Individual
operator i operator n operator k operator l i m

Therefore, the comparison of individuals and the as-

Population
sessment of the quality of the whole population cannot
be given. Even if some criteria to calculate fitness

186

TEAM LinG
Combining Induction Methods with the Multimethod Approach
function could be found, it probably would be very time-
consuming and computational-intensive. Therefore, the
idea of classical evolutionary algorithms controlling the
quality of population and guidance to the objective can-
not be applied.
To achieve self-adaptive behavior of the evolution-
ary algorithm, the strategy parameters have to be coded
directly into the chromosome (Thrun & Pratt, 1998).
But in our approach, the meta-level strategy does not
know about the structure of the chromosome, and not all
of the methods use the EA approach to produce a solu-
tion. Therefore, for meta-level chromosomes, the pa-
rameters of the method and its individuals are taken.
When dealing with self-adapting population with no
explicit evaluation/fitness function, there is also an
issue of the best or most promising individual (progar
et al. 2000). But, of course, the question of how to
control the population size or increase selection pres-
sure must be answered effectively. Our solution was to
classify population operators into three categories:
operators for reducing, operators for maintaining, and
operators for increasing the population size.
CONCRETE COMBINATION
TECHNIQUES
The multi-method approach searches for solutions in
huge (infinite) search space and exploits the acquisition
technique of each integrated method. Population of
individual solutions represents a different aspect of
extracted knowledge. Transmutation from one to an-
other knowledge representation introduces new aspects.
We can draw parallels between the multi-method ap-
proach and the scientific discovery. In real life, based on
the observed phenomena, various hypotheses are con-
structed. Different scientific communities draw differ-
ent conclusions (hypotheses) consistent with collected
data. For example, there are many theories about the
creation of the universe, but the current, widely ac-
cepted theory is the theory of the big bang. During the
following phase, scientists discuss their theories, knowl-
edge is exchanged, new aspects are encountered, and
data collected is reevaluated. In that manner, existing
hypotheses are improved, and new better hypotheses are
constructed.
Decision Trees
Decision trees are very appropriate to use as glue be-
tween different methods. In general, condition nodes
contain classifier (usually simple attribute comparison)
that enables quick and easy integration of different
Figure 3. Hypothesis separation using GA
C
h1
h3 h2
methods. On other hand, there are many different methods
for decision-tree induction that all generate knowledge in
the form of tree. The most popular method is greedy
heuristic induction of a decision tree, which produces a
single decision tree with respect to the purity measure
(heuristic) of each split in a decision tree. Altering purity
measure may produce totally different results with differ-
ent aspects (hypotheses) for a given problem. Hypoth-
esis induction is done with the use of evolutionary algo-
rithms (EA). When designing EA, algorithm operators for
mutation, crossover and selection have to be carefully
chosen (Podgorelec et al., 2001). Combining the EA ap-
proach with heuristic approaches dramatically reduces
hypothesis search space.
There is another issue when combining two methods
using another classifier for separation. For example, in
Figure 3, there are two hypotheses, h1 and h2, that could
be perfectly separately induced using an existing set of
methods. Lets suppose that there is no method that is
able to acquire both hypotheses in a single hypothesis.
Therefore, we need a separation of the problem using
another hypothesis, h3, which has no special meaning to
induce a successful composite hypothesis.
Problem Adaptation
In many domains, we encounter data with very unbal-
anced class distribution. That is especially true for
applications in the medical domain, where most of the
instances are regular and only a small percent are as-
signed to an irregular class. Therefore, for most of the
classifiers that want to achieve high accuracy and low
complexity, it is most rational to classify all new in-
stances into a majority class. But that feature is not
desired, because we want to extract knowledge (espe-
cially when we want to explain a decision-making pro-
cess) and determine reasons for separation of classes.
To cope with the presented problem, we introduced an
instance reweighting method that works in a similar
manner, boosting but on a different level. Instances that
are rarely correctly classified gain importance. Fitness
criteria of individuals take importance into account and
force competition among individual induction methods.
Of course, there is a danger of over-adapting to the
187
TEAM LinG
 Combining Induction Methods with the Multimethod Approach

noise, but in that case, overall classification ability would
be decreased, and other induced classifiers can perform
better classification (self adaptation). We achieve a simi-
lar effect in boosting by concentrating on hard learnable
instances and not dismissing already extracted knowl-
edge.
EXPERIMENTAL RESULTS
Our current data-mining research is performed mainly
in the medical domain; thereafter, the knowledge repre-
sentation that should be in a human understandable form
is very important; so we are focused on the decision-
tree induction. To make objective assessment of our
method, a comparison of extracted knowledge used for
classification was made with reference methods C4.5,
C5/See5 without boosting, C5/See5 with boosting, and
genetic algorithm for decision-tree construction
(Podgorelec et al., 2001). The following quantitative
measures were used:
num of correctly classified objects
accuracy =
num. of all objects
num of correctly classified objects in class c
accuracyc =
num. of all objects in class c
medicine. For that reason, we have selected only symbolic
knowledge from a whole population of resulting solu-
tions.
A detailed description of databases can be found in
Leni and Kokol (2002). Other databases have been down-
loaded from the online repository of machine-learning
datasets maintained at UCI. We compared two variations
of our multi-method approach (MultiVeDec) with four
conventional approaches; namely, C4.5 (Quinlan, 1993),
C5/See5, Boosted C5, and genetic algorithm (Podgorelec
& Kokol, 2001). The results are presented in Table 1. Gray
marked fields represent the best method on a specific
database.
FUTURE TRENDS
As confirmed in many application domains, methods
that use only single approaches often can lead to local
optima and do not necessarily provide the big picture
about the problem. By applying different methods, in-
duction power of combined methods can supersede
single methods. Of course, there is no single way to
wave methods together. Our approach emphasizes modu-
larity based on knowledge sharing. Implicit induction
method knowledge is shared with others via produced
hypothesis. Synergy of methods also can be improved
by weaving implicit knowledge of induction method
learning algorithms, which requires very tight coupling
of two or many induction methods.

accuracy i
average class accuracy = i CONCLUSION
num. of classes
The success of the multi-method approach can be ex-
We decided to use average class accuracy instead of plained with the fact that some methods converge to
sensitivity and specificity that are usually used when local optima. With the combination of multiple meth-
dealing with medical databases. Experiments have been ods (operators) in different order, a better local (and
made with seven real-world databases from the field of hopefully global) solution can be found. Static hybrid

Table 1. A comparison of the multimethod approach to conventional approaches

C4.5 C5 C5 Boost Genetically SVM(RBF) Multimethod Multimethod
induced without
decision problem
trees adaption

av 76.7 44.5 81.4 48.6 83.7 50.0 90.7 71.4 83.7 55.7 93.0 78.7 90.0 71.4
breastcancer 96.4 94.4 95.3 95.0 96.6 96.7 96.6 95.5 96.5 97.1 96.6 95.1 96.9 97.8
heartdisease 74.3 74.4 79.2 80.7 82.2 81.1 78.2 80.2 39.6 42.7 78.2 78.3 83.1 82.8
Hepatitis 80.8 58.7 82.7 59.8 82.7 59.8 86.5 62.1 76.9 62.5 88.5 75.2 88.5 75.2
mracid 94.1 75.0 100.0 100.0 100.0 100. 0 94.1 75.0 94.1 75.0 100.0 100.0 100.0 100.0
Lipids 60.0 58.6 66.2 67.6 71.2 64.9 76.3 67.6 25.0 58.9 75.0 73.3 75.0 73.3
Mvp 91.5 65.5 91.5 65.5 92.3 70.0 91.5 65.5 36.1 38.2 93.8 84.3 93.8 84.3
accuracy on test set average class accuracy on test set

188

TEAM LinG
Combining Induction Methods with the Multimethod Approach
systems usually work sequentially or in parallel on the
fixed structure and order, performing whole tasks. On
the other hand, the multi-method approach works simul-
taneously with several methods on a single task (i.e.,
some parts are induced with different classical heuris-
tics, some parts with hybrid methods, and still other
parts with evolutionary programming). The presented
multi-method approach enables a quick and modular way
to integrate different methods into an existing system
and enables the simultaneous application of several
methods. It also enables partial application of method
operations to improve and recombine aspects and has no
limitation to the order and number of applied methods.
REFERENCES
Auer, P., Holte, R.C., & Cohen, W.W. (1995). Theory
and applications of agnostic PAC-learning with small
decision trees. Proceedings of the 12th International
Conference on Machine Learning.
Cox, M.T., & Ram, A. (1999). Introspective multistrategy
learning: On the construction of learning strategies.
Artificial Intelligence, 112, 1-55.
Dietterich, T.G. (2000). Ensemble methods in machine
learning. Proceedings of the First International Work-
shop on Multiple Classifier Systems.
Goldberg, D.E. (1989). Genetic algorithms in search,
optimization, and machine learning. Reading, MA:
Addison Wesley.
Iglesias, C.J. (1996). The role of hybrid systems in
intelligent data management: The case of fuzzy/neural
hybrids. Control Engineering Practice, 4(6), 839-845.
Leni , M., & Kokol, P. (2002). Combining classifiers with
multimethod approach. Proceedings of the Second Inter-
national Conference on Hybrid Intelligent Systems, Soft
Computing Systems: Design, Management and Applica-
tions, Frontiers in Artificial Intelligence and Applica-
tions, Amsterdam.
McGarry, K., Wermter, S., & MacIntyre, J. (2001). The
extraction and comparison of knowledge from local
function networks, International Journal of Computa-
tional Intelligence and Applications, 1(4), 369-382.
Podgorelec, V., & Kokol, P. (2001). Evolutionary deci-
sion forestsDecision making with multiple evolu-
tionary constructed decision trees: Problems in applied
mathematics and computational intelligence. World
Scientific and Engineering Society Press, 97-103.
Podgorelec, V., Kokol, P., Yamamoto, R., Masuda, G., &
Sakamoto, N. (2001). Knowledge discovery with geneti-
cally induced decision trees. Proceedings of the Interna-
tional ICSC Congress on Computational Intelligence: C
Methods and Applications (CIMA2001).
Quinlan, J.R. (1993). C4.5: Programs for machine learn-
ing. San Mateo, CA: Morgan Kaufmann Publishers.
progar, M., et al. (2000). Vector decision trees. Intel-
ligent Data Analysis, 4(3,4), 305-321.
Thrun, S., & Pratt, L. (Eds.). (1998). Learning to learn.
Kluwer Academic Publishers.
Todorovski, L., & Dzeroski, S. (2000). Combining mul-
tiple models with meta decision trees. Proceedings of
the Fourth European Conference on Principles of
Data Mining and Knowledge Discovery.
Valiant, L.G. (1984). A theory of the learnable. Communi-
cations of the ACM, 27(11), 1134-1142.
Vapnik, V.N. (1995). The nature of statistical learning
theory. New York: Springer Verlag.
Wolpert, D.H., & Macready, W.G. (1995). No free lunch
theorems for search. Technical Report SFI-TR-95-02-010,
Santa Fe, NM.
Zorman, M., Kokol, P., & Podgorelec, V. (2000). Medi-
cal decision making supported by hybrid decision trees.
Proceedings of ISA2000.
KEY TERMS
Boosting: Creation of ensemble of hypothesis to
convert weak learner to strong one by modifying ex-
pected instance distribution.
Induction Method: Process of learning, from cases
or instances, resulting in a general hypothesis of hidden
concept in data.
Method Level Operator: Partial operation of in-
duction/knowledge transformation level of specific in-
duction method.
Multi-Method Approach: Investigation of a research
question using a variety of research methods, each of
which may contain inherent limitations, with the expecta-
tion that combining multiple methods may produce con-
vergent evidence.
Population Level Operator: Unusually parameter-
ized operation that applies method level operators (pa-
rameter) to part or whole evolving population.
Transmutator: Knowledge transformation opera-
tor that modifies learner knowledge by exploring learner
experience.
189
TEAM LinG
190
Comprehensibility of Data Mining Algorithms
Zhi-Hua Zhou
Nanjing University, China
INTRODUCTION
Data mining attempts to identify valid, novel, potentially
useful, and ultimately understandable patterns from huge
volume of data. The mined patterns must be ultimately
understandable because the purpose of data mining is to
aid decision-making. If the decision-makers cannot un-
derstand what does a mined pattern mean, then the pattern
cannot be used well. Since most decision-makers are not
data mining experts, ideally, the patterns should be in a
style comprehensible to common people. So, comprehen-
sibility of data mining algorithms, that is, the ability of a
data mining algorithm to produce patterns understand-
able to human beings, is an important factor.
BACKGROUND
A data mining algorithm is usually inherently associated
with some representations for the patterns it mines. There-
fore, an important aspect of a data mining algorithm is the
comprehensibility of the representations it forms. That is,
whether or not the algorithm encodes the patterns it mines
in such a way that they can be inspected and understood
by human beings. Actually, such an importance has been
argued by a machine learning pioneer many years ago
(Michalski, 1983):
The results of computer induction should be symbolic
descriptions of given entities, semantically and
structurally similar to those a human expert might
produce observing the same entities. Components of
these descriptions should be comprehensible as single
chunks of information, directly interpretable in natural
language, and should relate quantitative and qualitative
concepts in an integrated fashion.
Craven and Shavlik (1995) have indicated a number of
concrete reasons why the comprehensibility of machine
learning algorithms is very important. With slight modifi-
cation, these reasons are also applicable to data mining
algorithms.
Validation: If the designers and end-users of a data
mining algorithm are to be confident in the perfor-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
mance of the algorithm, they must understand how
it arrives at its decisions.
Discovery: Data mining algorithms may play an impor-
tant role in the process of scientific discovery. An
algorithm may discover salient features and relation-
ships in the input data whose importance was not
previously recognized. If the patterns mined by the
algorithm are comprehensible, then these discoveries
can be made accessible to human review.
Explanation: In some domains, it is desirable to be
able to explain actions a data mining algorithm
suggest take for individual input patterns. If the
mined patterns are understandable in such a do-
main, then explanations of the suggested actions on
a particular case can be garnered.
Improving performance: The feature representa-
tion used for a data mining task can have a signifi-
cant impact on how well an algorithm is able to mine.
Mined patterns that can be understood and ana-
lyzed may provide insight into devising better fea-
ture representations.
Refinement: Data mining algorithms can be used to
refine approximately-correct domain theories. In
order to complete the theory-refinement process, it
is important to be able to express, in a comprehen-
sible manner, the changes that have been imparted
to the theory during mining.
MAIN THRUST
It is evident that data mining algorithms with good com-
prehensibility are very desirable. Unfortunately, most
data mining algorithms are not very comprehensible and
therefore their comprehensibility has to be enhanced by
extra mechanisms. Since there are many different data
mining tasks and corresponding data mining algorithms,
it is difficult for such a short article to cover all of them.
So, the following discussions are restricted to the compre-
hensibility of classification algorithms, but some essence
is also applicable to other kinds of data mining algorithms.
Some classification algorithms are deemed as compre-
hensible because the patterns they mine are expressed in
an explicit way. Representatives are decision tree algo-
rithms that encode the mined patterns in the form of a
TEAM LinG
Comprehensibility of Data Mining Algorithms

decision tree which can be easily inspected. Some other
classification algorithms are deemed as incomprehensible
because the patterns they mine are expressed in an implicit
way. Representatives are artificial neural networks that
encode the mined patterns in real-valued connection
weights. Actually, many methods have been developed to
improve the comprehensibility of incomprehensible classi-
fication algorithms, especially for artificial neural networks.
The main scheme for improving the comprehensibility
of artificial neural networks is rule extraction, that is,
extracting symbolic rules from trained artificial neural
networks. It originates from Gallants work on
connectionist expert system (Gallant, 1983). Good reviews
can be found in (Andrews, Diederich, & Tickle, 1995;
Tickle, Andrews, Golea, & Diederich, 1998). Roughly
speaking, current rule extraction algorithms can be cat-
egorized into four categories, namely the decompositional,
pedagogical, eclectic, or compositional algorithms. Each
category is illustrated with an example below.
The decompositional algorithms extract rules from
each unit in an artificial neural network and then aggre-
gate. A representative is the RX algorithm (Setiono, 1997),
which prunes the network and discretizes outputs of
hidden units for reducing computational complexity in
examining the network. If a hidden unit has many connec-
tions then it is split into several output units and some
new hidden units are introduced to construct a subnet-
work, so that the rule extraction process is iteratively
executed. The RX algorithm is summarized in Table 1.
The pedagogical algorithms regard the trained artifi-
cial neural network as an opaque and aim to extract rules
that map inputs directly into outputs. A representative is
the TREPAN algorithm (Craven & Shavlik, 1996), which
regards the rule extraction process as an inductive learn-
ing problem and uses oracle queries to induce an ID2-of-
3 decision tree that approximates the concept represented
by a given network. The pseudo-code of this algorithm is
shown in Table 2.
The eclectic algorithms incorporate elements of both
the decompositional and pedagogical ones. A represen- C
tative is the DEDEC algorithm (Tickle, Orlowski, &
Diederich, 1996), which extracts a set of rules to reflect the
functional dependencies between the inputs and the
outputs of the artificial neural networks. Fig. 1 shows its
working routine.
The compositional algorithms are not strictly
decompositional because they do not extract rules from
individual units with subsequent aggregation to form a
global relationship, nor do them fit into the eclectic cat-
egory because there is no aspect that fits the pedagogical
profile. Algorithms belonging to this category are mainly
designed for extracting deterministic finite-state automata
(DFA) from recurrent artificial neural networks. A repre-
sentative is the algorithm proposed by Omlin and Giles
(1996), which exploits the phenomenon that the outputs
of the recurrent state units tend to cluster, and if each
cluster is regarded as a state of a DFA then the relation-
ship between different outputs can be used to set up the
transitions between different states. For example, assum-
ing there are two recurrent state units s0 and s1, and their
outputs appear as nine clusters, then the working style of
the algorithm is shown in Fig. 2.
During the past years, powerful classification algo-
rithms have been developed in the ensemble learning area.
An ensemble of classifiers works through training mul-
tiple classifiers and then combining their predictions,
which is usually much more accurate than a single classi-
fier (Dietterich, 2002). However, since the classification is
made by a collection of classifiers, the comprehensibility
of an ensemble is poor even when its component classi-
fiers are comprehensible.
A pedagogical algorithm has been proposed by Zhou,
Jiang, and Chen (2003) to improve the comprehensibility
of ensembles of artificial neural networks, which utilizes
the trained ensemble to generate instances and then
extracts symbolic rules from them. The success of this

Table 1. The RX algorithm

1. Train and prune the artificial neural network.

2. Discretize the activation values of the hidden units by clustering.
3. Generate rules that describe the network outputs using the discretized activation values.
4. For each hidden unit:
1) If the number of input connections is less than an upper bound, then extract rules to describe
the activation values in terms of the inputs.
2) Else form a subnetwork:
(a) Set the number of output units equal to the number of discrete activation values.
Treat each discrete activation value as a target output.
(b) Set the number of input units equal to the inputs connected to the hidden units.
(c) Introduce a new hidden layer.
(d) Apply RX to this subnetwork.
5. Generate rules that relate the inputs and the outputs by merging rules generated in Steps 3 and 4.

191

TEAM LinG
 Comprehensibility of Data Mining Algorithms

Table 2. The TREPAN algorithm

TREPAN(training_examples, features)
Queue
for each example E training_examples
E.label ORACLE(E)
initialize the root of the tree, T, as a leaf node
put <T, training_examples, {}> into Queue
while Queue and size(T) < tree_size_limit
remove node N from head of Queue
examplesN example set stored with N
constraintsN constraint set stored with N
use features to build set of candidate splits
use examplesN and calls to ORACLE(constraintsN) to evaluate splits
S best binary split
search for best MOFN splits, S, using S as a seed
make N an internal node with split S
for each outcome, s, of S
make C, a new child node of N
constraintsC constraintsN {S = s}
use calls to ORACLE(constraintsC) to determine if C should remain a leaf
otherwise
examplesC members of examplesN with outcome s on split S
put <C, examplesC, constraintsC> into Queue
return T

Figure 1. Working routine of the DEDEC algorithm

analyze the weights to obtain search for functional depen-
a rank of the inputs according dencies between the important
to their relative importance in inputs and the output, then
predicting the output extract the corresponding rules

ANN Training Weight Vector Analysis Iterative FD/Rule Extraction

Figure 2. The working style of Omlin and Giless algorithm

s1 s1 s1

1 2 1 1 2 1 1 2 1

0.5
3 4 5 3 4

0 0 0
0 0.5 1 s0 0 0.5 1 s0 0 0.5 1 s0

4 4 5

2
2 2
1
1 1
3 3

(a) All the possible transitions (b) All the possible transitions (c) All the possible transitions
from state 1 from state 2 from states 3 and 4

192

TEAM LinG
Comprehensibility of Data Mining Algorithms
algorithm suggests that research on improving compre-
hensibility of artificial neural networks can give illumina-
tion to the improvement of comprehensibility of other
complicated classification algorithms.
Recently, Zhou & Jiang (2003) proposed to combine
ensemble learning and rule induction algorithms to obtain
accurate and comprehensible classifiers. Their algorithm
uses an ensemble of artificial neural networks as a data
preprocessing mechanism for the induction of symbolic
rules. Later, they (Zhou & Jiang, 2004) presented a new
decision tree algorithm and shown that when the en-
semble is significantly more accurate than the decision
tree directly grown from the original training set and the
original training set has not fully captured the target
distribution, using an ensemble as the preprocessing
mechanism is beneficial. These works suggest the twice-
learning paradigm to develop accurate and comprehen-
sible classifiers, that is, using coupled classifiers where a
classifier devotes to the accuracy while the other devotes
to the comprehensibility.
FUTURE TRENDS
It was supposed that an algorithm which could produce
explicitly expressed patterns is comprehensible. How-
ever, such a supposition might not be so valid as it
appears to be. For example, as for a decision tree contain-
ing hundreds of leaves, whether or not it is comprehen-
sible? A quantitative answer might be more feasible than
a qualitative one. Thus, quantitative measure of compre-
hensibility is needed. Such a measure can also help solve
a long-standing problem, that is, how to compare the
comprehensibility of different algorithms.
Since rule extraction is an important scheme for im-
proving the comprehensibility of complicated data mining
algorithms, frameworks for evaluating the quality of ex-
tracted rules are important. Actually, the FACC (Fidelity,
Accuracy, Comprehensibility, Consistency) framework
proposed by Andrews, Diederich, and Tickle (1995) has
been used for almost a decade, which contains two impor-
tant criteria, i.e. fidelity and accuracy. Recently, Zhou
(2004) identified the fidelity-accuracy dilemma which
indicates that in some cases pursuing high fidelity and
high accuracy simultaneously is impossible. Therefore,
new evaluation frameworks have to be developed and
employed, while the ACC (eliminating Fidelity from FACC)
framework suggested by Zhou (2004) might be a good
candidate.
Most current rule extraction algorithms suffer from
high computational complexity. For example, in
decompositional algorithms, if all the possible relation-
ships between the connection weights and units in a
trained artificial neural network are considered, then com-
binatorial explosion is inevitable for even moderate-sized
networks. Although many mechanisms such as pruning C
have been employed to reduce the computational com-
plexity, the efficiency of most current algorithms is not
good enough. In order to work well in real-world applica-
tions, effective algorithms with better efficiency are
needed.
Until now almost all works on improving comprehen-
sibility of complicated algorithms rely on rule extraction.
Although symbolic rule is relatively easy to be under-
stood by human beings, it is not the only comprehensible
style that could be exploited. For example, visualization
may provide good insight into a pattern. However, al-
though there are a few works (Frank & Hall, 2004; Melnik,
2002) utilizing visualization techniques to improve the
comprehensibility of data mining algorithms, few work
attempts to exploit together rule extraction and visualiza-
tion, which is evidently very worth exploring.
Previous research on comprehensibility has mainly
focused on classification algorithms. Recently, some works
on improving the comprehensibility of complicated re-
gression algorithms have been presented (Saito & Nakano,
2002; Setiono, Leow, & Zurada, 2002). Since complicated
algorithms exist extensively in data mining, more sce-
narios besides classification should be considered.
CONCLUSION
This short article briefly discusses complexity issues in
data mining. Although there is still a long way to produce
patterns that can be understood by common people in any
data mining tasks, endeavors on improving the compre-
hensibility of complicated algorithms have paced a prom-
ising way. It could be anticipated that experiences and
lessons learned from these research might give illumina-
tion on how to design data mining algorithms whose
comprehensibility is good enough, not needed to be
further improved. Only when the comprehensibility is not
a problem, the fruits of data mining can be fully enjoyed.
REFERENCES
Andrews, R., Diederich, J., & Tickle, A.B. (1995). Survey
and critique of techniques for extracting rules from trained
artificial neural networks. Knowledge-Based Systems, 8(6),
373-389.
Craven, M.W., & Shavlik, J.W. (1995). Extracting compre-
hensible concept representations from trained neural
networks. In Working Notes of the IJCAI95 Workshop on
Comprehensiblility in Machine Learning (pp. 61-75),
Montreal, Canada.
193
TEAM LinG

Craven, M.W., & Shavlik, J.W. (1996). Extracting tree-
structured representations of trained networks. In D.
Touretzky, M. Mozer, & M. Hasselmo (Eds.), Advances in
Neural Information Processing Systems, 8 (pp. 24-30).
Cambridge, MA: MIT Press.
Dietterich, T.G. (2002). Ensemble learning. In M.A. Arbib
(Ed.), The handbook of brain theory and neural networks
(2nd ed.). Cambridge, MA: MIT Press.
Frank, E., & Hall, M. (2003). Visualizing class probability
estimation. In N. Lavra , D. Gamberger, H. Blockeel, & L.
Todorovski (Eds.), Lecture Notes in Artificial Intelli-
gence, 2838. Berlin: Springer, 168-179.
Gallant, S.I. (1983). Connectionist expert systems. Com-
munications of the ACM, 31(2), 152-169.
Melnik, O. (2002). Decision region connectivity analysis:
a method for analyzing high-dimensional classifiers.
Machine Learning, 48(1-3), 321-351.
Michalski, R. (1983). A theory and methodology of induc-
tive learning. Artificial Intelligence, 20(2), 111-161.
Omlin, C.W., & Giles, C.L. (1996). Extraction of rules from
discrete-time recurrent neural networks. Neural Networks,
9(1), 41-52.
Saito, K., & Nakano, R. (2002). Extracting regression rules
from neural networks. Neural Networks, 15(10), 1279-
1288.
Setiono, R. (1997). Extracting rules from neural networks
by pruning and hidden-unit splitting. Neural Computa-
tion, 9(1), 205-225.
Setiono, R., Leow, W.K., & Zurada, J.M. (2002). Extraction
of rules from artificial neural networks for nonlinear re-
gression. IEEE Transactions on Neural Networks, 13(3),
564-577.
Tickle, A.B., Andrews, R., Golea, M., & Diederich, J.
(1998). The truth will come to light: directions and chal-
lenges in extracting the knowledge embedded within
trained artificial neural networks. IEEE Transactions on
Neural Networks, 9(6), 1057-1067.
Tickle, A.B., Orlowski, M., & Diederich, J. (1996). DEDEC:
A methodology for extracting rule from trained artificial
neural networks. In Proceedings of the AISB96 Work-
shop on Rule Extraction from Trained Neural Networks
(pp. 90-102), Brighton, UK.
Zhou, Z.-H. (2004). Rule extraction: Using neural net-
works or for neural networks? Journal of Computer Sci-
ence and Technology, 19(2), 249-253.
194
Comprehensibility of Data Mining Algorithms
Zhou, Z.-H., & Jiang, Y. (2003). Medical diagnosis with
C4.5 rule preceded by artificial neural network ensemble.
IEEE Transactions on Information Technology in Bio-
medicine, 7(1), 37-42.
Zhou, Z.-H., & Jiang, Y. (2004). NeC4.5: neural ensemble
based C4.5. IEEE Transactions on Knowledge and Data
Engineering, 16(6), 770-773.
Zhou, Z.-H., Jiang, Y., & Chen, S.-F. (2003). Extracting
symbolic rules from trained neural network ensembles. AI
Communications, 16(1), 3-15.
KEY TERMS
Accuracy: The measure of how well a pattern can
generalize. In classification it is usually defined as the
percentage of examples that are correctly classified.
Artificial Neural Networks: A system composed of
many simple processing elements operating in parallel
whose function is determined by network structure, con-
nection strengths, and the processing performed at com-
puting elements or units.
Comprehensibility: The understandability of a pat-
tern to human beings; the ability of a data mining algo-
rithm to produce patterns understandable to human be-
ings.
Decision Tree: A flow-chart-like tree structure, where
each internal node denotes a test on an attribute, each
branch represents an outcome of the test, and each leaf
represents a class or class distribution.
Ensemble Learning: A machine learning paradigm
using multiple learners to solve a problem.
Fidelity: The measure of how well the rules extracted
from a complicated model mimic the behavior of that
model.
MOFN Expression: A boolean expression consisted
of an integer threshold m and n boolean antecedents,
which is fired when at least m antecedents are fired. For
example, the MOFN expression 2-of-{ a, b, c } is logically
equivalent to (a b) (a c) ( b c).
Rule Extraction: Given a complicated model such as
an artificial neural network and the data used to train it,
produce a symbolic description of the model.
Symbolic Rule: A pattern explicitly comprising an
antecedent and a consequent, usually in the form of IF
THEN .
TEAM LinG
Comprehensibility of Data Mining Algorithms

Twice-Learning: A machine learning paradigm using

coupled learners to achieve two aspects of advantages. In C
its original form, two classifiers are coupled together
where one classifier is devoted to the accuracy while the
other devoted to the comprehensibility.

195

TEAM LinG
 196

Computation of OLAP Cubes

Amin A. Abdulghani
Quantiva, USA

INTRODUCTION total sales) is aggregated over all locations. A cuboid is

The focus of Online Analytical Processing (OLAP) is to
provide a platform for analyzing data (e.g., sales data) with
multiple dimensions (e.g., product, location, time) and
multiple measures (e.g., total sales or total cost). OLAP
operations then allow viewing of this data from a number
of perspectives. For analysis, the object or data structure
of primary interest in OLAP is a cube.
BACKGROUND
An n-dimensional cube is defined as a group of k-dimen-
sional (k<=n) cuboids arranged by the dimensions of the
data. A cell represents an association of a measure m (e.g.,
total sales) with a member of every dimension (e.g.,
product=toys, location=NJ, year=2003). By defini-
tion, a cell in an n-dimensional cube can have fewer
dimensions than n. The dimensions not present in the cell
are aggregated over all possible members. For example,
you can have a two-dimensional (2-D) cell,
C1(product=toys, year=2003). Here, the implicit value
for the dimension location is *, and the measure m (e.g.,
a group-by of a subset of dimensions, obtained by aggre-
gating all tuples on these dimensions. In an n-dimensional
cube, a cuboid is a base cuboid if it has exactly n dimen-
sions. If the number of dimensions is fewer than n, then
it is an aggregate cuboid. Any of the standard aggregate
functions such as count, total, average, minimum, or
maximum can be used for aggregating. Figure 1 shows an
example of a three-dimensional (3-D) cube, and Figure 2
shows an aggregate 2-D cuboid.
In theory, no special operators or SQL extensions are
required to take a set of records in the database and
generate all the cells for the cube. Rather, the SQL group-
by and union operators can be used in conjunction with
d sorts of the dataset to produce all cuboids. However,
such an approach would be very inefficient, given the
obvious interrelationships between the various group-
bys produced.
MAIN THRUST
I now describe the essence of the major methods for the
computation of OLAP cubes.

Figure 1. A 3-D cube that consists of 1-D, 2-D, and 3-D Figure 2. An example 2-D cuboid on (product, year) for
cuboids the 3-D cube in Figure 1 (location='*'); total sales needs
to be aggregated (e.g., SUM)
dimension

Measure(total
sales) 2003

2003
2002
year

year

2002

PA 2001
2.5M
2001 NY
location toy clothes cosmetic
NJ nn s s
toy clothes cosmetic product
s product s

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
Computation of OLAP Cubes
Top-Down Computation
In a seminal paper, Gray, Bosworth, Layman, and Pirahesh
(1996) proposed the data cube operator as a means of
simplifying the process of data cube construction. The
algorithm presented forms the basis of the top-down
approach. In the approach, the aggregation functions are
categorized into three classes:
Distributive: An aggregate function F is called
distributive if there exists a function g such that the
value of F for an n-dimensional cell can be computed
by applying g to the value of F in an (n + 1)
dimensional cell. Examples of such functions in-
clude SUM and COUNT. For example, the COUNT()
of an n-dimensional cell can be computed by apply-
ing SUM to the value of COUNT() in an (n+1)
dimensional cell.
Algebraic: An aggregate function F is algebraic if
F of an n-dimensional cell can be computed by using
a constant number of aggregates of the (n + 1)
dimensional cell. An example is the AVERAGE()
function. The AVERAGE() of an n-dimensional cell
can be computed by taking the sum and count of the
(n+1)dimensional cell and then dividing the SUM
by the COUNT to produce the global average.
Holistic: An aggregate function F is called holistic
if the value of F for an n-dimensional cell cannot be
computed from a constant number of aggregates of
the (n+1)dimensional cell. Median and mode are
examples of holistic functions.
The top-down cube computation works with distribu-
tive or algebraic functions. These functions have the
property that more detailed aggregates (i.e., more dimen-
sions) can be used to compute less detailed aggregates.
This property induces a partial-ordering (i.e., a lattice) on
all the group-bys of the cube. A group-by is called a child
of some parent group-by if the parent can be used to
Figure 3. Cube lattice
ABCD
ABC ABD ACD BCD
AB AC AD BC BD CD
A B C D
all
compute the child (and no intermediate group-bys exist
between the parent and child). Figure 3 depicts a sample C
lattice where A, B, C, and D are dimensions, nodes repre-
sent group-bys, and the edges show the parent-child
relationship.
The basic idea for top-down cube construction is to
start by computing the base cuboid (group-by for which
no cube dimensions are aggregated). A single pass is
made over the data, a record is examined, and the appro-
priate base cell is incremented. The remaining group-bys
are computed by aggregating over already computed finer
grade group-by. If a group-by can be computed from one
or more possible parent group-bys, then the algorithm
uses the parent smallest in size. For example, for comput-
ing the cube ABCD, the algorithm starts out by computing
the cuboids for ABCD. Then, using ABCD, it computes
the cuboids for ABC, ABD, and BCD. The algorithm then
repeats itself by computing the 2-D cuboids, AB, BC, AD,
and BD. Note that the 2-Dcuboids can be computed from
multiple parents. For example, AB can be computed from
ABC or ABD. The algorithm selects the smaller group-by
(the group-by with the fewest number of cells). An ex-
ample top-down cube computation is shown in Figure 4.
Variants of this approach optimize on additional costs.
The best-known methods are the PipeSort and PipeHash
(Agarwal, Agrawal, Deshpande, Gupta, Naughton,
Ramakrishnan, et al., 1996). The basic idea of both algo-
rithms is that a minimum spanning tree should be gener-
ated from the original lattice such that the cost of travers-
ing edges will be minimized. The optimizations for the
costs that these algorithms include are as follows:
Cache-results: This optimization aims at ensuring
that the result of a group-by is cached (in memory)
so other group-bys can use it in the future.
Amortize-scans: This optimization amortizes the
cost of a disk read by computing the maximum
possible number of group-bys together in memory.
Share-sorts: For a sort-based algorithm, this aims
at sharing sorting cost across multiple group-bys.
Figure 4. Top-down cube computation
ABCD
ABC ABD ACD BCD
AB AC AD BC BD CD
A B C D
all
197
TEAM LinG

Share-partitions: For a hash-based algorithm, when
the hash-table is too large to fit in memory, data are
partitioned and aggregated to fit in memory. This can
be achieved by sharing this cost across multiple
group- bys.
Both PipeSort and PipeHash are examples of ROLAP
(Relational-OLAP) algorithms as they operate on multidi-
mensional tables. An alternative is MOLAP (Multidimen-
sional OLAP), where the approach is to store the cube
directly as a multidimensional array. If the data are in
relational form, then they are loaded into arrays, and the
cube is computed from the arrays. The advantage is that
no tuples are needed for comparison purposes; all opera-
tions are performed by using array indices.
A good example for a MOLAP-based cubing algorithm
is the MultiWay algorithm (Zhao, Deshpande, & Naughton,
1997). Here, to make efficient use of memory, a single large
d-dimensional array is divided into smaller d-dimensional
arrays, called chunks. It is not necessary to keep all chunks
in memory at the same time; only part of the group-by
arrays are needed for a given instance. The algorithm starts
at the base cuboid with a single chunk. The results are
passed to immediate lower-level cuboids of the top-down
cube computation tree. After the chunk processing is
complete, the next chunk is loaded. The process then
repeats itself. An advantage of the MultiWay algorithm is
that it also allows for shared computation, where interme-
diate aggregate values are reused for the computation of
successive descendant cuboids. The disadvantage of
MOLAP-based algorithms is that they are not scalable for
a large number of dimensions and sparse data.
Bottom-Up Computation
Computing and storing full cubes may be overkill. Con-
sider, for example, a dataset with 10 dimensions, each with
a domain size of 9 (i.e., can have nine distinct values).
Computing the full cube for this dataset would require
computing and storing 1010 cuboids. Not all cuboids in the
full cube may be of interest. For example, if the measure is
total sales, and a user is interested in finding the cuboids
that have high sales (greater than a given threshold), then
computing cells with low total sales would be redundant.
It would be more efficient here to only compute the cells
that have total sales greater than the threshold. The group
of cells that satisfies the given query is referred to as an
iceberg cube. The query returning the group of cells is
referred to as an iceberg query.
Iceberg queries are typically computed using a bottom-
up approach (Beyer & Ramakrishnan, 1999). These meth-
ods work from the bottom of the cube lattice and then work
their way up towards the cells with a larger number of
dimensions. Figure 5 illustrates this approach. The ap-
198
Computation of OLAP Cubes
Figure 5. Bottom-up cube computation
ABCD
ABC ABD ACD BCD
AB AC AD BC BD CD
A B C D
all
proach is typically more efficient for high-dimensional,
sparse datasets. The key observation to its success is in
its ability to prune cells in the lattice that do not lead to
useful answers. Suppose, for example, that the iceberg
query is COUNT()>100. Also, suppose there is a cell
X(A=a1, B=b1) with COUNT=75. Then, clearly, C does
not satisfy the query. However, in addition, you can see
that any cell X in the cube lattice that includes the
dimensions (e.g. A=a1, B=b1) will also not satisfy the
query. Thus, you need not look at any such cell after
having looked at cell X. This forms the basis of support
pruning. The observation was first made in the context of
frequent sets for the generation of association rules
(Agrawal, Imielinski, & Swami, 1993).
For more arbitrary queries, detecting whether a cell is
prunable is more difficult. The fundamental property that
allows pruning is called monotonicity of the query: Let
D be a database and XD be a cuboid. A query Q) is
monotonic at X if the condition Q(X) is FALSE implies
that Q(X) is FALSE for any XX. With this definition,
a cell X in the cube would be pruned if the query Q is
monotonic at X. However, determining whether a query
Q is monotonic in terms of this definition is an NP-hard
problem for many simple class of queries (Imielinski,
Khachiyan, & Abdulghani, 2002). To work around this
problem, another notion of monotonicity referred to as
view monotonicity of a query is introduced. Suppose
you have cuboids defined on a set S of dimension and
measures. A view V on S is an assignment of values to the
elements of the set. If the assignment holds for the
dimensions and measures in a given cell X, then V is a
view for X on the set S. So, for example, in a cube of rural
buyers, if the average sale of bread is 15 with 20 people,
then the view on the set {areaType, COUNT(),
AVG(salesBread)} for the cube is {areaType=rural,
COUNT()=20, AVG(salesBread)=15}. Extending the defi-
nition, a view on a query Q for X is an assignment of
values for the set of dimension and measure attributes of
the query expression.
TEAM LinG
Computation of OLAP Cubes
A query Q() view monotonic on view V(Q,X) if for any
cell X in any database D such that V is the view for X, the
condition Q is FALSE, for X implies Q is FALSE for all X
X. An important property of view monotonicity is that
the time and space required for checking it for a query
depends on the number of terms in the query and not on
the size of the database or the number of its attributes.
Because most queries typically have few terms, it would
be useful in many practical situations. Imielinski et al.
(2002) presents a method for checking view monotonicity
for a query that includes constraints of type (Agg {<, >,
=, !=} c), where c is a constant and Agg can be MIN, SUM,
MAX, AVERAGE, COUNT, aggregates that are higher
order moments about the origin, or aggregates that are an
integral of a function on a single attribute.
Hybrid Approach
Typically, for low-dimension, low-cardinality, dense
datasets, the top-down approach is more applicable than
the bottom-up one. However, combining the two ap-
proaches leads to an even more efficient algorithm (Xin,
Han, Li, & Wah, 2003). On the global computation order,
the work presented uses the top-down approach. At a
sublayer underneath, it exploits the potential of the bot-
tom-up model. Consider the top-down computation tree in
Figure 4. Notice that the dimension ABC is included for all
the cuboids in the leftmost subtree. Similarly, all the
cuboids in the second subtree include the dimensions
AB. These common dimensions are termed the shared
dimensions of the particular subtrees and enable bottom-
up computation. The observation is that if a query is
FALSE and (view)-monotonic on the cell defined by the
shared dimensions, then the rest of the cells generated
from this shared dimension are unneeded. The critical
requirement is that for every cell X, the cell for the shared
dimensions must be computed first. The advantage of
such an approach is that it allows for shared computation
as in the top-down approach as well as for pruning, when
possible.
Other Approaches
Until now, I have considered computing the cuboids from
the base data. Another commonly used approach is to
materialize the results of a selected set of group-bys and
evaluate all queries by using the materialized results.
Harinarayan, Rajaraman, and Ullman (1996) describe an
approach to materialize a limit of k group-bys. The first
group-by to materialize always includes the top group-by,
as none of the group-bys can be used to answer queries
for this group-by. The next group-by to materialize is
included such that the benefit of including it in the set
exceeds the benefit of any other nonmaterialized group-
by. Let S be the set of materialized group-bys. This benefit C
of including a group-by v in the set S is the total savings
achieved for computing the group-bys not included in S
by using v versus the cost of computing them through
some group-by already in S. Gupta, Harinarayan,
Rajaraman, and Ullman (1997) further extend this work to
include indices in the cost.
The subset of the cuboids selected for materialization
is referred to as a partial cube. There have been efficient
approaches suggested in the literature for computing the
partial cube. In one such approach suggested by Dehne,
Eavis, and Rau-Chaplin (2004), the cuboids are computed
in a top-down fashion. The process starts with the original
lattice or the PipeSort spanning tree of the original lattice,
organizes the selected cuboids into a tree of minimal cost,
and then further tries to reduce the cost by possibly
adding intermediate nodes.
After a set of cuboids has been materialized, queries
are evaluated by using the materialized results. Park, Kim,
and Lee (2001) describe a typical approach. Here, the
OLAP queries are answered in a three-step process. In the
first step, it selects the materialized results that will be
used for rewriting and identifies the part of the query
(region) that the materialized result can answer. Next,
query blocks are generated for these query regions. Fi-
nally, query blocks are integrated into a rewritten query.
FUTURE TRENDS
In this paper, I focus on the basic aspects of cube compu-
tation. The field is pretty recent, going back to no more
than 10 years. However, as the field is beginning to
mature, issues are becoming better understood. Some of
the issues that will get more attention in future work
include:
Advanced data structures for organizing input tuples
of the input cuboids (Han, Pei, Dong, & Wang, 2001;
Xin et al., 2003).
Making use of inherent property of the dataset to
reduce computation of the data cubes. An example
is the range cubing algorithm, which utilizes the
correlation in the datasets to reduce the computa-
tion cost (Feng, Agrawal, Abbadi, & Metwally,
2004).
Compressing the size of the data cube and storing
it efficiently. A good example is a quotient cube,
which partitions the cube into classes such that
each cell in a class has the same aggregate value,
and the lattice generated preserves the original
cubes semantics (Lakshmanan, Pei, & Han, 2002).
199
TEAM LinG

Additionally, I believe that future work in this direc-
tion would attack the problem from multiple issues rather
than a single focus. Sismanis, Deligiannakis,
Roussopoulus, and Kotidis (2002) describe one such
work, Dwarf. Its architecture integrates multiple fea-
tures including the compressing of data cubes, a tunable
parameter for controlling the amount of materialization,
and indexing and support for incremental updates (which
is important for the case when the underlying data are
periodically updated).
CONCLUSION
This paper focuses on the methods for OLAP cube com-
putation. All methods share the similarity that they make
use of the ordering defined by the cube lattice to drive the
computation. For the top-down approach, traversal oc-
curs from the top of the lattice. This has the advantage
that for the computation of successive descendant
cuboids, intermediate node results are used. The bottom-
up approach traverses the lattice in the reverse direction.
The method can no longer rely on making use of the
intermediate node results. Its advantage lies in the ability
to prune cuboids in the lattice that do not lead to useful
answers. The hybrid approach uses the combination of
both methods, taking advantages of both.
The other major option for computing the cubes is to
materialize only a subset of the cuboids and to evaluate
queries by using this set. The advantage lies in storage
costs, but additional issues, such as identification of the
cuboids to materialize, algorithms for materializing these,
and query rewrite algorithms, are raised.
REFERENCES
Agarwal, S., Agrawal, R., Deshpande, P., Gupta, A.,
Naughton, J. F., Ramakrishnan, R., et al. (1996). On the
computation of multidimensional aggregates. Proceed-
ings of the International Conference on Very Large Data
Bases (pp. 506-521), India.
Agrawal, R., Imielinski, T., & Swami, A. N. (1993). Mining
association rules between sets of items in large data-
bases. Proceedings of the ACM SIGMOD Conference (pp.
207-216), USA.
Beyer, K. S., & Ramakrishnan, R. (1999). Bottom-up com-
putation of sparse and iceberg cubes. Proceedings of the
ACM SIGMOD Conference (pp. 359-370), USA.
Dehne, F. K. H., Eavis, T., & Rau-Chaplin, A. (2004). Top-
down computation of partial ROLAP data cubes. Pro-
200
Computation of OLAP Cubes
ceedings of the Hawaii International Conference on
System Sciences, USA.
Feng, Y., Agrawal, D., Abbadi, A. E., & Metwally, A.
(2004). Range CUBE: Efficient cube computation by ex-
ploiting data correlation. Proceedings of the Interna-
tional Conference on Data Engineering (pp. 658-670),
USA.
Gray, J., Bosworth, A., Layman, A., & Pirahesh, H. (1996).
Data cube: A relational aggregation operator generalizing
group-by, cross-tab, and sub-total. Proceedings of the
International Conference on Data Engineering (pp. 152-
159), USA.
Gupta, H., Harinarayan, V., Rajaraman, A., & Ullman, J. D.
(1997). Index selection for OLAP. Proceedings of the
International Conference on Data Engineering (pp. 208-
219), UK.
Han, J., Pei, J., Dong, G., & Wang, K. (2001). Efficient
computation of iceberg cubes with complex measures.
Proceedings of the ACM SIGMOD Conference (pp. 1-12),
USA.
Harinarayan, V., Rajaraman, A., & Ullman, J. D. (1996).
Implementing data cubes efficiently. Proceedings of the
ACM SIGMOD Conference (pp. 205-216), Canada.
Imielinski, T., Khachiyan, L., & Abdulghani, A. (2002).
Cubegrades: Generalizing association rules. Journal of
Data Mining and Knowledge Discovery, 6(3), 219-257.
Lakshmanan, L. V. S., Pei, J., & Han, J. (2002). Quotient
cube: How to summarize the semantics of a data cube.
Proceedings of the International Conference on Very
Large Data Bases (pp. 778-789), China.
Park, C.-S., Kim, M. H., & Lee, Y.-J. (2001). Rewriting OLAP
queries using materialized views and dimension hierar-
chies in data warehouses. Proceedings of the Interna-
tional Conference on Data Engineering, Germany.
Sismanis, Y., Deligiannakis, A., Roussopoulus, N., &
Kotidis, Y. (2002). Dwarf: Shrinking the petacube. Pro-
ceedings of the ACM SIGMOD Conference (pp. 464-475),
USA.
Xin, D., Han, J., Li, X., & Wah, B. W. (2003). Star-cubing:
Computing iceberg cubes by top-down and bottom-up
integration. Proceedings of the International Confer-
ence on Very Large Data Bases (pp. 476-487), Germany.
Zhao, Y., Deshpande, P. M., & Naughton, J. F. (1997). An
array-based algorithm for simultaneous multidimensional
aggregates. Proceedings of the ACM SIGMOD Confer-
ence (pp. 159-170), USA.
TEAM LinG
Computation of OLAP Cubes
KEY TERMS
Algebraic Function: An aggregate function F is alge-
braic if F of an n-dimensional cuboid can be computed by
using a constant number of aggregates of the (n + 1)
dimensional cuboid.
Bottom-Up Cube Computation: Cube construction
that starts by computing from the bottom of the cube
lattice and then working up toward the cells with a greater
number of dimensions.
Cube Cell: Represents an association of a measure m
with a member of every dimension.
Cuboid: A group-by of a subset of dimensions, ob-
tained by aggregating all tuples on these dimensions.
Distributive Function: An aggregate function F is
called distributive if there exists a function g such that the
value of F for an n-dimensional cuboid can be computed
by applying g to the value of F in (n + 1)dimensional
cuboid. C
Holistic Function: An aggregate function F is called
holistic if the value of F for an n-dimensional cell cannot
be computed from a constant number of aggregates of the
(n+1)dimensional cell.
N-Dimensional Cube: A group of k-dimensional (k<=n)
cuboids arranged by the dimension of the data.
Partial Cube: The subset of the cuboids selected for
materialization.
Sparse Cube: A cube is sparse if a high ratio of the
cubes possible cells does not contain any measure value.
Top-Down Cube Computation: Cube construction that
starts by computing the base cuboid and then iteratively
computing the remaining cells by aggregating over al-
ready computed finer-grade cells in the lattice.
201
TEAM LinG
202
Concept Drift
Marcus A. Maloof
Georgetown University, USA
INTRODUCTION
Traditional approaches to data mining are based on an
assumption that the process that generated or is gener-
ating a data stream is static. Although this assumption
holds for many applications, it does not hold for many
others. Consider systems that build models for identify-
ing important e-mail. Through interaction with and feed-
back from a user, such a system might determine that
particular e-mail addresses and certain words of the
subject are useful for predicting the importance of e-
mail. However, when the user or the persons sending e-
mail start other projects or take on additional responsi-
bilities, what constitutes important e-mail will change.
That is, the concept of important e-mail will change or
drift. Such a system must be able to adapt its model or
concept description in response to this change. Coping
with or tracking concept drift is important for other
applications, such as market-basket analysis, intrusion
detection, and intelligent user interfaces, to name a few.
BACKGROUND
Concept drift may occur suddenly, what some call revo-
lutionary drift. Or it may occur gradually, what some
call evolutionary drift (Klenner & Hahn, 1994). Drift
may occur at different time scales and at varying rates.
Concepts may change and then reoccur, perhaps with
contextual cues (Widmer, 1997). For example, the con-
cept of warm is different in the summer than in the
winter. A contextual cue or variable is perhaps the
season or the daily mean temperature. It helps identify
the appropriate model to use for determining if, for
example, it is a warm day. Coping with concept drift
requires an online approach, meaning that the system
must mine a stream of data. If drift occurs slowly
enough, distinguishing between real and virtual concept
drift may be difficult (Klinkenberg & Joachims, 2000).
The former occurs when concepts indeed change over
time; the latter occurs when performance drops during
the normal online process of building and refining a
model. Such drops could be due to differences in the
data in different parts of the stream. The richness of this
problem has led to an equally rich collection of ap-
proaches.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Research on the problem of concept drift has been
empirical and theoretical (see Kuh, Petsche, & Rivest,
1991; Mesterharm, 2003); however, the focus of this
article is on empirical approaches. In this regard, re-
searchers have evaluated such approaches by using syn-
thetic data sets (for examples, see Maloof & Michalski,
2004; Schlimmer & Grainger, 1986; Street & Kim,
2001; Widmer & Kubat, 1996) and real data sets (for
examples, see Black & Hickey, 2002; Blum, 1997; Lane
& Brodley, 1998), both small (Maloof & Michalski,
2004; Schlimmer & Grainger, 1986; Widmer & Kubat,
1996) and large (Hulten, Spencer, & Domingos, 2001;
Kolter & Maloof, 2003; Street & Kim, 2001; Wang,
Fan, Yu, & Han, 2003).
Systems for coping with concept drift fall into three
broad categories: incremental, partial memory, and en-
semble. Incremental approaches use new instances to
modify existing models. Partial-memory approaches
maintain a subset of previously encountered instances,
previously built and refined models, or both. When new
instances arrive, such systems use the new instances and
their store of past instances and models to build new
models or refine existing ones. Finally, ensemble meth-
ods build and maintain multiple models to cope with
concept drift. Naturally, systems designed for concept
drift do not always fall neatly into only one of these
categories. For example, some partial-memory ap-
proaches are also incremental (Maloof & Michalski,
2004; Widmer & Kubat, 1996).
A model or concept description is a generalized
representation of instances or training data. Such repre-
sentations are important for prediction and for better
understanding the data set from which they were built.
Researchers have used a variety of representations for
drifting concepts, including the instances themselves,
probabilities, linear equations, decision trees, and deci-
sion rules. Methods for inducing and building such
representations include instance-based learning, nave
Bayes, support vector machines, C4.5, and AQ15, re-
spectively. See Hand, Mannila, and Smyth (2001) for
additional information.
MAIN THRUST
Researchers have proposed a variety of approaches for
learning concepts that drift. However, evaluating such
TEAM LinG
Concept Drift
approaches is a critical issue. In the next two sections, I
survey approaches for learning concepts that change
over time and discuss issues of evaluating such ap-
proaches.
Survey of Approaches for Concept Drift
STAGGER (Schlimmer & Grainger, 1986) was the first
system for coping with concept drift. Its model consists
of nodes, corresponding to features and class labels,
linked together with probabilistic arcs, representing the
strength of association between features and class labels.
As STAGGER processes new instances, it increases or
decreases probabilities, and it may add nodes and arcs. To
classify an unknown instance, STAGGER predicts the
most probable class.
Partial-memory approaches maintain a store of par-
tially built models, a portion of the previously encoun-
tered instances, or both. Such approaches vary in how
they use such information for adjusting current models.
The FLORA Systems (Widmer & Kubat, 1996) maintain
a sequence of examples over a dynamically adjusted
window of time. The Window Adjustment Heuristic
(WAH) adjusts the size of this window in response to
performance changes. Generally, if performance is de-
creasing or poor, then the heuristic reduces the windows
size; if it is increasing or acceptable, then it increases
the size. These systems also maintain a store of rules,
including ones that are overly general, although these
are not used for prediction. As the systems process
instances, they create new rules or refine existing ones.
To classify an instance, the FLORA systems select the
rule that best matches and return its class label.
The AQ-PM Systems maintain a set of examples
over a window of time, but the systems select examples
from the boundaries of rules, so they can retain ex-
amples that do not reoccur in the data stream. AQ-PM
(Maloof & Michalski, 2000) builds new rules when new
examples arrive, whereas AQ11-PM (Maloof &
Michalski, 2004) refines existing rules. These systems
maintain examples over a static window of time, but
AQ11-PM+WAH (Maloof, 2003) incorporates Widmer
and Kubats (1996) WAH for dynamically sizing this
window. Because these systems use rules, when classi-
fying an instance, they return as their prediction the
class label of the best matching rule.
The Concept-adapting Very Fast Decision Tree sys-
tem (Hulten et al., 2001), or CVFDT, progressively
grows a decision tree downward from the leaf nodes. It
maintains frequency counts for attribute values by class
and extends the tree when a statistical test indicates that
a change has occurred. CVFDT also maintains at each
node a list of alternate subtrees, which it swaps with the
current subtree when it detects drift. To classify an
unknown instance, the method uses the instances values
to traverse the current tree from the root to a leaf node, C
returning as the prediction the associated label.
The Concept Drift 3 system (Black & Hickey, 1999),
or CD3, uses batches of instances annotated with a time
stamp of either current or new to build a decision tree.
When drift occurs, time becomes more relevant for
prediction, so the time-stamp attribute will appear higher
in the decision tree. After pruning, CD3 converts the
tree to rules by enumerating all paths containing a new
time stamp and then removing conditions involving the
time stamp. CD3 predicts the class of the best matching
rule.
Ensemble methods maintain a set of models and use
a voting procedure to yield a global prediction. Blums
(1997) implementation of Weighted-Majority
(Littlestone & Warmuth, 1994) uses as models histo-
ries of labels associated with pairs of features. If a
models features are present in an instance, then it
predicts the most frequent label present in its history.
The method initializes each model with a weight of 1 and
reduces a models weight if it predicts incorrectly. It
predicts based on a weighted vote of the predictions of
the models.
The Streaming Ensemble Algorithm (Street & Kim,
2001) maintains a fixed-size collection of models, each
built from a fixed number of instances. When a new
batch of instances arrives, SEA builds a new model. If
space exists in the collection, then it adds the new
model. Otherwise, it replaces the worst performing
model with the new model, if one exists. SEA predicts
the majority vote of the predictions of the models in the
collection.
The Accuracy-Weighted Ensemble (Wang et al.,
2003) also maintains a fixed-size collection of models,
each built from a batch of instances. However, this
method weights each classifier in the collection based
on its performance on the most recent batch. When
adding a new weighted model, if there is no space in the
collection, then the method stores only the top weighted
models. The method predicts based on a weighted-
majority vote of the predictions of the models in the
collection.
Dynamic Weighted Majority (Kolter & Maloof,
2003), or DWM, maintains a collection of weighted
models but dynamically adds and removes models with
changes in performance. Instead of building a single
model with each batch, DWM uses new instances to
refine all the models in the collection. Each time a
model predicts incorrectly, DWM reduces its weight,
and DWM removes a model from the collection if its
weight falls below a threshold. Like the previous method,
DWM predicts based on a weighted-majority vote of the
predictions of the models, but if the global prediction
203
TEAM LinG

(i.e., the weighted-majority vote) is incorrect for an
instance, then DWM adds a new model to the collection.
Evaluation
Evaluating systems for concept drift is a critical issue.
Ideally, one wants to use a real-world data set, but finding
such a data set in which concept drift is easy to identify
is itself a challenge. After all, if it were easy to detect
concept drift in large data sets, then the task of writing
systems to cope with it would be trivial. Even if one
establishes that drift is occurring in a data set, to conduct
a proper evaluation, the phenomenon must produce a
measurable effect in a methods performance. More-
over, the effect of concept drift on the methods perfor-
mance must be greater than all other effects, such as that
due to the variability from processing new examples of
a target concept. As a result of these issues, the majority
of evaluations have involved synthetic or artificial data
sets.
A hallmark of synthetic data sets for concept drift is
that they cycle through a series of target concepts. The
first target concept persists for a period of time, then the
second persists, and so on. For each time step of a
period, one randomly generates a set of training ex-
amples, which the method uses to build or refine its
models. The method evaluates the resulting model on the
examples in a test set and computes a measure of perfor-
mance, such as predictive accuracy (i.e., the percentage
of the test examples the method predicted correctly).
One generates test cases every time step or every time
period. As the system processes examples, ideally, its
performance on the test set improves at a certain rate and
to a certain level of accuracy. Indeed, the slope and
asymptote are critical characteristics of a methods per-
formance.
Crucially, each time the target concept changes, one
generates a new set of testing examples. Naturally, when
the method applies the model built for the previous
target concept to the examples of the new concept, the
methods performance will be poor. However, as the
method processes training examples of the new target
concept, as before, performance will improve with some
slope and to some asymptote.
By far, the most widely used synthetic data set is the
so-called STAGGER Concepts. Originally used by
Schlimmer and Grainger (1986), it has been the center-
piece for many evaluations (e.g., Kolter & Maloof, 2003;
Maloof & Michalski, 2000, 2004; Widmer, 1997;
Widmer & Kubat, 1996). There are three attributes: size,
color, and shape. Size can be small, medium, or large;
color can be red, blue, or green; shape can be triangle,
circle, or rectangle. There are three target concepts, and
the presentation of examples lasts for 120 time steps.
204
Concept Drift
The target concept for the first 40 time steps is [size =
small] & [color = red]. For the next 40, it is [color =
green] [shape = circle]. For the final 40, it is [size =
medium large]. At each time step, one generates a
single training example and 100 test cases of the target
concept. Naturally, the method updates its model by
using the training example and evaluates it by using the
testing examples, calculating accuracy. One presenta-
tion of the STAGGER Concepts is not sufficient for a
proper evaluation, so researchers conduct multiple
runs, averaging accuracy at each time step.
Clearly, a small problem with only 27 possible
examples, researchers have recently proposed larger
synthetic data sets involving concept drift (e.g., Hulten
et al., 2001; Street & Kim, 2001; Wang et al., 2003). I
do not have the space here to survey the details, simi-
larities, and differences of these synthetic problems,
but they are all based on the same ideas present in the
STAGGER Concepts. For instance, researchers have
used rotating (Hulten et al., 2001) and shifting (Street
& Kim, 2001) hyperplanes as changing target concepts.
As noted previously, there have also been evalua-
tions involving concept drift in real data sets. Blum
(1997) used a calendar-scheduling task in which a
users preference for meetings changed over time.
Lane and Brodley (1998) examined an intrusion-detec-
tion application, mining sequences of UNIX commands.
Finally, Black and Hickey (2002) studied concept drift
in the phone records of customers of British Telecom.
FUTURE TRENDS
Tracking concept drift is a rich problem that has led to
a diverse set of approaches and evaluation methodolo-
gies, and there are many opportunities for further in-
vestigation. One is the development of better methods
for evaluating systems that cope with evolutionary
drift. I have already discussed the difficult nature of
this problem: To evaluate how systems cope with drift,
there must be a measurable effect. Slow or evolution-
ary drift may not produce an effect different enough
from that of mining the sequence of instances. If noise
is present, then it is even more difficult to distinguish
among the variability due to instances, drift, and noise.
Existing systems are probably capable of tracking slow
concept drift, but we need evaluation methodologies
for measuring how such drift affects performance.
I have made the case for the importance of strong
evaluations, and in this regard, researchers need to
better place their work in context with past efforts.
Presently, researchers often choose not to use data
sets from past studies, and create new ones for their
investigation. Because new studies typically consist of
TEAM LinG
Concept Drift
new methods evaluated on new data sets, it is difficult to
place new methods in context with previous work. As a
consequence, it is presently impossible to truly under-
stand the strengths and weaknesses of existing methods,
both old and new. This is not to say that researchers
should not create or introduce new data sets. Indeed, I
have already mentioned that the STAGGER Concepts is
a small problem, so creating a new, larger one is re-
quired if, say, researchers want to establish how meth-
ods scale to larger data streams. Nonetheless, by first
evaluating new methods on existing problems, research-
ers will be able to make stronger conclusions about the
performance of their method, and the community will
be able to better understand the contribution of the
method.
Finally, we need to develop a systems theory of
concept drift. Presently, we have little to guide the
development of future methods or to guide the selection
of a particular method for a new application. However,
before developing such a theory, we will need to develop
better methods for evolutionary concept drift, and we
will need stronger evaluations that place new work in
context with that of the past.
CONCLUSION
Tracking concept drift is important for many applica-
tions, from e-mail sorting to market-basket analysis.
Concepts may change quickly or gradually; they may
occur once or at regular intervals. The diversity and
complexity of coping with concept drift has led re-
searchers to propose an equally varied set of approaches.
Some modify their models, some do so with partial
memory of the past, and some rely on groups of models.
Although there have been evaluations of these approaches
involving real data sets, the majority have involved
synthetic data sets, which give researchers great control
in testing hypotheses. With stronger evaluations that
place new systems in context with past work, we will be
able to propose theories for systems that cope with
concept drift, theories that we can test and that will lead
to new systems for this critical problem for many appli-
cations.
REFERENCES
Black, M., & Hickey, R. (1999). Maintaining the perfor-
mance of a learned classifier under concept drift. Intel-
ligent Data Analysis, 3, 453-474.
Black, M., & Hickey, R. (2002). Classification of customer
call data in the presence of concept drift and noise. In
Lecture Notes in Computer Science: Vol. 2311. Software
2002: Computing in an imperfect world (pp. 74-87). New C
York: Springer.
Blum, A. (1997). Empirical support for Winnow and
Weighted-Majority algorithms: Results on a calendar
scheduling domain. Machine Learning, 26, 5-23.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of
data mining. Cambridge, MA: MIT Press.
Hulten, G., Spencer, L., & Domingos, P. (2001). Mining
time-changing data streams. Proceedings of the Sev-
enth ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (pp. 97-106).
Klenner, M., & Hahn, U. (1994). Concept versioning: A
methodology for tracking evolutionary concept drift in
dynamic concept systems. Proceedings of the 11th
European Conference on Artificial Intelligence, En-
gland, XX (pp. 473-477).
Klinkenberg, R., & Joachims, T. (2000). Detecting concept
drift with support vector machines. Proceedings of the
17th International Conference on Machine Learning
(pp. 487-494), USA.
Kolter, J., & Maloof, M. (2003). Dynamic weighted major-
ity: A new ensemble method for tracking concept drift.
Proceedings of the Third IEEE International Conference
on Data Mining (pp. 123-130), USA.
Kuh, A., Petsche, T., & Rivest, R. L. (1991). Learning time-
varying concepts. In Advances in neural information
processing systems 3 (pp. 183-189). San Francisco: Mor-
gan Kaufmann.
Lane, T., & Brodley, C. (1998). Approaches to online
learning and concept drift for user identification in
computer security. Proceedings of the Fourth Interna-
tional Conference on Knowledge Discovery and Data
Mining (pp. 259-263), USA.
Littlestone, N., & Warmuth, M. K. (1994). The Weighted
Majority algorithm. Information and Computation, 108,
212-261.
Maloof, M. (2003). Incremental rule learning with partial
instance memory for changing concepts. Proceedings of
the International Joint Conference on Neural Networks
(pp. 2764-2769), USA.
Maloof, M., & Michalski, R. (2000). Selecting examples for
partial memory learning. Machine Learning, 41, 27-52.
Maloof, M., & Michalski, R. (2004). Incremental learning
with partial instance memory. Artificial Intelligence, 154,
95-126.
205
TEAM LinG

Mesterharm, C. (2003). Tracking linear-threshold con-
cepts with Winnow. Journal of Machine Learning Re-
search, 4, 819-838.
Schlimmer, J., & Granger, R. (1986). Beyond incremental
processing: Tracking concept drift. Proceedings of the
Fifth National Conference on Artificial Intelligence (pp.
502-507), USA.
Street, W., & Kim, Y. (2001). A streaming ensemble algo-
rithm (SEA) for large-scale classification. Proceedings of
the Seventh ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (pp. 377-382),
USA.
Wang, H., Fan, W., Yu, P., & Han, J. (2003). Mining
concept-drifting data streams using ensemble classifi-
ers. Proceedings of the Ninth ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and Data
Mining (pp. 226-235), USA.
Widmer, G. (1997). Tracking context changes through
meta-learning. Machine Learning, 27, 259-286.
Widmer, G., & Kubat, M. (1996). Learning in the presence
of concept drift and hidden contexts. Machine Learning,
23, 69-101.
206
Concept Drift
KEY TERMS
Class Label: A label identifying the concept or
class of an instance.
Concept Drift: A phenomenon in which the class
labels of instances change over time.
Data Set: A set of instances of target concepts.
Data Stream: A data set distributed over time.
Instance: A set of attributes, their values, and a class
label. Also called example, record, or case.
Model: A representation of a data set or stream that
one can use for prediction or for better understanding
the data from which it was derived. Also called hypoth-
esis or concept description. Models include probabili-
ties, linear equations, decision trees, decision rules,
and the like.
Synthetic Data Set: A set of artificial target con-
cepts.
Target Concept: The true, typically unknown model
underlying a data set or data stream.
TEAM LinG

Condensed Representations for Data Mining
Jean-Francois Boulicaut
INSA de Lyon, France
INTRODUCTION
Condensed representations have been proposed in
Mannila and Toivonen (1996) as a useful concept for the
optimization of typical data-mining tasks. It appears as a
key concept within the inductive database framework
(Boulicaut et al., 1999; de Raedt, 2002; Imielinski & Mannila,
1996), and this article introduces this research domain, its
achievements in the context of frequent itemset mining
(FIM) from transactional data, and its future trends.
Within the inductive database framework, knowledge
discovery processes are considered as querying pro-
cesses. Inductive databases (IDBs) contain not only data,
but also patterns. In an IDB, ordinary queries can be used
to access and manipulate data, while inductive queries
can be used to generate (mine), manipulate, and apply
patterns. To motivate the need for condensed represen-
tations, let us start from the simple model proposed in
Mannila and Toivonen (1997). Many data-mining tasks
can be abstracted into the computation of a theory. Given
a language L of patterns (e.g., itemsets), a database
instance r (e.g., a transactional database) and a selection
predicate q, which specifies whether a given pattern is
interesting or not (e.g., the itemset is frequent in r), a data-
mining task can be formalized as the computation of
Th(L,q,r) = { L | q(,r) is true}. This also can be
considered as the evaluation for the inductive query q.
Notice that it specifies that every pattern that satisfies q
has to be computed. This completeness assumption is
quite common for local pattern discovery tasks but is
generally not acceptable for more complex tasks (e.g.,
accuracy optimization for predictive model mining). The
selection predicate q can be defined in terms of a Boolean
expression over some primitive constraints (e.g., a mini-
mal frequency constraint used in conjunction with a
syntactic constraint, which enforces the presence or the
absence of some subpatterns). Some of the primitive
constraints generally refer to the behavior of a pattern in
the data by using the so-called evaluation functions (e.g.,
frequency).
To support the whole knowledge discovery process,
it is important to support the computation of many differ-
ent but correlated theories.
It is well known that a generate-and-test approach that
would enumerate the sentences of L and then test the
selection predicate q is generally impossible. A huge
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
207
C
effort has been made by data-mining researchers to make
an active use of the primitive constraints occurring in q to
achieve a tractable evaluation of useful mining queries. It
is the domain of constraint-based mining (e.g., the seminal
paper) (Ng et al., 1998). In real applications, the computa-
tion of Th(L,q,r) can remain extremely expensive or even
impossible, and the framework of condensed representa-
tions has been designed to cope with such a situation.
The idea of -adequate representations was introduced in
Mannila and Toivonen (1996) and Boulicaut and Bykowski
(2000). Intuitively, they are alternative representations of
the data that enable answering to a class of query (e.g.,
frequency queries for itemsets in transactional data) with
a bounded precision. At a given precision , one can be
interested in the smaller representations, which are then
called concise or condensed representations. It means
that a condensed representation for Th(L,q,r) is a collec-
tion C Th(L,q,r) such that every pattern from Th(L,q,r)
can be derived efficiently from C. In the database-mining
context, where r might contain a huge volume of records,
we assume that efficiently means without further access
to the data. The following figure illustrates that we can
compute Th(L,q,r) either directly (Arrow 1) or by means of
a condensed representation (Arrow 2) followed by a
regeneration phase (Arrow 3).
We know several examples of condensed representa-
tions for which Phases 2 and 3 are much less expensive
than Phase 1. We now introduce the background for
understanding condensed representations in the well
studied context of FIM.
Figure 1.
Condensed representation C for Th(L,q,r)
3
2
1
Data r Theory Th(L,q,r)
TEAM LinG

BACKGROUND
In many cases (e.g., itemsets, inclusion dependencies,
sequential patterns) and for a given selection predicate or
constraint, the search space L is structured by an anti-
monotonic specialization relation, which provides a lat-
tice structure. For instance, in transactional data, when L
is the power set of items, and the selection predicate
enforces a minimal frequency, set inclusion is such an
anti-monotonic specialization relation. Anti-monotonic-
ity means that when a sentence does not satisfy q (e.g.,
an itemset is not frequent), then none of its specializations
can satisfy q (e.g., none of its supersets are frequent). It
becomes possible to prune huge parts of the search space,
which cannot contain interesting sentences. This has
been studied a lot within the learning as search frame-
work (Mitchell, 1982), and the generic level-wise algo-
rithm from Mannila and Toivonen (1997) has inspired
many algorithmic developments. It computes Th(L,q,r)
level-wise in the lattice by considering first the most
general sentences (e.g., the singleton in the FIM prob-
lem). Then, it alternates candidate evaluation (e.g., fre-
quency counting) and candidate generation (e.g., build-
ing larger itemsets from discovered frequent itemsets)
phases. The algorithm stops when it cannot generate new
candidates or, in other terms, when the most specific
sentences have been found (e.g., all the maximal frequent
itemsets). This collection of the most specific sentences
is called a positive border in Mannila and Toivonen
(1997), and it corresponds to the S set of a Version Space
in Mitchells terminology. The a priori algorithm (Agrawal
et al., 1996) is clearly the most famous instance of this
level-wise algorithm.
The dual property of monotonicity is interesting, as
well. A selection predicate is monotonic when its negation
is anti-monotonic (i.e., when a sentence satisfies it, all its
specializations satisfy it, as well). In the itemset pattern
domain, the maximal frequency constraint or a syntactic
constraint that enforces that a given item belongs to the
itemsets are two monotonic constraints. Thanks to the
duality of these definitions, a monotonic constraint gives
rise to a border G, which contains the minimally general
sentences w.r.t., the monotonic constraint (see the fol-
lowing figure).
Figure 2.
Specialization Border S
relation
Border G
208
Condensed Representations for Data Mining
When the predicate selection is a conjunction of an
anti-monotonic part and a monotonic part, the two bor-
ders define the solution set: solutions are between S and
G, and (S,G) is a version space. For this conjunction case,
several algorithms can be used (Bucila et al., 2002; Bonchi,
2003; de Raedt & Kramer, 2001; Jeudy & Boulicaut, 2002).
When arbitrary Boolean combinations of anti-monotonic
and monotonic constraints are used (e.g., disjunctions),
the solution space is defined as a union of several version
spaces (i.e., unions of couples of borders) (de Raedt et al.,
2002).
Borders appear as a typical case of condensed repre-
sentation. Assume that the collection of the maximal
frequent itemsets in r is available (i.e., the S border for the
minimal frequency constraint); this collection is generally
several orders of magnitude smaller than the complete
collection of the frequent itemsets in r, while all of them
can be generated from S without any access to the data.
However, in most of the applications of pattern discovery
tasks, the user not only wants to get the interesting pat-
terns, but also wants the results of some evaluation func-
tions about these patterns. This is obvious for the FIM
problem; these patterns are generally exploited in a post-
processing step to derive more useful statements about the
data (e.g., the popular frequent association rules that have
a high enough confidence) (Agrawal et al., 1996). This can
be done efficiently, if we compute not only the collection
of frequent itemsets, but also their frequencies.
In fact, the semantics of an inductive query are better
captured by extended theories (i.e., collections like {(,e)
LE | q(,r) est vrai et e=(,r)}), where e is the result
of an evaluation function in r with values in E. In our FIM
problem, denotes the frequency (e is a number in [0,1])
of an itemset in a transactional database r. The challenge
of designing condensed representations for an extended
theory ThE is then to identify subsets of ThE, from which
it is possible to generate ThE either exactly or with an
approximation on the evaluation functions.
MAIN RESULTS
We emphasize the main results concerning condensed
representations for frequent itemsets, since this is the
context for which it has been studied a lot.
Condensed Representations by
Borders
It makes sense to use borders as condensed representa-
tions. For FIM, specific algorithms have been designed
for computing directly the S border (Bayardo, 1998). Also,
the algorithm in Kramer and de Raedt (2001) computes
TEAM LinG
Condensed Representations for Data Mining
borders S and G and has been applied successfully to
feature extraction in the domain of molecular fragment
finding. In this case, a conjunction of a minimal frequency
in one set of molecule (e.g., the active ones) and a maximal
frequency in another set of molecules (e.g., the inactive
ones) is used. This kind of research is related to the so-
called emergent pattern discovery (Dong & Li, 1999).
Considering the extended theory for frequent itemsets,
it is clear that, given the maximal frequent sets and their
frequencies, we have an approximate condensed represen-
tation of the frequent itemsets. Without looking at the
data, we can regenerate the whole collection of the fre-
quent itemsets (subsets of the maximal ones), and we have
a bounded error on their frequencies: when considering a
subset of a maximal -frequent itemset, we know that its
frequency is in [,1]. Even though more precise bounds
can be computed, this approximation is useless in practice.
Indeed, when using borders, users have other applications
in mind (e.g., feature construction).
The maximal frequent itemsets can be computed in
cases where very large frequent itemsets hold such that
the regeneration process becomes impossible. Typically,
when a maximal frequent itemset has a size 30, it should lead
to the regeneration of around 1010 frequent sets.
Exact Condensed Representations of
Frequent Sets and Their Frequencies
Since Boulicaut and Bykowski (2000), a lot of attention
has been put on exact condensed representations based on
frequent closed sets. According to the classical framework
of Galois connection, the closure of an itemset X in r,
closure(X,r), is the maximal superset of X, which has the
same frequency as X in r. Furthermore, a set X is closed if
X=closure(X,r). Interestingly, the sets of itemsets that have
the same closures constitute equivalence classes of itemsets
that have the same frequencies, the maximal one in each
equivalence class being a closed set (Bastide et al., 2000).
Frequent closed sets are itemsets that are both closed
and frequent. In dense and/or highly correlated data, we
can have orders of magnitude less frequent closed sets
than frequent itemsets. In other terms, it makes sense to
materialize and, when possible, to look for the fastest
computations of the frequent closed sets only. It is then
easy to derive the frequencies of every frequent itemset
without any access to the data. Many algorithms have
been designed for computing frequent closed itemsets
(Pasquier et al., 1999; Zaki, 2002). Empirical evaluations of
many algorithms for the FIM problem are reported in
Goethals and Zaki (2004). Frequent closed set mining is a
real breakthrough to the computational complexity of FIM
in difficult contexts.
A specificity of frequent closed set mining algorithms
is the need for a characterization of (frequent) closed set
generators. Interestingly, these generators constitute
condensed representations, as well. An important char- C
acterization is the one of free sets (Boulicaut et al., 2000),
which has been proposed independently under the name
key patterns (Bastide et al., 2000). By definition, the
closures of (frequent) free sets are (frequent) closed sets.
Given the quivalence classes we quoted earlier, free sets
are their minimal elements, and the freeness property can
lead to efficient pruning, thanks to its anti-monotonicity.
Computing the frequent free sets plus an extra collection
of some non-free itemsets (part of the so-called negative
border of the frequent free sets), it is possible to regen-
erate the whole collection of the frequent itemsets and
their frequencies (Boulicaut et al., 2000; Boulicaut et al.,
2003). On one hand, we often have much more frequent
free sets than frequent closed sets but, on another hand,
they are smaller. The concept of freeness has been
generalized for other exact condensed representations
like the disjunct-free itemsets (Bykowski & Rigotti, 2001),
the non derivable itemsets (Calders & Goethals, 2002),
and the minimal k-free representations of frequent sets
(Calders & Goethals, 2003). Regeneration algorithms and
translations between condensed representations have
been studied, as well (Kryszkiewicz et al., 2004).
Approximate Condensed
Representations for Frequent Sets and
Their Frequencies
Looking for approximate condensed representations can
be useful for very hard datasets. We mentioned that
borders can be considered as approximate condensed
representations but with a poor approximation of the
needed frequencies. The idea is to be able to compute the
frequent itemsets with less counting at the price of an
acceptable approximation on their frequencies. One idea
is to compute the frequent itemsets and their frequencies
on a sample of the original data (Mannila & Toivonen,
1996), but bounding the error on the frequencies is hard.
In Boulicaut et al. (2000), the concept of -free set was
introduced. In fact, the free itemsets are a special case
when = 0. When > 0, we have less -free sets than free
sets, and, thus, the representation is more condensed.
Interestingly, experimentations on real datasets have
shown that the error in practice was much smaller than
the theoretical bound (Boulicaut et al., 2000; Boulicaut et
al., 2003). Another family of approximate condensed
representations has been designed in Pei, et al. (2002),
where the idea is to consider the maximal frequent itemsets
for different frequency thresholds and then approximate
the frequency of any frequent set by means of the
computed frequencies.
209
TEAM LinG

Multiples Uses of Condensed
Representations
Another interesting achievement is that condensed rep-
resentations of frequent itemsets are not only useful for
FIM in difficult cases, but they also derive more meaning-
ful patterns. In other terms, instead of a regeneration
phase, which can be impossible, however, due to the size
of the collection of frequent itemsets, it is possible to use
directly the condensed representations. Indeed, closed
sets can be used to derive informative or non-redundant
association rules. Also, frequent and valid association
rules with a minimal left-hand side can be derived from d-
free sets, and these rules can be used, among others, for
association-based classification. It is also possible to
derive formal concepts in Willes terminology and, thus,
apply the so-called formal concept analysis.
FUTURE TRENDS
So far, we consider that two promising directions of
research are considered and should provide new break-
throughs. First, it is useful to consider new mining tasks
and design condensed representations for them. For
instance, Casali, et al. (2003) consider the computation of
aggregates from data cubes, and Yan, et al. (2003) address
sequential pattern mining. An interesting open problem is
to use condensed representations during model mining
(e.g., classifiers). Then, merging condensed representa-
tions with constraint-based mining (with various con-
straints, including constraints that are neither anti-mono-
tonic nor monotonic) seems to be a major issue. One
challenging problem is to decide which kind of condensed
representation has to be materialized for optimizing se-
quences of inductive queries (e.g., for interactive associa-
tion rule mining); that is, the real context of knowledge
discovery processes.
CONCLUSION
We introduced the key concept of condensed represen-
tations for the optimization of data-mining queries and,
thus, the development of the inductive database frame-
work. We have summarized an up-to-date view on bor-
ders. Then, referencing the work on the various con-
densed representations for frequent itemsets, we have
pointed out the breakthrough of the computational com-
plexity of the FIM tasks. This is important, because the
applications of frequent itemsets go much further than the
classical association rule-mining task. Finally, the con-
cepts of condensed representations and -adequate rep-
210
Condensed Representations for Data Mining
resentations are quite general and might be considered
with success for many other pattern domains.
REFERENCES
Agrawal, R. et al. (1996). Fast discovery of association
rules. In (Ed.), Advances in knowledge discovery and
data mining (pp. 307-328). MIT Press.
Bastide, Y., Taouil, R., Pasquier, N., Stumme, G., & Lakhal,
L. (2000). Mining frequent patterns with counting infer-
ence. SIGKDD Explorations, 2(2), 66-75.
Bayardo, R. (1998). Efficiently mining long patterns from
databases. Proceedings of the International Conference
on Management of Data, Seattle, Washington.
Bonchi, F. (2003). Frequent pattern queries: Language
and optimisations [doctoral thesis]. Pisa, Italy: Univer-
sity of Pisa, Italy.
Boulicaut, J.-F. & Bykowski, A. (2000). Frequent closures
as a concise representation for binary data mining. Pro-
ceedings of the Knowledge Discovery and Data Mining,
Current Issues and New Applications, Kyoto, Japan.
Boulicaut, J.-F., Bykowski, A., & Rigotti, C. (2000). Ap-
proximation of frequency queries by means of free-sets.
Proceedings of Principles of Data Mining and Knowl-
edge Discovery, Lyon, France.
Boulicaut, J.-F., Bykowski, A., & Rigotti, C. (2003). Free-
sets: A condensed representation of Boolean data for the
approximation of frequency queries. Data Mining and
Knowledge Discovery, 7(1), 5-22.
Boulicaut, J.-F., Klemettinen, M., & Mannila, H. (1999).
Modeling KDD processes within the inductive database
framework. Proceedings of the Data Warehousing and
Knowledge Discover, Florence, Italy.
Bucila, C., Gehrke, J., Kifer, D., & White, W.M. (2003).
DualMiner: A dual-pruning algorithm for itemsets with
constraints. Data Mining and Knowledge Discovery,
7(3), 241-272.
Bykowski, A., & Rigotti, C. (2001). A condensed represen-
tation to find frequent patterns. Proceedings of the ACM
Principles of Database Systems, Santa Barbara, California.
Calders, T., & Goethals, B. (2002). Mining all non-derivable
frequent itemsets. Proceedings of the Principles of Data
Mining and Knowledge Discovery, Helsinki, Finland.
Calders, T., & Goethals, B. (2003). Minimal k-free repre-
sentations of frequent sets. Proceedings of the Prin-
ciples of Data Mining and Knowledge Discovery,
Dubrovnik, Croatia.
TEAM LinG
Condensed Representations for Data Mining
Casali, A., Cicchetti, R., & Lakhal, L. (2003). Cube lattices:
A framework for multidimensional data mining. Proceed-
ings of the SIAM International Conference on Data
Mining, San Francisco, California.
de Raedt, L. (2002). A perspective on inductive databases.
SIGKDD Explorations, 4(2), 66-77.
de Raedt, L., Jger, M., Lee, S.D., & Mannila, H. (2002). A
theory of inductive query answering. Proceedings of the
IEEE International Conference on Data Mining,
Maebashi City, Japan.
de Raedt, L., & Kramer, S. (2001). The levelwise version
space algorithm and its application to molecular fragment
finding. Proceedings of the International Joint Confer-
ence on Artificial Intelligence, Seattle, Washington.
Dong, G., & Li., J. (1999). Efficient mining of emerging
patterns: Discovering trends and differences. Proceed-
ings of the International Conference on Knowledge
Discovery and Data Mining, San Diego, Caifornia.
Goethals, B., & Zaki, M.J. (2004). Advances in frequent
itemset mining implementations. SIGKDD Explorations,
6(1), 109-117.
Imielinski, T., & Mannila, H. (1996). A database perspec-
tive on knowledge discovery. Communications of the
ACM, 39(11), 58-64.
Jeudy, B., & Boulicaut, J.-F. (2002). Optimization of asso-
ciation rule mining queries. Intelligent Data Analysis,
6(4), 341-357.
Kryszkiewicz, M., Rybinski, H., & Gajek, M. (2004). Dataless
transitions between concise representations of frequent
patterns. Intelligent Information Systems, 22(1), 41-70.
Mannila, H., & Toivonen, H. (1996). Multiple uses of
frequent sets and condensed representations. Proceed-
ings of the International Conference on Knowledge
Discovery and Data Mining, Portland, Oregon.
Mannila, H., & Toivonen, T. (1997). Levelwise search and
borders of theories in knowledge discovery. Data Mining
and Knowledge Discovery, 1(3), 241-258.
Mitchell, T.M. (1982). Generalization as search. Artificial
Intelligence, 18, 203-226.
Ng, R., Lakshmanan, L.V.S., Han, J., & Pang, A. (1998).
Exploratory mining and pruning optimizations of con-
strained associations rules. Proceedings of the Interna-
tional Conference on Management of Data, Seattle,
Washington. C
Pasquier, N., Bastide, Y., Taouil, R., & Lakhal, L. (1999).
Efficient mining of association rules using closed itemset
lattices. Information Systems, 24(1), 25-46.
Pei, J., Dong, G., Zou, W., & Han, J. (2002). On computing
condensed frequent pattern bases. Proceedings of the
IEEE International Conference on Data Mining,
Maebashi City, Japan.
Yan, X., Han, J., & Afshar, R. (2003). CloSpan: Mining
closed sequential patterns in large databases. Proceed-
ings of the SIAM International Conference on Data
Mining, San Francisco, California.
Zaki, M.J., & Hsiao, C.J. (2002). CHARM: An efficient
algorithm for closed itemset mining. Proceedings of the
SIAM International Conference on Data Mining, Arling-
ton, Texas.
KEY TERMS
Condensed Representations: Alternative representa-
tions of the data that preserve crucial information for
being able to answer some kind of queries. The most
studied example concerns frequent sets and their frequen-
cies. Their condensed representations can be several
orders of magnitude smaller than the collection of the
frequent itemsets.
Constraint-Based Data Mining: Concerns the active
use of constraints that specify the interestingness of
patterns. Technically, it needs strategies to push the
constraints, or at least part of them, deeply into the data-
mining algorithms.
Inductive Databases: An emerging research domain,
where knowledge discovery processes are considered as
querying processes. Inductive databases contain both data
and patterns, or models, which hold in the data. They are
queried by means of more or less ad-hoc query languages.
Pattern Domains: A pattern domain is the definition
of a language of patterns, a collection of evaluation
functions that provide properties of patterns in database
instances, and the kinds of constraints that can be used
to specify pattern interestingness.
211
TEAM LinG
212
Content-Based Image Retrieval
Timo R. Bretschneider
Nanyang Technological University, Singapore
Odej Kao
University of Paderborn, Germany
INTRODUCTION
Sensing and processing multimedia information is one of
the basic traits of human beings: The audiovisual system
registers and transports surrounding images and sounds.
This complex re-cording system, complemented by the
senses of touch, taste, and smell, enables perception and
provides humans with data for analysing and interpreting
the environment. Imitating this perception and the simu-
lation of the processing was and still is one of the major
leitmotifs of multimedia technology developments. The
goal is to find a representation for every type of knowl-
edge, which makes the reception and processing of infor-
mation as easy as possible. The need to process given
information, deliver it, and explain it to a certain audience
exists in nearly all areas of day-to-day life: commerce,
science, education, and entertainment (Smeulders,
Worring, Santini, Gupta, & Jain, 2000).
The development of digital technologies and applica-
tions allowed the production of huge amounts of multime-
dia data. This information has to be systematically col-
lected, registered, organised, and classified. Furthermore,
search procedures, methods to formulate queries, and
ways to visualise the results have to be provided. In early
years, this task was tended to by existing database man-
agement systems (DBMS) with multimedia extensions.
The basis for representing and modelling multimedia data
is so-called binary large objects, which store images,
video, and audio sequences without any formatting and
analysis done by the system. Often, however, only a
reference to the object is handled within the DBMS. For
the utilisation of the stored multimedia data, user-defined
functions (e.g., content analysis) access the actual data
and integrate their results in the existing database. Hence,
content-based retrieval becomes possible. A survey of
existing retrieval systems was presented, for example, by
Naphade & Huang (2002).
This article provides an overview of the complex
relations and interactions among the different aspects of
a content-based retrieval system, whereby the scope is
purposely limited to images. The main issues of the data
description, similarity expression, and access are ad-
dressed and illustrated for an actual system.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
The concept of content-based retrieval is datacentric per
se; that is, the design of a system has to reflect the
characteristics of the data. Hence, neither an optimal
solution that can span all kinds of multimedia data exists
nor is addressing the variety of data characteristics within
one type even possible. However, there are parallels that
lay the foundation, which then require tailor-made adap-
tation and specialisation. This section provides the gen-
eral groundwork by pointing out the different types of the
so-called metainformation, which describes the raw data:
Technical information refers to the details of the
recording, conversion, and saving process (i.e.,
format and name of the stored media).
Extracted attributes are those that have been de-
duced by analysing the media content. They are
usually called features and emphasise a certain
aspect of the media. Simple features describe, for
instance, statistical values of the contained infor-
mation, while complex features and their weighted
combinations attempt to describe the entire media
content.
Knowledge-based information links the objects,
people, scenarios, and so forth, detected in the
media to entities in the real world.
World-oriented information encompasses informa-
tion on the producer of the media, the date, location,
and so forth. Manually added keywords are espe-
cially in this group, which makes a primitive descrip-
tion and characterisation of the content possible.
As can be seen by this classification, technical and
world-oriented information can be modelled straightfor-
wardly in traditional database structures. Organising and
searching can be done by using existing database func-
tions. The utilisation of the extracted attributes and knowl-
edge-based information is more complex in nature. Al-
though most of the currently avail-able DBMSs can be
extended with multimedia add-ins, in many cases these are
not sufficient, because they cannot describe the stored
data to the required degree of retrieval accuracy. How-
TEAM LinG
Content-Based Image Retrieval
ever, only these two latter types of metainformation lift
the system to an abstract level that allows the full exploi-
tation of the content.
For an in-depth overview of content-based image
retrieval techniques and systems, refer to Deb and Zhang
(2004), Kalipsiz (2000), Smeulders et al. (2000), Vasconcelos
and Kunt (2001), and Xiang and Huang (2000).
MAIN THRUST
The goal of multimedia retrieval is the selection of one or
more images whose metainformation meets certain re-
quirements or is similar to a given sample media instance.
Searching the metainformation is usually based on a full-
text search among the assigned keywords. Furthermore,
content references, such as colour distributions in an
image, or more complex information, such as wavelet
coefficients, can be used. To solve the issue of having the
desired search characteristic in the first place, most sys-
tems prefer to use a query with an example media item. The
systems use this media as a starting point for the search
and processed it in the same manner as the other media
objects, when they were inserted in the database. The
content is then analysed with the selected procedures,
and the media is mapped to a vector consisting of (semi-
) automatically extracted features. Hereafter, the raw data
is only needed for display purposes, and all further
processing focuses on analysing and comparing the
representative vectors. The result of this comparison is a
similarity ranking. The following interfaces can be used to
specify a query in a multimedia database:
Browsing: Beginning with a predefined data set, the
user can navigate in any desired direction by using
a browser until a suitable media sample is found.
This approach is often used when no suitable start-
ing media is available.
Search with keywords: Technical and world-ori-
ented data are represented by alphanumerical fields.
These can be searched for a given keyword. Choos-
ing these keywords is extraordinarily difficult for
abstract structures such as textures, partially due to
the subjectivity of the human perception.
Similarity search: The similarity search is based on
comparing features extracted from the raw data.
Most of these features do not exhibit immediate
references to the image, making them highly ab-
stract for users without special knowledge (Assfalg,
Del Bimbo, & Pala, 2002; Brunelli & Mich, 2000).
Depending on the availability and characteristic of
the query medium, one differentiates between query
by pictorial example, query by painting, selection
from standards, and image montage.
All approaches have their individual advantages as
well as disadvantages, and a suitable selection depends C
on the domain. For example, a fingerprint database is best
realised by using the query-by-pictorial-example tech-
nique, but selection from standards is a suitable candidate
for a comic strip data-base with a limited number of
characters. However, the similarity search, in particular
the query-by-pictorial-example approach, is one of the
most powerful methods because it provides the greatest
degree of flexibility. Thus, it determines the focus hereaf-
ter.
Many different methods for feature extraction were
developed and can be classified by various criteria. Based
on the point in time in which the features are extracted, a-
priori and dynamically extracted features are distinguished.
Although the first group was extracted during insertion of
the corresponding media object in the database, the latter
kind is generated at query time. The advantage of the
dynamic feature extraction is that the user can define
relevant elements in the sample image, and the remaining
parts of the query image do not distract the actual search
objective. Note that both approaches can be combined.
Regardless of the chosen approach, the actual fea-
tures have to be extracted from the considered data.
Examples for this step are histogram-based methods,
calculation of statistical colour information (Mojsilovic,
Hu, & Soljanin, 2002), contour descriptors (Berretti, Del
Bimbo, & Pala, 2000), texture analysis (Gevers, 2002), and
wavelet coefficient selection (Albuz, Kocalar, & Khokhar,
2001). The gained information possibly from different
algorithms is combined in a so-called feature vector
that is, by orders of magnitude, smaller than the raw data.
This reduction in volume enables not only a suitable
handling within the DBMS but also a higher level of
abstraction. Therefore, it can often be utilised directly by
semantic-based approaches (Djeraba, 2003; Fan, Luo, &
Elmagarmid, 2004; Lu, Zhang, Liu, & Hu, 2003) and data-
mining techniques (Datcu, Daschiel, & Pelizzari, 2003; Li
& Narayanan, 2004).
The similarity of two multimedia objects in the con-
tent-based retrieval process is determined by comparing
the representing feature vectors. Over the years, a large
variety of metrics and similarity functions was developed
for this pur-pose, whereby the best-known methods com-
pute a multi-dimensional distance between the vectors:
The smaller the distance, the higher the similarity of the
corresponding media objects. Through the introduction
of weights for the individual positions within the feature
vectors, it is possible to emphasise and/or suppress
desirable and undesirable query characteristics, respec-
tively. In particular, the approach can help to particularise
the query in iterative retrieval systems; that is, if the users
select suitable and unsuitable retrievals, which are used
by the system for adaptation in the next iteration (Jing, Li,
213
TEAM LinG

Zhang, H.-J., & Zhang, B., 2004). However, the application
of distance-based similarity measures is not undisputed,
because the meaning of distance in a multidimensional
vector space spanned by arbitrary features can be ambigu-
ous. Proposed alternatives to the general problem include
angular measures or the incorporation of the data statistic
in the actual distance measure. Nevertheless, the applica-
tion of even the simplest measures can be successful for
all intents and pur-poses if they are tailor-made with
respect to the actual database. Still, if this requirement is
not met, the retrieval can nevertheless be sufficient as long
as a -sufficient number of matches for all the kinds of
possible queries exist.
Hitherto the discussion on the evaluation of similarity
among different feature vectors was limited to fairly simple
structures; that is, the order and length of the vectors were
clearly defined. However, these assumptions are insuffi-
cient when dealing with more complex feature extraction
algorithms. An example is given by algorithms that result
in an arbitrary number of features solely depending on the
content of the multimedia object. Thus, feature vectors of
different lengths have to be compared. To circumvent this
problem, most of the current systems limit the dimension-
ality to a fixed extent either by the choice of extraction
algorithms or through a reducing postprocessing. In-
stances of the latter approach are the principal component
analysis (PCA) and the vector quantisation (VQ). The
retrieval results are acceptable, but the overall system
performance suffers tremendously that is, either by
constantly repeating the PCA or optimising the code-book
for the VQ dynamically in systems with a high insertion
rate, because a permanent adaptation is required. Mean-
while, a variety of alternative measures that can handle
more complex feature vectors were developed, and one of
the most prominent cases is the Earth Movers Distance
(EMD) (Rubner, Tomasi, & Guibas, 2000).
Selected features of an object, file, or other data struc-
tures are stored in indices, a offering accelerated access.
This fact implies that the construction and maintenance
method of an index is of utmost importance for database
efficiency. Data structures employed to support such
queries are called multidimensional index structures. Well-
known examples are the k-d-trees, grid files, R/R*-trees,
SS/SR-trees, VP-trees, and VA files. A general overview of
the context of multimedia retrieval can be found in Lu
(2002).
FUTURE TRENDS
Future trends in image retrieval are manifold, covering
areas such as detailed search for image objects via consid-
eration of semantic information, from identified entities on
the image as well as the extension of the retrieval methods
214
Content-Based Image Retrieval
to special applications such as remote sensing. A sample
application of the latter is given as an example in order to
clarify the necessary adaptations.
The developments and applications of space-borne
remote sensing platforms result in the production of
huge amounts of image data. In particular, the step
towards higher spatial and spectral resolution increases
the obtained amount of data by orders of magnitude. To
enable maintenance as well as retrieval, a large number of
operational DBMS for remotely sensed imagery is avail-
able (Bretschneider, Cavet, & Kao, 2002; Datcu et al.,
2003). However, due to the approach to base the queries
on world-oriented descriptions (e.g., location, scanner,
and date), the systems are not always suitable for users
with little expertise in remote sensing. Furthermore, con-
tent-related inquiries are not feasible in situations like
the following scenario:
A certain region reveals symptoms of saltification and
a corresponding satellite image of the area was
purchased. It is of interest to find other regions which
suffered from the same phenomenon and which were
successfully recovered or preserved, respectively. Thus
the applied strategies in these regions can help to
develop effective counteractions for the specific case
under investigation.
Generic content-based retrieval systems as intro-
duced earlier in this paper are not suitable, because their
extracted features do not consider the special character-
istic of the satellite imagery; that is, for these systems, all
remotely sensed images exhibit a high degree of similar-
ity that prohibits an appropriate differentiation. The Grid
Retrieval System for Remotely Sensed Imagery, G(RS)2I,
is a tailor-made retrieval system that consists of highly
specialised feature extraction modules, corresponding
similarity evaluation, and an adapted indexing technique
under the umbrella of a web-accessible DBMS.
Most of the image content in terms of remote sensing
is contained in the spectral characteristic of an obtained
scene, whereby in contrast to generic images, the number
of colour bands can vary between a few and several
hundred. For the description of such data, a ground
cover classification approach is most suitable, because
it is not only an abstract description of the content but
also is easily linkable with the human understanding of
observed features, for example, water surfaces, forests,
and urban areas. For data that consists of multiple bands,
this approach leads to a highly precise retrieval. Sec-
ondly, the spatial arrangement of the detected regions,
as well as their textural composition, are retrieved as
features. Last but not least, highly specialised extraction
techniques ana-lyse the data for specific features such
as airports, rivers, and road networks. Due to the large
TEAM LinG
Content-Based Image Retrieval
spatial coverage of a satellite scene (covering hundreds
of kilometres and therefore containing highly varying
landscapes), the extraction of several feature vectors at
different positions within a scene is required, because a
global descriptor provides only insufficient accuracy. To
solve this issue, the G(RS)2I uses feature functions that
describe the under-lying data; that is, the multidimen-
sional feature vectors are approximated by a hyper sur-
face, which is modelled by radial basis function
(Bretschneider & Li, 2003). These analytically described
surfaces enable powerful access approaches to the un-
derlying data. Hence, the search for a best match of an
extracted feature vector from a query image becomes an
optimisation problem that is easily solvable, due to the
explicit existence of the first derivative of the feature
function.
For the measurement of similarity, throughout the
entire system the G(RS)2I uses a modified version of the
EMD (Li & Bretschneider, 2003), because the amount of
content in a satellite scene and therefore the length of
the feature vector is generally not predictable. This
concept extends even within the indexing of the data that
is based on a VP-tree and the realisation of the iterative
search engine. With respect to the latter aspect, the
problem in content-based retrieval is that one feature
vector often cannot de-scribe the desired search charac-
teristic precisely enough. Instead of adapting weights
obtained through the users feedback regarding the rel-
evance of the previously retrieved data, the G(RS)2I is not
limited to moving a single query point in the search space.
The approach is to actually fuse the information content
from the positively rated feature vectors by analysing the
corresponding EMD flow matrix (Li & Bretschneider,
2003).
CONCLUSION
Content-based retrieval is beneficial for most types of
data because it resembles the human approach of access-
ing the respective medium. In particular, this idea holds
true for data that mankind can directly process. The actual
conceptual and technical realisation of this natural pro-
cess is a major challenge, because knowledge is fairly
limited with respect to which way this is accomplished.
REFERENCES
Albuz, E., Kocalar, E., & Khokhar, A. A. (2001). Scalable
color image indexing and retrieval using vector wavelets.
IEEE Transactions on Knowledge and Data Engineer-
ing, 13(5), 851-861.
Assfalg, J., Del Bimbo, A., & Pala, P. (2002). Three-
dimensional interfaces for querying by example in con- C
tent-based image retrieval. IEEE Transactions on Visual-
ization and Computer Graphics, 8(4), 305-318.
Berretti, S., Del Bimbo, A., & Pala, P. (2000). Retrieval by
shape similarity with perceptual distance and effective
indexing. IEEE Transactions on Multimedia, 2(4), 225-
239.
Bretschneider, T., Cavet, R., & Kao, O. (2002). Retrieval of
remotely sensed imagery using spectral information con-
tent. Proceedings of the Geoscience and Remote Sensing
Symposium, 4 (pp. 2253-2256).
Bretschneider, T., & Li, Y. (2003). On the problems of
locally defined content vectors in image databases for
large images. Proceedings of the Pacific-Rim Conference
on Multimedia, 3 (pp. 1604-1608).
Brunelli, R., & Mich, O. (2000). Image retrieval by ex-
amples. IEEE Transactions on Multimedia, 2(3), 164-171.
Datcu, M., Daschiel, H., & Pelizzari, A. (2003). Information
mining in remote sensing image archives: System con-
cepts. IEEE Transactions on Geoscience and Remote
Sensing, 41(12), 2923-2936.
Deb, S., & Zhang, Y. (2004). An overview of content-
based image retrieval techniques. Proceedings of the
International Conference on Advanced Information
Networking and Applications, 1 (pp. 59-64).
Djeraba, C. (2003). Association and content-based re-
trieval. IEEE Transactions on Knowledge and Data En-
gineering, 15(1), 118-135.
Fan, J., Luo, H., & Elmagarmid, A. K. (2004). Concept-
oriented indexing of video databases: Toward semantic
sensitive retrieval and browsing. IEEE Transactions on
Image Processing, 13(7), 974-992.
Gevers, T. (2002). Image segmentation and similarity of
color-texture objects. IEEE Transactions on Multimedia,
4(4), 509-516.
Jing, F., Li, M., Zhang, H.-J., & Zhang, B. (2004). Rel-
evance feedback in region-based image retrieval. IEEE
Transactions on Circuits and Systems for Video Technol-
ogy, 14(5), 672-681.
Kalipsiz, O. (2000). Multimedia databases. Proceedings of
the IEEE International Conference on Information Visu-
alization (pp. 111-115).
Li, J., & Narayanan, R. M. (2004). Integrated spectral and
spatial information mining in remote sensing imagery.
IEEE Transactions on Geoscience and Remote Sensing,
42(3), 673-685.
215
TEAM LinG

Li, Y., & Bretschneider, T. (2003). Supervised content-
based satellite image retrieval using piecewise defined
signature similarities. Proceedings of the Geoscience and
Remote Sensing Sympo-sium, 2 (pp. 734-736).
Lu, G.-J. (2002). Techniques and data structures for effi-
cient multimedia retrieval based on similarity. IEEE Trans-
actions on Multimedia, 4(3), 372-384.
Lu, Y., Zhang, H., Liu, W., & Hu, C. (2003). Joint semantics
and feature based image retrieval using relevance feed-
back. IEEE Transactions on Multimedia, 5(3), 339-347.
Mojsilovic, A., Hu, H., & Soljanin, E. (2002). Extraction of
perceptually important colors and similarity measurement
for image matching, retrieval and analysis. IEEE Transac-
tions on Image Processing, 11(11), 1238-1248.
Mojsilovic, A., Kovacevic, J., Hu, J.-Y., Safranek, R. J., &
Ganapathy, S. K. (2000). Matching and retrieval based on
the vocabulary and grammar of color patterns. IEEE
Transactions on Image Processing, 9(1), 38-54.
Naphade, M. R., & Huang, T. S. (2002). Extracting seman-
tics from audio-visual content: The final frontier in multi-
media retrieval. IEEE Transactions on Neural Networks,
13(4), 793-810.
Rubner, Y., Tomasi, C., & Guibas, I. J. (2000). The earth
movers distance as a metric for image retrieval. Journal
of Computer Vision, 40(2), 99-121.
Smeulders, A., Worring, M., Santini, S., Gupta, A., & Jain,
R. (2000). Content-based image retrieval at the end of the
early years. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 22(12), 1349-1380.
Vasconcelos, N., & Kunt, M. (2001). Content-based re-
trieval from image databases: Current solutions and fu-
ture directions. Proceedings of the IEEE International
Conference on Image Processing, 3 (pp. 6-9).
216
Content-Based Image Retrieval
Xiang, S. Z., & Huang, T. S. (2000). Image retrieval: Feature
primitives, feature representation, and relevance feed-
back. Proceedings of the IEEE Workshop on Content-
based Access of Image and Video Libraries (pp. 10-14).
KEY TERMS
Content-Based Retrieval: The search for suitable
objects in a database based on the content; often used to
retrieve multimedia data.
Dynamic Feature Extraction: Analysis and descrip-
tion of the media content at the time of querying the
database. The information is computed on demand and
discarded after the query has been processed.
Feature Vector: Data that describes the content of the
corresponding multimedia object. The elements of the
feature vector represent the extracted descriptive infor-
mation with respect to the utilised analysis.
Index Structures: Adapted data structures to accel-
erate the retrieval. The a-priori extracted features are
organised in such a way that the comparisons can be
focused to a certain area around the query.
Multimedia Database: A multimedia database system
consists of a high-performance database management
system and a database with a large storage capacity and
supports and manages, in addition to alphanumerical data
types, multimedia objects regarding storage, querying,
and searching.
Query by Pictorial Example: The query is formulated
by using a user-provided example for the desired retrieval.
Both the query and stored media objects are analysed in
the same way.
Similarity: Correspondence of two data objects of
the same medium. The similarity is determined by compar-
ing the corresponding feature vectors, for example, by a
metric or distance function.
TEAM LinG

Continuous Auditing and Data Mining
Edward J. Garrity
Canisius College, USA
Joseph B. ODonnell
Canisius College, USA
G. Lawrence Sanders
State University of New York at Buffalo, USA
INTRODUCTION
Investor confidence in the financial markets has been
rocked by recent corporate frauds and many in the invest-
ment community are searching for solutions. Meanwhile,
due to changes in technology, organizations are increas-
ingly able to produce financial reports on a real-time basis.
Access to this timely information can help investors,
shareholders, and other third parties, but only if this
information is accurate and verifiable. Real-time financial
reporting requires real-time or continuous auditing (CA)
to ensure integrity of the reported information. Continu-
ous auditing is a type of auditing which produces audit
results simultaneously, or a short period of time after, the
occurrence of relevant events (Kogan, Sudit, &
Vasarhelyi, 2003, p. 1). CA is facilitated by eXtensible
Business Reporting Language (XBRL), which enables
seamless transmission of company financial information
to auditor data warehouses. Data mining of these ware-
houses provides opportunities for the auditor to deter-
mine financial trends and identify erroneous transac-
tions.
BACKGROUND
Auditing is a systematic process of objectively obtain-
ing and evaluating evidence of assertions about eco-
nomic actions and events to ascertain the correspon-
dence between those assertions and established criteria
and communicating the results to interested parties
(Konrath, 2002, p. 5). In CA, the collection of evidence is
constant, and evaluation of the evidence occurs promptly
after collection (Kogan et al., 2003).
Computerized Assisted Auditing Techniques
(CAATs) are computer programs or software applications
that are used to improve audit efficiency. CAATs offer
great promise to improve audits but have not met expec-
tations due to a lack of a common interface with IT
systems (Liang et al., 2001, p. 131). Also, concurrent
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
217
C
CAATS. often require that special audit software mod-
ules be embedded at the EDP system design stage (Liang
et al., 2001, p. 131). Many entities are reluctant to allow the
implementation of embedded audit modules, which per-
form CA, due to concerns these CAATs could adversely
affect systems processing in areas such as reducing
response times. Considering these difficulties, it is not
surprising that continuous transaction monitoring tools
are the second least used software by auditors (Daigle &
Lampe, 2003).
These hurdles to CAAT usage are being minimized by
the emergence of eXtensible Markup Language (XML)
and eXtensible Business Reporting Language that mini-
mize system interface issues. XML is a mark-up language
that allows tagging of data to give the data meaning.
XBRL is a variant of XML that is designed specifically for
financial reporting and provides the capability of real-time
online performance reporting. Both XML and XBRL en-
able the receiver of the data to seamlessly download
information to the receivers data warehouse
According to David & Steinbart (2000), data ware-
houses improve audit quality and efficiency by reducing
the time needed to access data and perform data analysis.
Improved audit quality should lead to early detection, and
possible prevention, of fraudulent financial reporting.
Auditor data warehouses may also be used in financial
fraud litigations in providing evidence to evaluate the
legitimacy of transactions and appropriateness of auditor
actions in assessing transactions.
Data mining techniques are well suited to evaluate CA
generated warehouse data but advances in audit tools are
needed. Data mining and analysis software is the most
commonly used audit software (Bierstaker, Burnaby, &
Hass, 2003). Auditor data mining and analysis software
typically includes low level statistical tools and auditor
specific models like Benfords Law. Benfords Law holds
that there is a naturally occurring pattern of values in the
digits of a number (Nigrini, 2002). Significant variation
from the expected number pattern may be due to errone-
ous or fraudulent transactions.
TEAM LinG

More robust auditing tools, using more sophisticated
data mining methods, are needed for mining large data-
bases and to help auditors meet auditing requirements.
According to auditing standards (Statement on Audit
Standards 99), auditors should incorporate
unpredictability in procedures performed (Ramos, 2003).
Otherwise, perpetrators of frauds may become familiar
with common audit procedures and conceal fraud by
placing it in areas that auditors are least likely to look.
MAIN THRUST
It is critical for the modern auditor to understand the
nature of CA, and the capabilities of different data mining
methods in designing an effective audit approach. To-
ward this end, this paper addresses these issues through
discussion of CA, comparison of data mining methods,
and we also provide a potential CA and data mining
architecture. Although it is beyond the scope of this
paper to provide an in-depth technical discussion of the
details of the proposed architecture, we hope this stimu-
lates technical research in this area and provides a start-
ing point for CA system designers.
Continuous Auditing (CA)
Audits involve three major components: audit planning,
conducting the audit, and reporting on audit findings
(Konrath, 2002). The CA approach can be used for the
audit planning and conducting the audit phases. Accord-
ing to Pushkin (2003), CA is useful for the strategic audit
planning component that addresses the strategic risk of
reaching an inappropriate conclusion by not integrating
essential activities into the audit plan (p. 27). Strategic
information may be captured from the entitys Intranets
and from the global internet using intelligent agents
(Pushkin, 2003, p. 28).
CA is also useful for performing the audit or what
Pushkin (2003) refers to as the tactical component of the
audit. Tactical activities are most often directed at ob-
taining transactional evidence as a basis on which to
assess the validity of assertions embodied in account
balances (p.27). For example, CA is useful in testing that
entities comply with financial performance measures of
debt covenants in loan agreements (Woodroof & Searcy,
2001).
CA requires prompt responses to high-risk transac-
tions and the ability to identify financial trends from large
volumes of data. Intelligent agents can be used to promptly
identify and respond to erroneous transactions. Under-
standing the capabilities of data mining methods in iden-
tifying financial trends is useful in selecting an appropri-
ate data mining approach for CA.
218
Continuous Auditing and Data Mining
Comparing Methods of Data Mining
Data mining is a process by which one discovers previ-
ously unknown information from large sets of data. Data
mining algorithms can be divided into three major groups:
(1) mathematical-based methods, (2) logic-based meth-
ods, and (3) distance-based methods (Weiss & Indurkhya,
1998). Common examples from each major category are
described below.
Mathematical-Based Methods
Neural Networks
An Artificial Neural Network (ANN) is a network of nodes
modeled after a neuron or neural circuit. The neural
network mimics the processing of the human brain. In a
neural network, neurons are grouped into layers or slabs
(Lam, 2004). An input layer consists of neurons that
receive input from the external environment. The output
layer communicates results of the ANN to the user or
external environment. The ANN may also consist of a
number of intermediate (hidden) layers. The processing of
an ANN starts with inputs being received by the input
layer and, upon being excited; the neurons are fired and
produce outputs to the other layers of the system. The
nodes or neurons are interconnected and they will send
signals or fire only if the signals it receives exceed a
certain threshold value. The value of a node is a non-
linear, (usually logistic), function of the weighted sum of
the values sent to it by nodes that are connected to it
(Spangler, May, & Vargas, 1999).
Programming a neural network to process a set of
inputs and produce the desired output is a matter of
designing the interactions among the neurons. This pro-
cess consists of the following: (1) arranging neurons in
various layers, (2) deciding both the connections among
neurons of different layers, as well as the neurons within
a layer, (3) determining the way a neuron receives input
and produces output (e.g., the type of function used), and
(4) determining the strength of connections within the
network by selecting and using a training data set so that
the ANN can determine the appropriate values of connec-
tion weights (Lam, 2004).
Prior neural network research has addressed audit
areas of risk assessment, errors and fraud, going concern
audit opinion, financial distress, and bankruptcy predic-
tion (Lin, Hwang, & Becker, 2003). Research has identified
successful uses of ANN, however, there are still many
other issues to address. For instance, ANN is effective for
analytical review procedures although there is no clear
guideline for the performance measures to use for this
analysis (Koskivaara, 2004). Neural network research
found differences between the patterns of quantitative
TEAM LinG
Continuous Auditing and Data Mining
and qualitative (textual) information from financial state-
ments (Back, Toivenen, Vanharanta, & Visa, 2001). Audi-
tors would benefit from advancement in identifying appro-
priate financial inputs for ANN and developing models
that integrate quantitative and textual information.
Discriminant Analysis
Linear discriminant analysis is similar to multiple regres-
sion analysis. The major difference is that discriminant
analysis uses a nominal or ordinal dependent variable
whereas regression uses a continuous dependent vari-
able. Discriminant analysis constructs classification func-
tions of the form,
Y = c 1 a 1 + c 2a 2 + c m a m + c 0 view. A big advantage of Tree and Rule Induction is the
Where ci is the coefficient for the case attribute ai and
c0 is a constant, for each of the n categories (Spangler et
al., 1999). One field in the set of data is assigned as the
target (response or class, Y) variable, and the algorithm
produces models for target variables as a function of the
other fields in the data set, that are identified as explana-
tory variables (features, cases) (Apte et al., 2003).
Distance-Based Methods
Clustering
Clustering is a data mining approach that partitions large
sets of data objects into homogeneous groups. Clustering
uses unsupervised classification where little manual pre-
screening of data is necessary thus it is useful in situa-
tions where no predefined knowledge of categories is
available (Maltseva et al., 2000). Generally, an object is
described by a set of m attributes (features), and can be
represented by an m-dimensional vector, called a pattern.
Therefore, if a dataset contains n objects, it can be viewed
as an n by m pattern matrix. Rows of the matrix correspond
to patterns and columns correspond to features. Math-
ematical techniques are then employed to identify clus-
Table 1. Data mining approaches compared1
Decision Tree Neural
Rule Induction Networks
Method Type Logic-based Math-based
Learning Supervised Supervised
approach
Representational Decision nodes, Functional
Scheme rules relationship
between
attributes and
classes
ters, which can be described as regions of this space
containing points that are close to each other (Maltseva C
et al., 2000).
Logic-Based Methods
Tree and Rule Induction
The Tree and Rule Induction approach to data mining
uses an algorithm (e.g., the ID3 algorithm) to induce a
decision tree from a file of individual cases, where the
case is described by a set of attributes and the class to
which the case belongs (Spangler et al., 1999). Once the
decision tree is generated, it is a simple matter to convert
from the decision tree format to an equivalent rule-based
ability to communicate and understand the information
derived from the data mining. Tree and Rule induction
research has addressed audit areas of bankruptcy, bank
failure, and credit risk (Spangler et al., 1999).
A comparison of data mining approaches based on
representational scheme and learning approach is pre-
sented in Table 1. Learning approaches are classified as
either supervised, where induction rules are used in
assigning observations to predefined classifications, or
unsupervised, where both categories and classification
rules are determined by the data mining method (Spangler
et al., 1999). Representational scheme relates to how the
data mining methods model relationships.
Criteria and Issues in Selecting a Data
Mining Method
Discriminant analysis and statistical techniques have
enjoyed a long run of popularity in data mining. However,
because of increasing availability of large volumes of
data, a trend toward increased use of search-based, non-
parametric modeling techniques such as neural networks
and rule-based or decision trees has been taking place
(Apte et al., 2003). Ultimately however, when evaluating
and deciding on a data mining method, users must weigh
Discriminant Clustering
Analysis
Math-based Distance-based
Supervised Unsupervised
Functional Functional
relationship relationship
between between
attributes and attributes and
classes classes
219
TEAM LinG

and often compromise between predictive accuracy, level
of understandability, and computational demand (Apte et
al., 2003).
Indeed, when one examines issues such as scalability
(i.e., how well the data mining method works regardless of
the size of the data set), accuracy (i.e., how well the
information extracted remains stable and constant be-
yond the boundaries of the data from which it was ex-
tracted, or trained), robustness (i.e., how well the data
mining method works in a wide variety of domains), and
interpretability (i.e., how well the data mining method
provides understandable information and insight of value
to the end user), it becomes clear that no data mining
methods currently excel in all areas (Apte et al., 2003).
The specific demands of CA raise a number of issues
that relate to the selection of data mining methods and
their appropriate application in this domain. The next
section explores these issues in greater detail.
The CA Challenge: Difficulties and
Issues
The demand for more timely communication of auditing
information to business stakeholders requires auditors to
find new ways to monitor, assemble and analyze audit
information. A number of continuous audit tools and
techniques will need to be developed to enable auditors
to assess risk, evaluate internal controls, extract data,
download data for analytical review, and identify excep-
tions and unusual transactions.
One of the challenges in developing a CA capability
is developing a technology infrastructure for extracting
Figure 1. Continuous auditing and data mining architecture2
Entity Systems
Databases
Transactions
Capture
Transactions
and
Transmit
Transactions
______________
220
Continuous Auditing and Data Mining
data with differing file formats and record structures from
heterogeneous platforms (Rezaee et al., 2002). Another
consideration in developing a technology infrastructure
to gather transactions and other activity for auditing is
the degree of automation employed. The degree of auto-
mation can vary depending on the audit system design
but at least three possibilities are:
1. Embedded audit modules where audit programs are
tightly integrated with application source code to
constantly monitor and report on exceptional con-
ditions (limited use of this approach due to potential
adverse effects on entity processing, see Back-
ground section for further discussion);
2. The automatic capture and transformation of data
and storage in data warehouses but still requiring
auditor involvement in running queries to isolate
exceptions and detect unusual patterns (automatic
capture but with auditor intervention);
3. The automatic capture and transformation of data
and storage in data warehouses and the integration
of intelligent agents to modify data capture routines
and exception reporting and trend spotting via
multiple data mining methods (automatic, modified
capture and modified data mining for trend analy-
sis).
Finally, the appropriate selection of a data mining
method is critical for determining unusual trends and
spotting fraudulent behavior. Important considerations
include: (1) the size of the data set, (2) the accuracy of the
given data mining method in the particular domain, and (3)
Auditor Systems
Environmental
Monitoring
XBRL Intelligent-
Audit Data
XML Audit Warehouse
Evaluator
Data
Transformation
Data Mart
Data Mart
Intelligent Agent: Data Mining
Applications Manager
TEAM LinG
Continuous Auditing and Data Mining
the interpretability of the data mining output information.
Regardless of the particular context, in the area of CA a
prime consideration is the frequent and (automatic) alter-
ation or change of the data mining technique and the
appropriate selection and changing of the selected audit-
ing information to be reviewed. These are important con-
siderations because the audited parties should not be
able to anticipate the audit tests and thus create errone-
ous transactions that might go undetected (Ramos, 2003).
A Potential Architecture
In order to handle the issues identified above, a proposed
architecture for the CA process is presented in Figure 1.
Two important features of the architecture are: (1) the use
of data warehousing and data marts, and (2) the use of
intelligent agents to periodically modify the data extrac-
tion and data mining methods. The architecture involves
the transfer of data from the entitys systems to the
auditors systems through the use of XBRL and XML. The
auditors systems include environmental monitoring of
political, economic, and technological factors to address
strategic audit issues and data mining of transactions to
address tactical audit issues.
FUTURE TRENDS
In the future, auditing and information systems research-
ers will need to identify additional tools and data mining
methods that are most appropriate for this new applica-
tion area. Emerging innovations in Web technology and
the general publics demand for reliable performance
reporting are expected to spur demand for the use of data
mining for CA.
Expected growth in data mining for CA will provide
opportunities and issues for auditors and researchers in
several areas. Advancement in the areas of data mining of
textual information will be useful to CA. Recognizing
patterns in qualitative information provides a broader
perspective and richer understanding of the entitys
internal and external situation (Back et al., 2001). Also, for
risk based auditing ANN model builders will need quali-
tative and quantitative factors that capture political, eco-
nomic, and technological factors, as well as balanced
scorecard metrics that signal the extent to which an entity
is achieving its strategic objectives (Lin et al., 2003, p. 230).
CONCLUSION
The use of data mining for continuous auditing is poised
to improve the effectiveness of audits, and ultimately the
reliability of performance reporting. Auditors need to
understand the capabilities of different data mining ap- C
proaches to ensure effective continuous auditing. Look-
ing ahead, researchers will need to further develop data
mining approaches and tools for the expanding needs of
the audit profession.
REFERENCES
Apte, C.V., Hong, S.J., Natarajan, R., Pednault, E.P.D.,
Tipu, F.A., & Weiss, S.M. (2003). Data-intensive analytics
for predictive modeling. IBM Journal of Research and
Development, 47(1), 17-23.
Back, B., Toivenen, J., Vanharanta, H., & Visa, A. (2001).
Comparing numerical data and text information from an-
nual reports using self-organizing maps. International
Journal of Accounting Information Systems, 2(2001),
249-269.
Bierstaker, J.L., Burnaby, P., & Hass, S. (2003). Recent
changes in internal auditors use of technology. Internal
Auditing, 18(4), 39-45.
David, J.S., & Steinbart, P.J. (2000). Data warehousing
and data mining: Opportunities for internal auditors.
Altamonte Springs, FL: The Institute of Internal Auditors
Research Foundation.
Kogan, A., Sudit, E.F., & Vasarhelyi, M.A. (2003). Con-
tinuous online auditing: An evolution (pp. 1-25). Unpub-
lished Workpaper.
Konrath, L.F. (2002). Auditing: A risk analysis approach.
Cincinnati, OH: South-Western.
Koskivaara, E. (2004). Artificial neural networks in ana-
lytical procedures. Managerial Auditing Journal,
19(2), 191-223.
Lam, M. (2004). Neural network techniques for finan-
cial performance prediction: Integrating fundamental
and technical analysis. Decision Support Systems, 37(4),
567-581.
Liang, D., Fengyi, L., & Wu, S. (2001). Electronically
auditing EDP systems with the support of emerging infor-
mation technologies. International Journal of Account-
ing Information Systems, 2, 130-147.
Lin, J.W., Hwang, M.I., & Becker, J.D. (2003). A fuzzy
neural network for assessing the risk of fraudulent finan-
cial reporting. Managerial Auditing Journal, 18(8),
657-665.
221
TEAM LinG

Maltseva, E., Pizzuti, C., & Talia, D. (2000). Indirect knowl-
edge discovery by using singular value decomposition.
In Data Mining II. Southhampton, UK: WIT Press.
Nigrini, M.J. (2002). Analysis of digits and number pat-
terns. In J.C. Robertson (Ed.), Fraud examination for
managers and auditors (pp. 495-518). Austin, TX: Atex
Austin, Inc.
Pushkin, A.B. (2003). Comprehensive continuous audit-
ing: The strategic component. Internal Auditing, 18(1),
26-33.
Ramos, M. (2003). Auditors responsibility for fraud de-
tection. Journal of Accountancy, 195(1), 28-36.
Rezaee, Z., Sharbatoghlie, A., Elam, R., & McMickle, P.L.
(2002). Continuous auditing: Building automated audit-
ing capability. Auditing: A Journal of Practice & Theory,
21(1), 147-163.
Spangler, W.E., May, J.H., & Vargas, L.G. (1999). Choos-
ing data mining methods for multiple classification: Rep-
resentational and performance measurement implications
for decision support. Journal of Management Informa-
tion Systems, 16(1), 37-62.
Weiss, S.M., & Indurkhya, N. (1998). Predictive data
mining. San Francisco, CA: Morgan Kaufmann.
Woodroof, J., & Searcy, D. (2001). Continuous audit
model development and implementation within a debt
covenant compliance domain. International Journal of
Accounting Information Systems, 2, 169-191.
KEY TERMS
Artificial Neural Network (ANN): A network of nodes
modeled after a neuron or neural circuit. The neural
network mimics the processing of the human brain.
222
Continuous Auditing and Data Mining
Auditing: Systematic process of objectively obtain-
ing and evaluating evidence of assertions about eco-
nomic actions and events to ascertain the correspon-
dence between those assertions and established criteria
and communicating the results to interested parties.
Clustering: Data mining approach that partitions
large sets of data objects into homogeneous groups.
Computerized Assisted Auditing Techniques
(CAATs): Software applications that are used to improve
the efficiency of an audit.
Continuous Auditing: Type of auditing, which pro-
duces audit results simultaneously, or a short period of
time after, the occurrence of relevant events.
Discriminant Analysis: Statistical methodology used
for classification that is based on the general regression
model, and uses a nominal or ordinal dependent variable.
eXtensible Business Reporting Language (XBRL):
Mark up language that allows for the tagging of data and
is designed for performance reporting. It is a variant of
XML.
eXtensible Markup Language (XML): Mark-up lan-
guage that allows for the tagging of data in order to add
meaning to the data.
Tree and Rule Induction: Data mining approach that
uses an algorithm (e.g., the ID3 algorithm) to induce a
decision tree from a file of individual cases, where the case
is described by a set of attributes and the class to which
the case belongs.
ENDNOTES
1
Adapted from Spangler et al. (1999).
2
Adapted from Rezaee et al. (2002).
TEAM LinG

Data Driven vs. Metric Driven Data
Warehouse Design
John M. Artz
The George Washington University, USA
INTRODUCTION
Although data warehousing theory and technology have
been around for well over a decade, they may well be the
next hot technologies. How can it be that a technology
sleeps for so long and then begins to move rapidly to the
foreground? This question can have several answers.
Perhaps the technology had not yet caught up to the
theory or that computer technology 10 years ago did not
have the capacity to delivery what the theory promised.
Perhaps the ideas and the products were just ahead of
their time. All these answers are true to some extent.
But the real answer, I believe, is that data warehousing is
in the process of undergoing a radical theoretical and
paradigmatic shift, and that shift will reposition data
warehousing to meet future demands.
BACKGROUND
Just recently I started teaching a new course in data
warehousing. I have only taught it a few times so far, but
I have already noticed that there are two distinct and
largely incompatible views of the nature of a data ware-
house. A prospective student, who had several years of
industry experience in data warehousing but little theo-
retical insight, came by my office one day to find out
more about the course. Are you an Inmonite or a
Kimballite? she inquired, reducing the possibilities to
the core issues. Well, I suppose if you put it that way,
I replied, I would have to classify myself as a Kimballite.
William Inmon (2000, 2002) and Ralph Kimball (1996,
1998, 2000) are the two most widely recognized au-
thors in data warehousing and represent two competing
positions on the nature of a data warehouse.
The issue that this student was trying to get at was
whether or not I viewed the dimensional data model as
the core concept in data warehousing. I do, of course,
but there is, I believe, a lot more to the emerging
competition between these alternative views of data
warehouse design. One of these views, which I call the
data-driven view of data warehouse design, begins with
existing organizational data. These data have more than
likely been produced by existing transaction processing
systems. They are cleansed and summarized and are
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
223
,
used to gain greater insight into the functioning of the
organization. The analysis that can be done is a function
of the data that were collected in the transaction pro-
cessing systems. This was, perhaps, the original view of
data warehousing and, as will be shown, much of the
current research in data warehousing assumes this view.
The competing view, which I call the metric-driven
view of data warehouse design, begins by identifying key
business processes that need to be measured and tracked
over time in order for the organization to function more
efficiently. A dimensional model is designed to facili-
tate that measurement over time, and data are collected
to populate that dimensional model. If existing organi-
zational data can be used to populate that dimensional
model, so much the better. But if not, the data need to be
acquired somehow. The metric-driven view of data ware-
house design, as will be shown, is superior both theo-
retically and philosophically. In addition, it dramati-
cally changes the research program in data warehousing.
The metric-driven and data-driven approaches to data
warehouse design have also been referred to, respec-
tively, as metric pull versus data push (Artz, 2003).
MAIN THRUST
Data-Driven Design
The classic view of data warehousing sees the data
warehouse as an extension of decision support systems.
Again, in a classic view, decision support systems sit
atop management information systems and use data
extracted from management information and transac-
tion processing systems to support decisions within the
organization. This view can be thought of as a data-
driven view of data warehousing, because the exploita-
tions that can be done in the data warehouse are driven by
the data made available in the underlying operational
information systems.
This data-driven model has several advantages. First,
it is much more concrete. The data in the data warehouse
are defined as an extension of existing data. Second, it is
evolutionary. The data warehouse can be populated and
exploited as new uses are found for existing data. Fi-
nally, there is no question that summary data can be
TEAM LinG

derived, because the summaries are based upon existing
data. However, it is not without flaws. First, the integra-
tion of multiple data sources may be difficult. These
operational data sources may have been developed inde-
pendently, and the semantics may not agree. It is diffi-
cult to resolve these conflicting semantics without a
known end state to aim for. But the more damaging
problem is epistemological. The summary data derived
from the operational systems represent something, but
the exact nature of that something may not be clear.
Consequently, the meaning of the information that de-
scribes that something may also be unclear. This is
related to the semantic disintegrity problem in rela-
tional databases. A user asks a question of the database
and gets an answer, but it is not the answer to the
question that the user asked. When the somethings that
are represented in the database are not fully understood,
then answers derived from the data warehouse are likely
to be applied incorrectly to known somethings. Unfor-
tunately, this also undermines data mining. Data mining
helps people find hidden relationships in the data. But if
the data do not represent something of interest in the
world, then those relationships do not represent any-
thing interesting, either.
Research problems in data warehousing currently
reflect this data-driven view. Current research in data
warehousing focuses on a) data extraction and integra-
tion, b) data aggregation and production of summary
sets, c) query optimization, and d) update propagation
(Jarke, Lenzerini, Vassiliou, & Vassiliadis, 2000). All these
issues address the production of summary data based on
operational data stores.
A Poverty of Epistemology
The primary flaw in data-driven data warehouse design is
that it is based on an impoverish epistemology. Episte-
mology is that branch of philosophy concerned with
theories of knowledge and the criteria for valid knowl-
edge (Fetzer & Almeder, 1993; Palmer, 2001). That is
to say, when you derive information from a data ware-
house based on the data-driven approach, what does that
information mean? How does it relate to the work of the
organization? To see this issue, consider the following
example. If I asked each student in a class of 30 for their
ages, then summed those ages and divided by 30, I should
have the average age of the class, assuming that every-
one reported their age accurately. If I were to generate
a list of 30 random numbers between 20 and 40 and took
the average, that average would be the average of the
numbers in that data set and would have nothing to do
with the average age of the class. In between those two
extremes are any number of options. I could guess the
ages of students based on their looks. I could ask mem-
224
Data Driven vs. Metric Driven Data Warehouse Design
bers of the class to guess the ages of other members. I
could rank the students by age and then use the ranking
number instead of age. The point is that each of these
attempts is somewhere between the two extremes, and
the validity of my data improves as I move closer to the
first extreme. That is, I have measurements of a specific
phenomenon, and those measurements are likely to
represent that phenomenon faithfully. The epistemo-
logical problem in data-driven data warehouse design is
that data is collected for one purpose and then used for
another purpose. The strongest validity claim that can be
made is that any information derived from this data is
true about the data set, but its connection to the organi-
zation is tenuous. This not only creates problems with
the data warehouse, but all subsequent data-mining dis-
coveries are suspect also.
METRIC-DRIVEN DESIGN
The metric-driven approach to data warehouse design
begins by defining key business processes that need to
be measured and tracked in order to maintain or improve
the efficiency and productivity of the organization.
After these key business processes are defined, they are
modeled in a dimensional data model and then further
analysis is done to determine how the dimensional
model will be populated. Hopefully, much of the data
can be derived from operational data stores, but the
metrics are the driver, not the availability of data from
operational data stores.
A relational database models the entities or objects
of interest to an organization (Teorey, 1999). These
objects of interest may include customers, products,
employees, and the like. The entity model represents
these things and the relationships between them. As
occurrences of these entities enter or leave the organi-
zation, that addition or deletion is reflected in the
database. As these entities change in state, somehow,
those state changes are also reflected in the database.
So, theoretically, at any point in time, the database
faithfully represents the state of the organization. Que-
ries can be submitted to the database, and the answers to
those queries should, indeed, be the answers to those
questions if they were asked and answered with respect
to the organization.
A data warehouse, on the other hand, models the
business processes in an organization to measure and
track those processes over time. Processes may include
sales, productivity, the effectiveness of promotions,
and the like. The dimensional model contains facts that
represent measurements over time of a key business
process. It also contains dimensions that are attributes
of these facts. The fact table can be thought of as the
TEAM LinG
Data Driven vs. Metric Driven Data Warehouse Design
dependent variable in a statistical model, and the dimen-
sions can be thought of as the independent variables. So
the data warehouse becomes a longitudinal dataset track-
ing key business processes.
A Parallel with Pre-Relational Days
You can see certain parallels between the state of data
warehousing and the state of database prior to the rela-
tional model. The relational model was introduced in
1970 by Codd but was not realized in a commercial
product until the early 1980s (Date, 2004). At that time,
a large number of nonrelational database management
systems existed. All these products handled data in dif-
ferent ways, because they were software products devel-
oped to handle the problem of storing and retrieving data.
They were not developed as implementations of a theo-
retical model of data. When the first relational product
came out, the world of databases changed almost over-
night. Every nonrelational product attempted, unsuc-
cessfully, to claim that it was really a relational product
(Codd, 1985). But no one believed the claims, and the
nonrelational products lost their market share almost
immediately.
Similarly, a wide variety of data warehousing prod-
ucts are on the market today. Some are based on the
dimensional model, and some are not. The dimensional
model provides a basis for an underlying theory of data
that tracks processes over time rather than the current
state of entities. Admittedly, this model of data needs
quite a bit of work, but the relational model did not come
into dominance until it was coupled with entity theory, so
the parallel still holds. We may never have an announce-
ment in data warehousing as dramatic as Codds paper in
relational theory. It is more likely that a theory of tem-
poral dimensional data will accumulate over time. How-
ever, in order for data warehousing to become a major
force in the world of databases, an underlying theory of
data is needed and will eventually be developed.
The Implications for Research
The implications for research in data warehousing are
rather profound. Current research focuses on issues
such as data extraction and integration, data aggregation
and summary sets, and query optimization and update
propagation. All these problems are applied problems in
software development and do not advance our under-
standing of the theory of data.
But a metric-driven approach to data warehouse de-
sign introduces some problems whose resolution can
make a lasting contribution to data theory. Research
problems in a metric-driven data warehouse include a)
How do we identify key business processes? b) How do
we construct appropriate measures for these processes?
c) How do we know those measures are valid? d) How do ,
we know that a dimensional model has accurately cap-
tured the independent variables? e) Can we develop an
abstract theory of aggregation so that the data aggrega-
tion problem can be understood and advanced theoreti-
cally? and, finally, f) can we develop an abstract data
language so that aggregations can be expressed math-
ematically by the user and realized by the machine?
Initially, both data-driven and metric-driven de-
signs appear to be legitimate competing paradigms for
data warehousing. The epistemological flaw in the data-
driven approach is a little difficult to grasp, and the
distinction that information derived from a data-
driven model is information about the data set, but
information derived from a metric-driven model is
information about the organization may also be a bit
elusive. However, the implications are enormous. The
data-driven model has little future in that it is founded
on a model of data exploitation rather than a model of
data. The metric-driven model, on the other hand, is
likely to have some major impacts and implications.
FUTURE TRENDS
The Impact on White-Collar Work
The data-driven view of data warehousing limits the
future of data warehousing to the possibilities inherent
in summarizing large collections of old data without a
specific purpose in mind. The metric-driven view of
data warehousing opens up vast new possibilities for
improving the efficiency and productivity of an organi-
zation by tracking the performance of key business
processes. The introduction of quality management
procedures in manufacturing a few decades ago dra-
matically improved the efficiency and productivity of
manufacturing processes, but such improvements have
not occurred in white-collar work.
The reason that we have not seen such an improve-
ment in white-collar work is that we have not had
metrics to track the productivity of white-collar work-
ers. And even if we did have the metrics, we did not have
a reasonable way to collect them and track them over
time. The identification of measurable key business
processes and the modeling of those processes in a data
warehouse provides the opportunity to perform quality
management and process improvement on white-collar
work.
Subjecting white-collar work to the same rigorous
definition as blue-collar work may seem daunting, and
indeed that level of definition and specification will
not come easily. So what would motivate a business to
225
TEAM LinG

do this? The answer is simple: Businesses will have to do
this when the competitors in their industry do it. Whoever
does this first will achieve such productivity gains that
competitors will have to follow suit in order to compete.
In the early 1970s, corporations were not revamping their
internal procedures because computerized accounting
systems were fun. They were revamping their internal
procedures because they could not protect themselves
from their competitors without the information for deci-
sion making and organizational control provided by their
accounting information systems. A similar phenomenon
is likely to drive data warehousing.
Dimensional Algebras
The relational model introduced Structured Query Lan-
guage (SQL), an entirely new data language that allowed
nontechnical people to access data in a database. SQL
also provided a means of thinking about record selec-
tion and limited aggregation. Dimensional models can
be exploited by a dimensional query language such as
MDX (Spofford, 2001), but much greater advances are
possible.
Research in data warehousing will likely yield some
sort of a dimensional algebra that will provide, at the
same time, a mathematical means of describing data
aggregation and correlation and a set of concepts for
thinking about aggregation and correlation. To see how
this could happen, think about how the relational model
led us to think about the organization as a collection of
entity types or how statistical software made the con-
cepts of correlation and regression much more con-
crete.
A Unified Theory Of Data
In the organization today, the database administrator and
the statistician seem worlds apart. Of course, the statis-
tician may have to extract some data from a relational
database in order to do his or her analysis. And the
statistician may engage in limited data modeling in
designing a data set for analysis by using a statistical
tool. The database administrator, on the other hand, will
spend most of his or her time in designing, populating,
and maintaining a database. A limited amount of time
may be devoted to statistical thinking when counts,
sums, or averages are derived from the database. But
these two individuals will largely view themselves as
participating in greatly differing disciplines.
With dimensional modeling, the gap between data-
base theory and statistics begins to close. In dimen-
sional modeling we have to begin thinking in terms of
construct validity and temporal data. We need to think
about correlations between dependent and independent
226
Data Driven vs. Metric Driven Data Warehouse Design
variables. We begin to realize that the choice of data types
(e.g., interval or ratio) will affect the types of analysis we
can do on the data and will hence potentially limit the
queries. So the database designer has to address con-
cerns that have traditionally been the domain of the
statistician. Similarly, the statistician cannot afford the
luxury of constructing a data set for a single purpose or
a single type of analysis. The data set must be rich enough
to allow the statistician to find relationships that may not
have been considered when the data set was being con-
structed. Variables must be included that may potentially
have impact, may have impact at some times but not
others, or may have impact in conjunction with other
variables. So the statistician has to address concerns that
have traditionally been the domain of the database de-
signer.
What this points to is the fact that database design
and statistical exploitation are just different ends of the
same problem. After these two ends have been con-
nected by data warehouse technology, a single theory of
data must be developed to address the entire problem.
This unified theory of data would include entity theory
and measurement theory at one end and statistical ex-
ploitation at the other. The middle ground of this theory
will show how decisions made in database design will
affect the potential exploitations, so intelligent design
decisions can be made that will allow full exploitation
of the data to serve the organizations needs to model
itself in data.
CONCLUSION
Data warehousing is undergoing a theoretical shift from
a data-driven model to a metric-driven model. The met-
ric-driven model rests on a much firmer epistemologi-
cal foundation and promises a much richer and more
productive future for data warehousing. It is easy to haze
over the differences or significance between these two
approaches today. The purpose of this article was to
show the potentially dramatic, if somewhat speculative,
implications of the metric-driven approach.
REFERENCES
Artz, J. (2003). Data push versus metric pull: Compet-
ing paradigms for data warehouse design and their impli-
cations. In M. Khosrow-Pour (Ed.), Information tech-
nology and organizations: Trends, issues, challenges
and solutions. Hershey, PA: Idea Group Publishing.
Codd, E. F. (1970). A relational model of data for large
shared data banks. Communications of the ACM, 13(6),
377-387.
TEAM LinG
Data Driven vs. Metric Driven Data Warehouse Design
Codd, E. F. (1985, October 14). Is your DBMS really
relational? Computerworld.
Date, C. J. (2004). An introduction to database systems
(8th ed.). Addison-Wesley.
Fetzer, J., & Almeder, F. (1993). Glossary of epistemology/
philosophy of science. Paragon House.
Inmon, W. (2002). Building the data warehouse. Wiley.
Inmon, W. (2000). Exploration warehousing: Turning
business information into business opportunity. Wiley.
Jarke, M., Lenzerini, M., Vassiliou, Y., & Vassiliadis, P.
(2000). Fundamentals of data warehouses. Springer-
Verlag.
Kimball, R. (1996). The data warehouse toolkit. Wiley.
Kimball, R. (1998). The data warehouse lifecycle toolkit.
Wiley.
Kimball, R., & Merz, R. (2000). The data webhouse toolkit.
Wiley.
Palmer, D. (2001). Does the center hold? An introduction
to Western philosophy. McGraw-Hill.
Spofford, G. (2001). MDX Solutions. Wiley.
Teorey, T. (1999). Database modeling & design. Morgan
Kaufmann.
Thomsen, E. (2002). OLAP solutions. Wiley.
KEY TERMS
,
Data-Driven Design: A data warehouse design that
begins with existing historical data and attempts to
derive useful information regarding trends in the orga-
nization.
Data Warehouse: A repository of time-oriented
organizational data used to track the performance of key
business processes.
Dimensional Data Model: A data model that repre-
sents measurements of a process and the independent
variables that may affect that process.
Epistemology: A branch of philosophy that attempts
to determine the criteria for valid knowledge.
Key Business Process: A business process that can
be clearly defined, is measurable, and is worthy of
improvement.
Metric-Driven Design: A data warehouse design
that beings with the definition of metrics that can be
used to track key business processes.
Relational Data Model: A data model that repre-
sents entities in the form of data tables.
227
TEAM LinG
228
Data Management in Three-Dimensional
Structures
Xiong Wang
California State University at Fullerton, USA
INTRODUCTION
Data management in its general term refers to activities
that involve the acquisition, storage, and retrieval of data.
Traditionally, information retrieval is facilitated through
queries, such as exact search, nearest neighbor search,
range search, etc. In the last decade, data mining has
emerged as one of the most dynamic fields in the frontier
of data management. Data mining refers to the process of
extracting useful knowledge from the data. Popular data
mining techniques include association rule discovery,
frequent pattern discovery, classification, and clustering.
In this chapter, we discuss data management in a specific
type of data i.e., three-dimensional structures. While
research on text and multimedia data management has
attracted considerable attention and substantial progress
has been made, data management in three-dimensional
structures is still in its infancy (Castelli & Bergman, 2001;
Paquet & Rioux, 1999). Data management in 3D structures
raises several interesting problems:
1. Similarity search
2. Pattern discovery
3. Classification
4. Clustering
Given a database of 3D structures and a query 3D
structure, similarity search looks for those structures in
the database that match the query structure within a range
of tolerable errors. The similarity could be defined in two
different measurements. The first measurement compares
the data structure with the query structure in their entirety
i.e., a point-to-point match. We will call this aggregate
similarity search. The second measurement compares
only the contours or shapes of the data structure with that
of the query structure. This is generally referred to as
shape-based similarity search. The range of tolerable
errors specifies how close the match should be when the
data structure is aligned with the query structure. Pattern
discovery is concerned with similar substructures that
occur in multiple structures. Classification and clustering
when applied to these domains attempt to group 3D
structures with similar shapes or containing similar pat-
terns together.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
Three-dimensional structures can be used to describe
data in different domains. In biology and chemistry, for
example, a molecule is represented as a 3D structure with
connections. Each point is the center of an atom and the
connections are bonds between atoms. In computer-
aided design, an object is specified as a set of 3D vectors
that describes the shape of the object (Veltkamp, 2001;
Suzuki & Sugimoto, 2003). In computer vision, the shape
of a 3D object can be caught by X-ray or ultrasonic
scanning devices. The result is a set of 3D points (Hilaga,
Shinagawa, Kohmura, & Kunii, 2001). In medical imaging,
3D images of tissues or tumors can be collected using
magnetic resonance imaging or computer tomography
(Akutsu, Arakawa, & Murase, 2002). With advances in
the Internet, scanning devices, and storage, the World
Wide Web is becoming a huge reservoir of all kinds of
data. The 3D models available over the Internet dramati-
cally increased in the last two decades. Similarity search
is a highly desirable technique in all these domains.
Classification and clustering of biological data or chemi-
cal compounds have special significances. For example,
traditionally proteins are classified to families according
to their specific functions. However, recently, many ap-
proaches have been proposed to classify proteins ac-
cording to their structures. Some of these approaches
achieve very high accuracy when compared with their
biological counterparts. Classification and clustering can
also help build index structures in 3D model retrieval to
speed up similarity search.
Currently, there is not a universal model or framework
for the representation, storage, and retrieval of three-
dimensional structures. Most of these data are stored in
plain text files in some specific format. The format is
different from application to application. Likewise, the
existing techniques for information retrieval and data
mining in three-dimensional structures take root in the
areas of application. Two main areas of application are
computer vision and scientific data mining, where compu-
tation-intensive techniques have been developed and are
still in demand. We focus on data management in these
two areas.
TEAM LinG
Data Management in Three-Dimensional Structures
ADVANCES IN COMPUTER VISION
Shape-based recognition of 3D objects is a core problem
in computer vision and has been studied for decades.
There are roughly three categories of approaches: vol-
ume-based, feature-based, and interactive. For example,
Keim (1999) proposed a geometric-based similarity search
tree to deal with 3D volume-based search. He suggested
using voxels to approximate the 3D objects. For each 3D
object, a Minimum Surrounding Volume (MSV) and Maxi-
mum Including Volume (MIV) are constructed using voxels.
Similar objects are clustered in data pages and the MSV
and MIV approximations of the objects are stored in the
directory pages. Another interesting scheme uses
superellipsoids to approximate the shape of the volume.
Superellipsoids are similar to ellipsoids except the terms
in the definition are raised to parameterized exponents
which are not necessarily integers. Indexing the
superellipsoids is very difficult due to their different
shapes and sizes.
Feature-based approaches have been developed ex-
tensively in the literature. In (Belongie, Malik, & Puzicha,
2001, 2002), Belongie and co-authors designed a descrip-
tor, called the shape context, to characterize the distribu-
tion of the structure. For each point pi in the structure, the
set of 3D vectors originating from pi to every other point
in the structure is collected as the shape context. A
histogram is constructed based on comparison between
the shape context of the query shape and that of the data
shape. Osada, Funkhouser, Chazelle and Dobkin (2001)
suggested using a similar descriptor, called the shape
distribution. The shape distribution is a set of values that
are calculated by a shape function, such as the Euclidean
distance between two randomly selected points on the
surface of a 3D object. A histogram is calculated based on
the shape distributions of the two shapes under consid-
eration. Kriegel et al. introduced a 3D shape similarity
model that decomposes the 3D model into concentric
shells and sectors around the center point (Ankerst,
Kastenmller, Kriegel, & Seidl, 1999). The histograms are
determined by counting the number of points within each
cell. The similarity is calculated using a quadratic form
distance function. Korn, Sidiropoulos, Faloutsos, Siegel
and Protopapas (1998) proposed another descriptor, called
size distribution that is similar to the pattern spectrum.
They introduced a multi-scale distance function based on
mathematical morphology. The distance function is inte-
grated to the GEMINI framework to prune the search
space. GEMINI is an index structure for high dimensional
feature spaces. Bespalov, Shokoufandeh, Regli, & Sun
(2003) used Singular Value Decomposition to find suitable
feature vectors. A data level shape descriptor, called the
spin image, was used in (Johnson & Hebert, 1999) to match
surfaces represented as surface meshes. The system is
capable of recognizing multiple objects in 3D scenes that
contain clutter and occlusion. Saupe and Vranic (2001) ,
suggested using rays that cast from the center of the mass
of the object as feature vectors. The representation is
compared using spherical harmonics and moments. All
these descriptors are very high dimensional spaces and
indexing them is a well known difficult problem, due to
the curse of dimensionality. Furthermore, the approaches
are not suitable for pattern discovery.
An interactive search scheme was introduced in (Elad,
Tal, & Ar, 2001). The algorithm sets the center of a
structure to the origin and calculates the normalized
moment value of each point. The normalized moment
values are used to approximate the structure. Two struc-
tures are compared according to a weighted Euclidean
distance. The weights are adapted based on the feedback
from the user, using Singular Value Decomposition. Chui
and Rangarajan (2000) developed an iterative optimiza-
tion algorithm to estimate point correspondences and
image transformations, such as affine or thin plate splines.
The cost function is the sum of Euclidean distances
between the transformed query shape and the trans-
formed data shape. The distance function needs an initial
correspondence between the two structures. Interactive
refinement was also used in a medical image database
system developed by Lehmann et al. (2004). A survey of
shape matching in computer vision can be found in
(Veltkamp & Hagedoorn, 1999).
Shape-based similarity search in a large database of
3D objects is much more challenging and is still a young
research area. The most recent achievements are a search
engine developed at Princeton University (Funkhouser,
et al., 2003) and a search system 3DESS for 3D engineering
shapes built at Purdue University (Lou, Prabhakar, &
Ramani, 2004). These techniques were concentrated on
computer vision and they did not compare the 3D models
point-to-point. Many of them only search for 3D objects
that look similar. The effectiveness of such techniques
often depends on subjective perception.
EMERGING TECHNIQUES IN
SCIENTIFIC DATA MINING
In structural biology, detecting similarity in protein struc-
tures has been a useful tool for discovering evolutionary
relationships, analyzing functional similarity, and pre-
dicting protein structures. The differences in proteins or
chemical compounds are very subtle. To discover func-
tionally related proteins or chemical compounds, in many
cases we have to match them atom-to-atom. Aggregate
similarity search is known as an NP complete problem in
combinatorial pattern matching. Even though the signifi-
cance of the problem surfaced with applications to
229
TEAM LinG

bioinformatics, few practical approaches have been devel-
oped. The latest results are a performance study based on
2D structures that was conducted by Gavrilov, Indyk,
Motwani and Venkatasubramanian at Stanford (1999) and
a novel algorithm that was developed by Wang and col-
laborators in (Wang & Wang, 2000). The algorithm discov-
ers patterns in 3D data with no assumptions of edges. The
research group headed by Karypis at University of Minne-
sota studies pattern discovery in graphs, with vertices and
edges (Kuramochi & Karypis, 2002).
Pattern discovery is another highly desirable tech-
nique in these domains. A motif is a substructure in
proteins that has specific geometric arrangement and, in
many cases, is associated with a particular function, such
as DNA binding. Active sites are another type of patterns
in protein structures. They play an important role through
protein-protein and protein-ligand interaction, i.e. the bind-
ing process. In drug design, scientists try to determine the
binding sites of the target molecules and seek inhibitors
that can be bound to it. For example in determining the
structure of HIV protease and looking for effective inhibi-
tors, over 120 of these structure determinations have been
done and at least two inhibitors of HIV protease are now
being regularly used to treat AIDS. In the past two de-
cades, the number of 3D protein structures dramatically
increased. The Protein Data Bank maintains 26,800 entries
as of August 2004. New structures are deposited every-
day. Performing similarity search and pattern discovery in
such an enormous data set urgently demands highly effi-
cient computational tools. Wang and coauthors (2002)
developed a framework for discovering frequently occur-
ring patterns in 3D structures and applied the approach to
scientific data mining. The algorithm is a variant of the
geometric hashing technique invented for model based
recognition in computer vision in 1988 (Lamdan & Wolfson,
1988). Similar approaches were also introduced in
(Verbitsky, Nussinov, & Wolfson, 1999). Since hash func-
tions map floating point numbers to integers when calcu-
lating the hash bin addresses, they do not preserve precise
information of the data. As a consequence, these ap-
proaches are not suitable for similarity search that allows
variable ranges of tolerable errors. Furthermore, false
matches must be filtered out via a verification process. It
is well known in the literature that geometric hashing is too
sensitive to noise in the data. Due to regularity of biologi-
cal and chemical structures, dissimilarity is often very
subtle. Inaccuracy introduced by the scanning devices
adds noise to the data (Ankerst, Kastenmller, & Kriegel,
Seidl, 1999). It is extremely difficult to choose a fixed range
of tolerable errors, especially, when the data are collected
by different domain experts, using different equipments,
such as in the case of Protein Data Bank (Berman, et al.,
2000; Westbrook, et al., 2002). It is critical that the range
of tolerable errors be set to a tunable parameter, so that the
230
Data Management in Three-Dimensional Structures
domain expert can choose an optimal value according to
the context. A new index structure, B+ Tree, was intro-
duced by Wang in 2002 to overcome these weaknesses
(Wang, 2002). The B+ Trees preserve precise informa-
tion of the data, allow pattern discovery with variable
ranges of tolerable errors, and remove false matches
totally. In an attempt to compare the shapes of 3D
structures, Wang also invented a new notion called the
a-surface to capture the surface of a 3D structure in
variable details (Wang, 2001).
FUTURE TRENDS
A universal model or framework for the representation,
storage, and retrieval of three-dimensional structures is
highly desirable. However, due to the different nature
and applications in the two areas, a model or framework
that fits both is not feasible. In computer vision and 3D
model retrieval, the number of points in each object is
huge and a single point does not impact perception
substantially. In fact, a point that does not get along with
other points is likely to be considered an outlier or noise.
The number of objects increases tremendously everyday
in different sources over the Internet. We are likely to see
a 3D model search engine much like the search engine for
text documents. A centralized database for 3D models is
possible, but will be only a small portion of the available
search space. The search engine will be capable to access
data in different formats to answer the user query. Shape-
based statistical approaches will remain a main stream.
On the other hand, in scientific data, each individual
point is very important. For example, it may represent the
center of an atom. Each point may also be associated with
different properties. Thus blindly searching through
different sources for data in different formats is not
practical. Instead, we are likely to see large centralized
data reservoir like The Protein Data Bank. For most of the
applications, pattern discovery, classification, and clus-
tering are most desirable techniques.
CONCLUSION
The human interest about three-dimensional structures
began even before Euclid. After centuries of investiga-
tion and learning, it became clear that although we have
a better perception of 3D objects compared with our
comprehension of text and multimedia data our tech-
niques in the representation, storage, search, and dis-
covery of 3D structures are lagged far behind. In com-
puter vision, 3D object recognition is a well know difficult
problem. With advances in computational power and
storage devices, maintaining an enormous amount of 3D
TEAM LinG
Data Management in Three-Dimensional Structures
structures is not only feasible but also inexpensive. How-
ever, few effective approaches have been developed to
facilitate similarity search and pattern discovery in a very
large database of 3D structures. Applications of 3D struc-
tures to bioinformatics pose even more intricate challenges,
due to the nature that these structures often need to be
compared point-to-point. Fortunately, research in these
topics has attracted more and more attention from computer
scientists in different areas, such as computer vision, data-
base systems, artificial intelligence, etc. It can be foreseen
that substantial progress will be achieved and novel tech-
niques will emerge in the near future.
REFERENCES
Akutsu, T., Arakawa, K., & Murase H. (2002). Shape from
contour using adaptive image selection. Systems and
Computers in Japan, 33(11), 50-60.
Ankerst, M., Kastenmller, G., Kriegel, H-P., & Seidl, T.
(1999). Nearest neighbor classification in 3D protein da-
tabases. Proc. of the 7th International Conference on
Intelligent Systems for Molecular Biology (pp. 34-43),
Heidelberg, Germany.
Belongie, S. Malik, J., & Puzicha, J. (2001). Matching
shapes. Proc. of the Eighth International Conference on
Computer Vision (pp. 454-463), Los Alamitos, California.
Belongie, S., Malik, J., & Puzicha, J. (2002). Shape match-
ing and object recognition using shape contexts. IEEE
Transactions on Pattern Analysis and Machine Intelli-
gence, 24(4), 509-522.
Berman, H.M., Westbrook, J., Feng, Z., Gilliland, G., Bhat,
T.N., Weissig, H., et al. (2000). The protein data bank.
Nucleic Acids Research, 28(1), 235-242.
Bespalov, D., Shokoufandeh, A., Regli, W.C., & Sun, W.
(2003). Scale-space representation of 3D models and to-
pological matching. ACM Symposium on Solid Modeling
and Applications (pp. 208-215).
Castelli, V., & Bergman, L. (2001). Image databases:
Search and retrieval of digital imagery. John Wiley & Sons.
Chui, H., & Rangarajan A. (2000). A new algorithm for non-
rigid point matching. Proc. of the IEEE Conference on
Computer Vision and Pattern Recognition (pp. 44-51),
Hilton Head Island, South Carolina.
Elad, M., Tal, A., & Ar, S. (2001), Content based retrieval
of VRML objects An iterative and interactive approach.
Proc. of the Eurographics Workshop on Multimedia (pp.
97-108), Manchester, UK.
Funkhouser, T.A., Min, P. Kazhdan, M.M., Chen, J.,
Halderman, A., Dobkin, D.P., et al. (2003). A search engine ,
for 3D models. ACM Transactions on Graphics, 22(1), 83-105.
Gavrilov, M., Indyk, P., Motwani, R., &
Venkatasubramanian, S. (1999). Geometric pattern match-
ing: A performance study. Proc. of the Fifteenth Annual
Symposium on Computational Geometry (pp. 79-85),
Miami Beach, Florida.
Hilaga, M., Y. Shinagawa, Y., T. Kohmura, T., & T. L. Kunii
T.L. (2001). Topology matching for fully automatic simi-
larity estimation of 3D shapes. SIGGRAPH, 203-212.
Johnson, A.E., & Hebert, M. (1999). Using spin images for
efficient object recognition in cluttered 3D scenes. IEEE
Transactions on Pattern Analysis and Machine Intelli-
gence, 21(5), 433-449.
Keim, D.A. (1999). Efficient geometry-based similarity
search of 3D spatial databases. Proc. of ACM SIGMOD
International Conference on Management of Data (pp.
419-430), Philadephia, Pennsylvania.
Korn, F., Sidiropoulos, N., Faloutsos, C., Siegel, E., &
Protopapas, Z. (1998). Fast and effective retrieval of
medical tumor shapes. IEEE Transactions on Knowledge
and Data Engineering, 10(6), 889-904.
Kuramochi, M., & Karypis, G. (2002). Discovering geo-
metric frequent subgraphs. Proc. of the 2002 IEEE Inter-
national Conference on Data Mining (pp. 258-265),
Maebashi, Japan.
Lamdan, Y., & Wolfson, H. (1988). Geometric hashing: A
general and efficient model-based recognition scheme.
Proc. of International Conference on Computer Vision
(pp. 237-249).
Lehmann, T.M., Plodowski, B., Spitzer, K., Wein, B.B.,
Ney, H., & Seidl, T. (2004). Extended query refinement for
content-based access to large medical image databases.
Proc. of SPIE Medical Imaging (pp. 90-98).
Lou, K., Prabhakar, S., & Ramani, K. (2004). Content-
based three-dimensional engineering shape search. Proc.
of the 2004 IEEE International Conference on Data
Engineering (pp. 754-765), Boston.
Osada, R., Funkhouser, T., Chazelle, B., & Dobkin, D.
(2001). Matching 3D models with shape distributions.
Proc. of the International Conference on Shape Model-
ing and Applications, Genova, Italy.
Paquet, E., & Rioux, M. (1999). Crawling, indexing and
retrieval of three-dimensional data on the web in the
framework of MPEG-7. VISUAL, 179-18.
231
TEAM LinG

Saupe, D., & Vranic, D. V. (2001). 3D model retrieval with
spherical harmonics and moments. DAGM-Symposium
2001 (pp. 392-397).
Suzuki, M.T., & Sugimoto, Y.Y. (2003). A search method
to find partially similar triangular faces from 3D polygonal
models. Modeling and Simulation, 323-328.
Veltkamp, R. C. (2001). Shape matching: Similarity mea-
sures and algorithms. IEEE Shape Modeling Interna-
tional, 188-197.
Veltkamp, R. C., & Hagedoorn, M. (1999). State-of-the-art
in shape matching. Technical Report UU-CS-1999-27,
Utrecht.
Verbitsky, G., Nussinov, R., & Wolfson, H. J. (1999).
Flexible structural comparison allowing hinge bending
and swiveling motions. Proteins, 34, 232-254.
Wang, X. (2001). -Surface and its application to mining
protein data. Proc. of the IEEE International Conference
on Data Mining (pp. 659-662), San Jose, California.
Wang, X. (2002). B+ tree: Indexing 3D point sets for
pattern discovery. Proc. of the IEEE International Con-
ference on Data Mining (pp. 701-704), Maebashi, Japan.
Wang X., & Wang, J.T.L. (2000). Fast similarity search in
three-dimensional structure databases. Journal of Chemi-
cal Information and Computer Sciences, 40(2), 442-451.
Wang, X., Wang, J.T.L., Shasha, D., Shapiro, B.A.,
Rigoutsos, I., & Zhang, K. (2002). Finding patterns in
three dimensional graphs: Algorithms and applications to
scientific data mining. IEEE Transaction on Knowledge
and Data Engineering, 14(4), 731-749.
Westbrook, J., Feng, Z., Jain, S., Bhat, T. N., Thanki, N.,
Ravichandran, V. et al. (2002). The protein data bank:
Unifying the archive. Nucleic Acids Research, 30(1), 245-
248.
232
Data Management in Three-Dimensional Structures
KEY TERMS
B+ Tree: An index structure that decomposes a 3D
structure into point-triplets and indexes the triplets in a
three-dimensional B + tree.
-Surface: The surface of a 3D structure that is
constructed by rolling a solid ball with radius along the
contour and extracting every point the solid ball touches.
Aggregate Similarity Search: The search operation
in 3D structures that matches the structures point-to-
point in their entirety.
Feature Vector: A vector in which every dimension
represents a property of a 3D structure. A good feature
vector captures similarity and dissimilarity of 3D struc-
tures.
Histogram: The original term refers to a bar graph that
represents a distribution. In information retrieval, it also
refers to a weighted vector that describes the properties
of an object or the comparison of properties between two
objects. Like a feature vector a good histogram captures
similarity and dissimilarity of the objects of interest.
Shape-Based Similarity Search: The search opera-
tion in 3D structures that matches only the surfaces of the
structures without referring to the points inside the sur-
faces.
Superimposition: The process of matching two 3D
structures by alignment through rigid translations and
rotations.
The Curse of Dimensionality: The original term refers
to the exponential growth of hyper-volume as a function
of dimensionality. In information retrieval, it refers to the
phenomenon that the performance of an index structure
for nearest neighbor search and -range search deterio-
rates rapidly due to the growth of hyper-volume.
TEAM LinG
 233

Data Mining and Decision Support for ,

Business and Science
Auroop R. Ganguly
Oak Ridge National Laboratory, USA

Amar Gupta
University of Arizona, USA

Shiraj Khan
University of South Florida, USA

INTRODUCTION
Analytical Information Technologies
Information by itself is no longer perceived as an asset.
Billions of business transactions are recorded in enter-
prise-scale data warehouses every day. Acquisition, stor-
age, and management of business information are com-
monplace and often automated. Recent advances in re-
mote or other sensor technologies have led to the devel-
opment of scientific data repositories. Database tech-
nologies, ranging from relational systems to extensions
like spatial, temporal, time series, text, or media, as
well as specialized tools like geographical information
systems (GIS) or online analytical processing (OLAP),
have transformed the design of enterprise-scale busi-
ness or large scientific applications. The question in-
creasingly faced by the scientific or business decision-
maker is not how one can get more information or design
better information systems but what to make of the infor-
mation and systems already in place. The challenge is to
be able to utilize the available information, to gain a better
understanding of the past, and to predict or influence the
future through better decision making. Researchers in
data mining technologies (DMT) and decision support
systems (DSS) are responding to this challenge. Broadly
defined, data mining (DM) relies on scalable statistics,
artificial intelligence, machine learning, or knowledge
discovery in databases (KDD). DSS utilize available infor-
mation and DMT to provide a decision-making tool usu-
ally relying on human-computer interaction. Together,
DMT and DSS represent the spectrum of analytical infor-
mation technologies (AIT) and provide a unifying plat-
form for an optimal combination of data dictated and
human-driven analytics.

Table 1. Analytical information technologies Table 2. Application examples

Data-Mining Technologies Science and Engineering

Association, correlation, clustering, Bio-Informatics
classification, regression, database Genomics
knowledge discovery
Hydrology, Hydrometeorology
Signal and image processing, nonlinear
systems analysis, time series and spatial Weather Prediction
statistics, time and frequency domain Climate Change Science
analysis Remote Sensing
Expert systems, case-based reasoning, Smart Infrastructures
system dynamics Sensor Technologies
Econometrics, management science Land Use, Urban Planning
Decision Support Systems Materials Science
Automated analysis and modeling Business and Economics
o Operations research Financial Planning
o Data assimilation, estimation, and Risk Analysis
tracking
Supply Chain Planning
Human computer interaction
Marketing Plans
o Multidimensional OLAP and
spreadsheets Text and Video Mining
o Allocation and consolidation Handwriting/Speech Recognition
engine, alerts Image and Pattern Recognition
o Business workflows and data Long-Range Economic Planning
sharing Homeland Security

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
 Data Mining and Decision Support for Business and Science
BACKGROUND
Tables 1 and 2 describe the state of the art in DMT and DSS
for science and business, and provide examples of their
applications.
Researchers and practitioners have reviewed the
state of the art in analytic technologies for business
(Apte et al., 2002; Kohavi et al., 2002; Linden & Fenn, 2003)
or science (Han et al., 2002), as well as data mining methods,
software, and standards (Fayyad & Uthurusamy, 2002;
Ganguly, 2002a; Grossman et al., 2002; Hand et al., 2001;
Smyth et al., 2002) and decision support systems (Carlsson
& Turban, 2002; Shim et al., 2002).
MAIN THRUST
Scientific and Business Applications
Rapid advances in information and sensor technologies
(IT and ST) along with the availability of large-scale
scientific and business data repositories or database
management technologies, combined with breakthroughs
in computing technologies, computational methods, and
processing speeds, have opened the floodgates to data-
dictated models and pattern matching (Fayyad &
Uthurusamy, 2002; Hand et al., 2001). The use of so-
phisticated and computationally-intensive analytical
methods is expected to become even more common-
place with recent research breakthroughs in computa-
tional methods and their commercialization by leading
vendors (Bradley et al., 2002; Grossman et al., 2002;
Smyth et al., 2002).
Scientists and engineers have developed innovative
methodologies for extracting correlations and associa-
tions, dimensionality reduction, clustering or classifi-
cation, regression, and predictive modeling tools based
on expert systems and case-based reasoning, as well as
decision support systems for batch or real-time analy-
sis. They have utilized tools from areas like traditional
statistics, signal processing, and artificial intelligence,
as well as emerging fields like data mining, machine
learning, operations research, systems analysis, and
nonlinear dynamics. Innovative models and newly dis-
covered patterns in complex, nonlinear, and stochastic
systems encompassing the natural and human environ-
ments have demonstrated the effectiveness of these
approaches. However, applications that can utilize these
tools in the context of scientific databases in a scalable
fashion have only begun to emerge (Curtarolo et al.,
2003; Ganguly, 2002b; Grossman & Mazzucco, 2002;
Grossman et al., 2001; Han et al., 2002; Kamath et al.,
2002; Thompson et al., 2002).
234
Business solution providers and IT vendors, on the
other hand, have focused primarily on scalability, pro-
cess automation and workflows, and the ability to com-
bine results from relatively simple analytics with judg-
ments from human experts. For example, e-business
applications in the areas of supply-chain planning, fi-
nancial analysis, and business forecasting traditionally
rely on decision-support systems with embedded data
mining, operations research and OLAP technologies,
business intelligence (BI), and reporting tools, as well
as an easy-to-use GUI (graphical user interface) and
extensible business workflows (Geoffrion & Krishnan,
2003). These applications can be custom built by utiliz-
ing software tools or are available as prepackaged e-
business application suites from large vendors like SAP,
PeopleSoft, and Oracle, as well as best of breed and
specialized applications from smaller vendors like
Seibel and i2 . A recent report by the market research
firm Gartner (Linden & Fenn, 2003) summarizes the
relative maturity and current industry perception of
advanced analytics. For reasons ranging from excessive
(IT) vendor hype to misperceptions among end users
caused by inadequate quantitative background, the busi-
ness community is barely beginning to realize the value
of data-dictated predictive, analytical, or simulation
models. However, there are notable exceptions to this
trend (Agosta et al., 2003; Apte et al., 2002; Geoffrion
& Krishnan, 2003; Kohavi et al., 2002; Wang & Jain, 2003;
Yurkiewicz, 2003).
Solutions Utilizing DMT and DSS
For a scientist or an engineer, as well as for a business
manager or management scientist, DMT and DSS are
tools used for developing domain-specific applications.
These applications might combine knowledge about the
specific scientific or business domain (e.g., through the
use of physically-based or conceptual-scientific mod-
els, business best practices and known constraints, etc.)
with data-dictated or decision-making tools like DSS
and DMT. Within the context of these applications,
DMT and DSS can aid in the discovery of novel patterns,
development of predictive or descriptive models, miti-
gation of natural or manmade hazards, preservation of
civil societies and infrastructures, improvement in the
quality and span of life as well as in economic prosperity
and well being, and development of natural and built
environments in a sustainable fashion. Disparate appli-
cations utilizing DMT and DSS tools tend to have inter-
esting similarities. Examples of current best practices
in the context of business and scientific applications are
provided next.
Business forecasting, planning, and decision sup-
port applications (Carlsson & Turban, 2002; Shim et al.,
TEAM LinG
Data Mining and Decision Support for Business and Science
2002; Wang & Jain, 2003; Yurkiewicz, 2003) usually need
to read data from a variety of sources like online transac-
tional processing (OLTP) systems, historical data ware-
houses and data marts, syndicated data vendors, legacy
systems, or public domain sources like the Internet, as well
as in the form of real-time or incremental data entry from
external or internal collaborators, expert consultants, plan-
ners, decision makers, and/or executives. Data from dis-
parate sources usually are mapped to a predefined com-
mon data model and incorporated through extraction, trans-
formation, and loading (ETL) tools. End users are provided
GUI-based access to define application contexts and set-
tings, structure business workflows and planning cycles
and format data models for visualization, judgmental up-
dates, or analytical and predictive modeling. The param-
eters of the embedded data mining models might be preset,
calculated dynamically based on data or user inputs, or
specified by a power user. The results of the data mining
models can be utilized automatically for optimization and
recommendation systems and/or can be used to serve as
baselines for planners and decision makers. Tools like BI,
Reports, and OLAP (Hammer, 2003) are utilized to help
planners and decision makers visualize key metrics and
predictive modeling results, as well as to utilize alert
mechanisms and selection tools to manage by exception or
by objectives. Judgmental updates at various levels of
aggregation and their reconciliation, collaboration among
internal experts and external trading partners, as well as
managerial review processes and adherence to corporate
directives, are aided by allocation and consolidation
engines, tools for simulation, ad hoc and predefined
reports, user-defined business workflows, audit trails
with comments and reason codes, and flexible informa-
tion transfer and data-handling capabilities.
Emerging technologies include the use of automated
DMT for aiding traditional DSS tasks; for example, the
use of data mining to zero down on the cause of aggregate
exceptions in multidimensional OLAP cubes. The end
results of the planning process are usually published in a
pre-defined placeholder (e.g., a relational database table),
which, in turn, can be accessed by execution systems or
other planning applications. The use of elaborate mecha-
nisms for user-driven analysis and judgmental or col-
laborative decisions, as opposed to reliance on auto-
mated DMT, remains a guiding principle for the current
genre of business-planning applications. The value of
collaborative decision making and global visibility of
information is near axiomatic for business applications.
However, researchers need to design better DMT applica-
tions that can utilize available information from disparate
sources through advanced analytics and account for spe-
cific domain knowledge, constraints, or bottlenecks. Valu-
able and/or scarce human resources can be conserved by
automating routine tasks and by reserving expert resources
for high value-added jobs (e.g., after a Pareto classifica-
tion) or for exceptional situations (e.g., large prediction ,
variance or significant risks). In addition, certain research
studies have indicated that judgmental overrides may not
improve upon the results of automated predictive models
on the (longer-term) average.
Scientists and engineers traditionally have utilized
advanced quantitative approaches for making sense of
observations and experimental results, formulating theo-
ries and hypotheses, and designing experiments. For
users of statistical and numerical approaches in these
domains, DMT often seems like the proverbial old wine
in new bottles. However, innovative use of DMT in-
cludes the development of algorithms, systems, and
practices that can not only apply novel methodologies,
but also can scale to large scientific data repositories
(Connover et al., 2003; Graves, 2003; Han et al., 2002; He
et al., 2003; Ramachandran et al., 2003). While scientific
and business data mining have a lot in common, the
incorporation of domain knowledge is probably more
critical in scientific applications. When appropriately
combined with domain specific knowledge about the
physics or the data sources/uncertainties, DMT ap-
proaches have the potential to revolutionize the pro-
cesses of scientific discovery, verification, and predic-
tion (Han et al., 2002; Karypis, 2002). This potential has
been demonstrated by recent applications in diverse
areas like remote sensing (Hinke et al., 2000), material
sciences (Curtarolo et al., 2003), bioinformatics (Graves,
2003), and the earth sciences (Ganguly, 2002b; Kamath et
al., 2002; Potter et al., 2003; Thompson et al., 2002; see
http://datamining.itsc.uah.edu/adam/). Besides physi-
cal and data-dictated methods, human-computer interac-
tion retains a significant role in real-world scientific
decision making. This necessitates the use of DSS, where
the results of DMT can be combined with expert judg-
ment and techniques from simulation, operations re-
search (OR), and other DSS tools. The Reviews of
Geophysics (American Geophysical Union, 1995) pro-
vides a slightly dated discussion on the use of data
assimilation, estimation, and OR, as well as DSS, (http:/
/www.agu.org/journals/rg/rg9504S/contents.html# hy-
drology). Examples of decision support systems and
tools in scientific and engineering applications also
can be found in dedicated journals like Decision Sup-
port Systems or Journal of Decision Systems (see
Vol. 8, Number 2, 1998, and the latest issues), as well
as in journals or Web sites dealing with scientific and
engineering topics (McCuistion & Birk, 2002) (see the
NASA air traffic control Web site at http://
www.asc.nasa.gov/aatt/dst.html; NASA research Web
sites for a global carbon DSS http://geo.arc.nasa.gov/
website/cquestwebsite/index.html; and institutes like
235
TEAM LinG
 Data Mining and Decision Support for Business and Science
MIT Lincoln Laboratories at http://www.ll.mit.edu/
AviationWeather/index2.html).
FUTURE TREND
Business applications have focused on DSS with embed-
ded and scalable implementations of relatively straight-
forward DMT. Scientific applications have focused tradi-
tionally on advanced DMT in prototype applications with
sample data. Researchers and practitioners of the future
need to utilize advanced DMT for business applications
and scalable DMT, and DSS for scientists and engineers.
This provides a perfect opportunity for innovative and
multi-disciplinary collaborations.
CONCLUSION
The power of information technologies has been uti-
lized to acquire, manage, store, retrieve, and represent
data in information repositories, and to share, report,
process, collaborate on, and move data in scientific and
business applications. Database management and data
warehousing technologies have matured significantly
over the years. Tools for building custom and packaged
applications, including, but not limited to, workflow
technologies, Web servers, and GUI-based data entry
and viewing forms, are steadily maturing. There is a
clear and present need to exploit the available data and
technologies to develop the next generation of scien-
tific and business applications, which can combine data-
dictated methods with domain-specific knowledge. Ana-
lytical information technologies, which include DMT
and DSS, are particularly suited for these tasks. These
technologies can facilitate both automated (data-dic-
tated) and human expert-driven knowledge discovery
and predictive analytics, and can also be made to utilize
the results of models and simulations that are based on
process physics or business insights. If DMT and DSS
were to be defined broadly, a broad statement can per-
haps be made that, while business applications have
scalable but straightforward DMT embedded within DSS,
scientific applications have utilized advanced DMT but
focused less on scalability and DSS. Multidisciplinary
research and development efforts are needed in the
future for maximal utilization of analytical information
technologies in the context of these applications.
ACKNOWLEDGMENTS
Most of the work was completed while the first author was
on a visiting faculty appointment at the University of
236
South Florida and the second author was a member of the
faculty at the MIT Sloan School of Management.
REFERENCES
Agosta, L., Orlov, L.M., & Hudso R. (2003). The future
of data mining: Predictive analytics. Forrester Brief.
Apte, C., Liu, B., Pednault, E.P.D., & Smyth, P. (2002,
August). Business applications of data mining. Commu-
nications of the ACM, 45(8), 49-53.
Bradley, P. et al. (2002). Scaling mining algorithms to large
databases. Communications of the ACM, 45(8), 38-43.
Carlsson, C., & Turban, E. (2002). DSS: Directions for the next
decade. Decision Support Systems, 33(2), 105-110.
Conover, H. et al. (2003). Data mining on the TeraGrid.
Proceedings of the Supercomputing Conference, Phoe-
nix, Arizona.
Curtarolo, S. et al. (2003). Predicting crystal structures
with data mining of quantum calculations. Physics Review
Letters, 91(13).
Fayyad, U., & Uthurusamy, R. (2002, August). Evolving
data mining into solutions for insights. Communica-
tions of the ACM, 45(8), 28-31.
Ganguly, A.R. (2002a). Software reviewData mining
components. ORMS Today, 29(5), 56-59.
Ganguly, A.R. (2002b). A hybrid approach to improving
rainfall forecasts. Computers in Science and Engi-
neering, 4(4), 14-21.
Geoffrion, A.M., & Krishnan, R. (Eds.). (2003). E-
business and management scienceMutual impacts
(Parts 1 and 2). Management Science, 49(10-11).
Graves, S.J. (2003). Data mining on a bioinformatics
grid. Proceedings of the SURA BioGrid Workshop,
Raleigh, North Carolina.
Grossman, R. et al. (2001). Data mining for scientific and
engineering applications. Kluwer.
Grossman, R.L., Hornick, M.F., & Meyer, G. (2002). Data
mining standards initiative. Communications of the ACM,
45(8), 59-61.
Grossman, R.L., & Mazzucco, M. (2002). DataSpace: A data
Web for the exploratory analysis and mining of data.
Computers in Science and Engineering, 4(4), 44-51.
Hammer, J. (Ed.). (2003). Advances in online analytical
processing. Data & Knowledge Engineering, 45(2), 127-
256.
TEAM LinG
Data Mining and Decision Support for Business and Science
Han, J. et al. (2002). Emerging scientific applications in data
mining. Communications of the ACM, 45(8), 54-58.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of
data mining. Cambridge, MA: MIT Press.
He, Y. et al. (2003). Framework for mining and analysis of
space science data. Proceedings of the SIAM Interna-
tional Conference on Data Mining, San Francisco, Cali-
fornia.
Hinke, T., Rushing, J., Ranganath, H.S., & Graves, S.J.
(2000). Techniques and experience in mining remotely
sensed satellite data. Artificial Intelligence Review,
14(6), 503-531.
Kamath, C. et al. (2002). Classifying of bent-double galaxies.
Computers in Science and Engineering, 4(4), 52-60.
Karypis, G. (2002). Guest editors introduction: Data mining.
Computers in Science and Engineering, 4(4), 12-13.
Kohavi, R., Rothleder, N.J., & Simoudis, E. (2002). Emerg-
ing trends in business analytics. Communications of the
ACM, 45(8), 45-48.
Linden, A., & Fenn, J. (2003). Hype cycle for advanced
analytics, 2003. Gartner Strategic Analysis Report.
McCuistion, J.D., & Birk, R. (2002). From observations
to decision support: The new paradigm for satellite data.
NASA Technical Report. Retrieved from http://
www.iaanet.org/symp/berlin/IAA-B4-0102.pdf
Potter, C. et al. (2003). Global teleconnections of ocean
climate to terrestrial carbon flux. Journal of Geophysical
Research, American Geophysical Union, 108(D17), 4556.
Ramachandran, R. et al. (2003). Flexible framework for
mining meteorological data. Proceedings of the Ameri-
can Meteorological Societys (AMS) 19th International
Conference on Interactive Information Processing
Systems (IIPS) Meteorology, Oceanography, and Hy-
drology, Long Beach, California.
Shim, J. et al. (2002). Past, present, and future of decision
support technology. Decision Support Systems, 33(2),
111-126.
Smyth, P., Pregibon, D., & Faloutsos, C. (2002). Data-
driven evolution of data mining algorithms. Communica-
tions of the ACM, 45(8), 33-37.
Thompson, D.S. et al. (2002). Physics-based feature min-
ing for large data exploration. Computers in Science and
Engineering, 4(4), 22-30.
Wang, G.C.S., & Jain, C.L. (2003). Regression analysis:
Modeling and forecasting. Institute of Business Fore- ,
casting.
Yurkiewicz, J. (2003). Forecasting software survey: Pre-
dicting which product is right for you. ORMS Today.
KEY TERMS
Analytical Information Technologies (AIT): Informa-
tion technologies that facilitate tasks like predictive mod-
eling, data assimilation, planning, or decision making
through automated data-driven methods, numerical solu-
tions of physical or dynamical systems, human-computer
interaction, or a combination. AIT includes DMT, DSS, BI,
OLAP, GIS, and other supporting tools and technologies.
Business and Scientific Applications: End-user mod-
ules that are capable of utilizing AIT along with domain-
specific knowledge (e.g., business insights or constraints,
process physics, engineering know-how). Applications
can be custom-built or pre-packaged and are often distin-
guished form other information technologies by their
cognizance of the specific domains for which they are
designed. This can entail the incorporation of domain-
specific insights or models, as well as pre-defined infor-
mation and process flows.
Business Intelligence (BI): Broad set of tools and
technologies that facilitate management of business
knowledge, performance, and strategy through auto-
mated analytics or human-computer interaction.
Data Assimilation: Statistical and other automated
methods for parameter estimation, followed by predic-
tion and tracking.
Data Mining Technologies (DMT): Broadly de-
fined, these include all types of data-dictated analytical
tools and technologies that can detect generic and inter-
esting patterns, scale (or can be made to scale) to large
data volumes, and help in automated knowledge discov-
ery or prediction tasks. These include determining asso-
ciations and correlations, clustering, classifying, and
regressing, as well as developing predictive or forecast-
ing models. The specific tools used can range from
traditional or emerging statistics and signal or image
processing, to machine learning, artificial intelligence,
and knowledge discovery from large databases, as well
as econometrics, management science, and tools for
modeling and predicting the evolutions of nonlinear dy-
namical and stochastic systems.
237
TEAM LinG
 Data Mining and Decision Support for Business and Science
Decision Support Systems (DSS): Broadly defined,
these include technologies that facilitate decision making.
These can embed DMT and utilize these through auto-
mated batch processes and/or user-driven simulations or
what-if scenario planning. The tools for decision support
include analytical or automated approaches like data as-
similation and operations research, as well as tools that
help the human experts or decision makers manage by
objectives or by exception, like OLAP or GIS.
Geographical Information Systems (GIS): Tools that
rely on data management technologies to manage, pro-
cess, and present geospatial data, which, in turn, can vary
with time.
Online Analytical Processing (OLAP): Broad set of
technologies that facilitate drill-down or aggregate analy-
238
ses, as well as presentation, allocation, and consolidation
of information along multiple dimensions (e.g., product,
location, and time). These technologies are well suited for
management by exceptions or objectives, as well as auto-
mated or judgmental decision making.
Operations Research (OR): Mathematical and con-
straint programming and other techniques for mathemati-
cally or computationally determining optimal solutions
for objective functions in the presence of constraints.
Predictive Modeling: The process through which
mathematical or numerical technologies are utilized to
understand or reconstruct past behavior and predict
expected behavior in the future. Commonly utilized
tools include statistics, data mining, and operations
research, as well as numerical or analytical methodolo-
gies that rely on domain-knowledge.
TEAM LinG

Data Mining and Warehousing in Pharma
Industry
Andrew Kusiak
The University of Iowa, USA
Shital C. Shah
The University of Iowa, USA
INTRODUCTION
Most processes in pharmaceutical industry are data
driven. Companys ability to capture the data and making
use of it will grow in significance and may become the
main factor differentiating the industry. Basic concepts
of data mining, data warehousing, and data modeling are
introduced. These new data-driven concepts lead to a
paradigm shift in pharmaceutical industry.
BACKGROUND
The volume of data in pharmaceutical industry has been
growing at an unprecedented rate. For example, a
microarray (equivalent of one test) may provide thou-
sands of data points. These huge datasets offer chal-
lenges and opportunities. The primary challenges are
data storage, management, and knowledge discovery.
The pharmaceutical industry is one of the most data-
driven industries and getting value out of the data is key
to its success. Thus processing the data with novel tools
and methods for extracting useful knowledge for speedy
drug discovery and optimal matching of the drugs with
the patients may well become the main factor differen-
tiating the pharmaceutical companies.
The main sources of pharmaceutical data are pa-
tients, genetic tests, regulatory sources, pharmaceuti-
cal literature, and so on. Data and information collected
from different sources, agencies, and clinics, has been
traditionally used for narrow reporting (Berndt et al.,
2001). Massive clinical trials may lead to errors such as
protocol violations, data integrity, data formats, trans-
fer errors, and so on. Traditionally the analysis in phar-
maceutical industry [from prediction of drug stability
(King et al., 1984) to drug discovery techniques (Tye,
2004)] is performed using the population based statis-
tical techniques. To minimize possible errors and in-
crease confidence in predictions there is a need for a
comprehensive methodology for data collection, stor-
age, and analysis. Automation of the pharmaceutical data
over its lifecycle seems to be an obvious alternative.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
239
,
The vision for pharmaceutical industry is captured in
Figure 1. It has become apparent that data will play a
central role in drug discovery and delivery to an individual
patient. We are about to see pharmaceutical companies
delivering drugs, developing test kits (including genetic
tests), and computer programs to deliver the best drug to
the patient. The current data mining (DM) tools are ca-
pable of issuing customized prescriptions, providing most
effective drugs, and dosages with minimal adverse ef-
fects. It has become practical to think of designing,
producing, and delivering drugs intended for an indi-
vidual patient (or a small population of patients). Here,
analogy of the one-of-a-kind production of today versus
mass production is worthy attention. Many industries are
attempting to produce customized products. The pharma-
ceutical industry may soon become a new addition to this
collection of industries.
MAIN THRUST
Paradigm Shift
Predicting implies knowing in advance, which in a busi-
ness environment translates into competitive advantage.
Figure 1. The future of pharmaceutical industry
Pharma
company
Data
Drugs Test kits
Computer
program
Today Tomorrow
Customized Customized
prescription medication
TEAM LinG

A future event can be predicted in two major ways:
population-based and individual-based. The population-
based prediction says, for example, a drug A has been
effective in treating 80% of patients in the population P
as symbolically illustrated in Figure 2. Of course, any
patient would like to belong to the 80% rather than the 20%
category before the drug A is administered. Statistics and
other tools have been widely used in support of the
population-paradigm, among others in medicine and phar-
maceutical industry.
The individual-based approach supported by numer-
ous DM algorithms emphasizes an individual patient
rather than the population (Kusiak et al., 2005). One of
many decision-making scenarios is illustrated in Figure 3,
where the original population P of patients has been
partitioned into two segments 1 and 2. The decisions for
each patient in Segment 1 are made with high confidence;
say 99%, while the decisions for Segment 2 are predicted
with lower confidence.
It is quite possible that Segment 2 patients would
seek an alternative drug or a treatment. There are differ-
ent ways of using DM algorithms. They cover the range
between the population and individual-based paradigms.
The exiting DM algorithms can be grouped into the
following basic ten classes (Kusiak, 2001):
A. Classical statistical methods (e.g., linear, qua-
dratic, and logistic discriminant analyses)
B. Modern statistical techniques (e.g., projection
pursuit classification, density estimation, k-near-
est neighbor, Bayes algorithm)
Figure 2. Population-based paradigm
80% P cured
Drug A Population P Confidence = 95%
20% P not cured
Figure 3. Individual-based paradigm using data
mining tools
70% P cured
Population
Confidence = 99%
Segment 1
15% P not cured
Drug A
10% P cured
Population
Confidence = 85%
Segment 2
5% P not cured
240
Data Mining and Warehousing in Pharma Industry
C. Neural network (Mitchell, 1997)
D. Support vector machines
E. Decision tree algorithms [C4.5 (Quinlan, 1992)]
F. Decision rule algorithms [Rough set algorithms
(Pawlak, 1991)]
G. Association rule algorithms
H. Learning classifier systems
I. Inductive learning algorithms
J. Text learning algorithms
Each class containing numerous algorithms, for ex-
ample, there are more than 100 implementations of the
decision tree algorithm (class E).
Data Warehouse Design
A warehouse has to be designed to meet users require-
ments. DM, online analytical processing (OLAP), and
reporting are the top items on the list of requirements.
Systems design methodologies, and tools can be used to
facilitate the requirements capture. Examples of meth-
odologies for analysis of data warehouse (DW) require-
ments include AND/OR graphs and the house of quality
(Kusiak, 2000).
The architecture of a typical DW embedded in a
pharmaceutical environment is shown in Figure 4. The
pharmaceutical data is extracted from numerous sources
and preprocessed to minimize inconsistencies. Also, data
transformation will capture intricate solution spaces to
improve knowledge discovery. The cleaned and trans-
formed data is loaded and refreshed directly into a DW
or data marts. A data mart might be a precursor to the
full-fledged DW or function as a specialized DW. A
special purpose (exploratory) data mart or a DW might
be created for exploratory data analysis and research.
The warehouse and data marts serve various applications
that justify the development and maintenance cost in
this data storage technology. The range of services that
could be developed off a DW could be expanded beyond
OLAP and DM into almost all pharmaceutical business
areas, including interactions with federal agencies and
other businesses.
Data Flow Analysis
A task that may parallel the capture of requirements for
a DW involves analysis of data flow. A warehouse is to
integrate various streams of data that have to be identi-
fied. The information analysts and users need to feel
comfortable with the data flow methodology selected
for capturing the data flow logic. At minimum this data
flow modeling exercise should increase efficiency of
the data handling and management. An example to a
methodology that can be used to model data flow is the
TEAM LinG
Data Mining and Warehousing in Pharma Industry

Figure 4. Data warehouse architecture

,
Data marts

Data mining

Data
Transformation Datya
Data
Servive
and Loading
Raw Data
OLAP

Data
Exploratory
warehouse
data
Information
warehouse
Data and
Sources Knowledge
Services

Integrated Definition (Kusiak, 2000) process modeling Decision 1 = (Score_9_months - Score_3_months)/

methodology, IDEF3. (Number of months)
Decision 2 = Slope of the scores over time
Preprocessing For Data Warehousing Decision 3 = Average score over time

Before a viable and accurate DW is built, there is a need

to solve specific problems associated with data manage-
ment. The collection process, coding schemes and stan-
dards may lead to errors, which must be fixed. Protocols
for data collection and transfer should concentrate, among
other things, on feature related unwanted characters,
illegal values, out of range feature values, non-uniform
primary keys, and so on.
Clinical trials are costly and there are various methods
to reducing the number of participating subjects. Selecting
the top and bottom quartile patients (score/criterion based)
for both placebo and drug subjects can significantly re-
duce the trial magnitude. Thus special consideration is
required to handle data for smaller number of subjects. The
outcomes are often measured with subjective scores that
vary from subject to subject and also for each subject.
Handling these subjective outcomes is a challenging prob-
lem for developing practical solutions. An example of
subjective outcomes measured over a nine-month period
is shown in Table 1.
Decision 4 = Percentage improvement between
first and last reading
Decision 5 = Score based on group means with first
and last reading over time
The clinical trials performed over an extended period
of time lead to temporal data. There is a need to incorpo-
rate additional parameters in the temporal dataset. These
parameters can be a trend variable, time, or a transformed
parameter. Table 1 provides an example of temporal
decision handling by employing five different decision
measures. Case 2 is a unique example as decision 3 and
5 imply that there is no improvement, contrary to the
actual fact. Thus a new meta-decision measure can be
compiled based on the above five decisions.
Handling noisy data, including genetic data, is a
challenge (Table 2). There are various models that can
compress, identify and reduce noisy genetic data such as
clustering and feature reduction methods (Shah & Kusiak,
2004; Dudoit et al., 2000).

Table 1. Example of subjective and temporal outcomes

Subjective Score Possible Decision Measures

ID 3_months 6_months 9_months Decision 1 Decision 2 Decision 3 Decision 4 Decision 5
1 70.6 78.41 97.3 2.97 4.4496 82.1 112.09 -2.98
2 30.01 62.78 77.29 5.25 7.8805 56.7 171.41 -4.88
3 61.12 86.55 26.77 -3.82 -5.7244 58.15 101.38 3.2
4 74.05 44.44 55.67 -2.04 -3.0637 58.05 81.16 1.54
5 75.28 81.95 41.2 -3.79 -5.6812 66.14 92.89 3.12

241

TEAM LinG
 Data Mining and Warehousing in Pharma Industry

Table 2. Noisy genetic data 2000). The snowflake schema is a variation of star schema
in which the dimensional tables are organized into a
ID Treatment SNP1 SNP2 SNP3 SNP4 SNP5 Decision
hierarchy by normalizing the tables. Another important
1 Drug A/T C/T A/C G/T C/T Improved
issue while designing the DW especially for pharmaceu-
2 Drug A/T T/T A/A G/G T/T Improved
3 Drug A/T C/T A/C G/T T/T Not_Improved
tical industry is the security issues, that is, avoiding
4 Drug A/A C/C A/A T/T C/T Not_Improved unauthorized access.
5 Drug A/T C/T A/C G/T C/T Not_Improved
6 Placebo A/T C/T C/C G/T T/T Improved Data Modeling
7 Placebo A/A C/T A/C G/G C/T Improved
8 Placebo A/T C/C A/C G/T C/T Improved The data stored in DW is relatively error free and com-
9 Placebo A/T C/T A/C G/T C/T Not_Improved pact. This data contains hidden information, which needs
10 Placebo T/T T/T C/C T/T C/C Not_Improved
to be extracted. It forms a staging ground for knowledge
discovery, and decision-making (Elmasri & Navathe,
Data Warehouse Characteristics 2000). The desired functionality of DW access tools is
as follows:
A DW is a subject-oriented, integrated, nonvolatile, time-
variant collection of data in support of users (Elmasri & Tabular reporting and information mapping
Navathe, 2000). The prominent feature of the DW is to Complex queries and sophisticated criteria search
support large volume of data and relatively small number Ranking
of users with relatively long interactions, high perfor- Multivariable and time series analysis
mance levels, multidimensional view, usability, manage- Data visualization, graphing, charting, and pivoting
ability, and flexible reporting. It must efficiently extract Complex textual search
and process data. Advanced statistical analysis
Data cuboids in the DW are multidimensional con- Trend discovery and analysis
structs reflecting the relationships in the underlying Pattern and associations discovery
data. A three-dimensional cuboid is called a data cube
and is illustrated in Figure 5. The data cube represents There are three main streams for data analysis namely,
three dimensions, namely cancer type, medication, and OLAP, statistics, and DM. OLAP supports various view-
time factor. The data cubes can be views through multiple points at different levels of abstraction, which helps in
orientations using a technique called pivoting (Elmasri & data visualization and analysis. Population-based statis-
Navathe, 2000). Thus the data cube in can be pivoted to tical analysis of the data can be performed through various
form a separate medication and time factor table for each tools ranging from simple regression to complex multivari-
cancer type. This data structure allows for roll-up and ate analysis. DM offers tools (decision trees, decision
drill-down capabilities. Roll-up moves up the hierarchy, rules, support vector machines, neural networks, associa-
for example grouping the cancer type dimension to present tion rules, and so on) for discovery of new knowledge by
a single medication and time factor view. Similarly drill- converting hidden information into business models. It
down helps in finer grained view, for example, the medi- provides tools for identifying valid, novel, potentially
cations can be broken down into cardiac, respiratory, and useful, and ultimately understandable patterns from data
skin medications. and constructs high confidence predictions for individu-
The DW is normally designed with star schema or the als (Fayyad et al., 1997). This may represent valuable
snowflake schema. The star schema is designed with a knowledge that might lead to medical discoveries, for
fact table (tuples arranged in one per recorded fact) and example, certain ranges of parameter values leading to
single table for each dimension (Elmasri & Navathe, longer survival time.
Thus OLAP with DM complements each other in the
analysis and provides different but much needed func-
Figure 5. Data cube
tionality for data understanding, visualization, and mak-
Time ing individualized decisions. Data modeling provides ana-
Cancer lytical reasoning for decision-making to solve specific
type patient and business specific issues.

Applications
Medication
DM, OLAP, and decision-making algorithms will lead to
outcome predictions, identification of significant patterns

242

TEAM LinG
Data Mining and Warehousing in Pharma Industry
Figure 6. Pharmaceutical and medical applications
Application Area
Business
Prediction information
Identification
Classification Drug discovery
Optimization
Outcome
predictions
and parameters, patient-specific decisions, and ultimately
process/goal optimization. Prediction, identification, clas-
sification, and optimization are key to drug discovery,
adverse effect analysis, outcome predictions, and indi-
vidualized protocols (Figure 6).
The issues highlighted in this article are illustrated
with various medical informatics research projects. DM
has led to identifying predictive parameters, formulating
individualized treatment protocols, categorizing model
patients, and developing a decision-making model for
predictions of survival for dialysis patients (Kusiak et al.,
2005). A gene/SNP selection approach (Shah & Kusiak,
2004) was developed using weighted decision trees and
genetic algorithm. The high quality significant gene/SNP
subset can be targeted for drug development and valida-
tion. An intelligent DM system determined the wellness
score for the infants with hypoplastic left heart syndrome
based on 73 physiologic, laboratory, and nurse-assessed
parameters (Kusiak et al., 2003). For cancer-related analy-
sis, a patient acceptance model can be developed to
predict the treatment type, drug toxicity, and the length of
disease free status after the treatment. These concept
discussed in this chapter can be applied across all medical
topics including phenotypic and genotypic data.
Other examples are Merck gene sequencing (Eckman
et al., 1998), epidemiological and clinical toxicology
(Helma et al., 2000), prediction of rodent carcinogenic-
ity bioassays (Bahler et al., 2000), gene expression
level analysis (Dudoit et al., 2000), predicting risk of
coronary artery disease (Tham et al., 2003), and VA
health care system (Smith & Joseph, 2003).
FUTURE TRENDS
The future of data mining, warehousing, and modeling in
pharmaceutical industry offers numerous challenges.
The first one is the scale of the data, measured with the
number of features. New scalable schemes for parameter
selection and knowledge discovery will be developed.
There is a need to develop new tools for rapid evaluation
,
Parameter
selections &
Individualized
identification
protocols
Adverse effects Treatments
Predictions
Customized
protocols
of parameter relevancy. Dynamic clinical studies would
provide additional facet to interact and intervene, result-
ing in clean, error free, and necessary data collection and
analysis.
CONCLUSION
Data warehousing, data modeling, data mining, OLAP,
and decision-making algorithms will ultimately lead to
targeted drug discovery and individualized treatments
with minimum adverse effects. The success of the phar-
maceutical industry will largely depend on following
the course outlined by the new data paradigm.
REFERENCES
Bahler, D., Stone, B., Wellington, C., & Bristol, D.W.
(2000). Symbolic, neural, and Bayesian machine learn-
ing models for predicting carcinogenicity of chemical
compounds. Journal of chemical information and com-
puter sciences, 40(4), 906-914.
Berndt, D.J., Fisher, J.W., Hevner, R.A., & Studnicki, J.
(2001). Healthcare data warehousing and quality assur-
ance. IEEE Computer, 34(12), 56-65.
Dudoit, S., Fridlyand, J., & Speed, T.P. (2000). Com-
parison of discrimination methods for the classifica-
tion of tumors using gene expression data. Technical
Report 576. Berkeley, CA: Department of Statistics,
University of California.
Eckman, B.A., Aaronson, J.S., Borkowski, J.A., Bailey,
W.J., Elliston, K.O., Williamson, A. R., & Blevins, R.A.
(1998). The Merck gene index browser: An extensible
data integration system for gene finding, gene character-
ization and EST data mining. Bioinformatics, 14, 2-13.
Elmasri, R., & Navathe, S. B. (2000). Fundamentals of
database systems. New York: Addison-Wesley.
243
TEAM LinG

Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P., &
Uthurusamy, R. (1997). Advances in knowledge discov-
ery and data mining. Cambridge, MA: MIT Press.
Helma, C., Gottmann, E., & Kramer, S. (2000). Knowledge
discovery and data mining in toxicology. Statistical Meth-
ods in Medical Research, 9(4), 329-358.
King, S.Y., Kung, M.S., & Fung, H.L. (1984). Statistical
prediction of drug stability based on nonlinear parameter
estimation. Journal of pharmaceutical sciences, 73(5),
657-662.
Kusiak, A. (2000). Computational intelligence in design
and manufacturing. New York: John Wiley.
Kusiak, A. (2001). Feature transformation methods in
data mining. IEEE Transactions on Electronics Pack-
aging Manufacturing, 24(3), 214-221.
Kusiak, A., Caldarone, C.A., Kelleher, M.D., Lamb, F.S.,
Persoon, T.J., Gan, Y., & Burns, A. (2003, April). Mining
temporal data sets: Hypoplastic left heart syndrome case
study. In Proceedings of the SPIE Conference on Data
Mining and Knowledge Discovery: Theory, Tools, and
Technology V 5098, SPIE (pp. 93-101). Belingham, WA.
Kusiak, A., Dixon, B., & Shah, S. (2005). Predicting sur-
vival time for kidney dialysis patients: A data mining
approach. Computers in Biology and Medicine, 35(4),
311-327.
Kusiak, A., Kern, J.A., Kernstine, K.H., & Tseng, T.L.
(2000). Autonomous decision-making: a data mining ap-
proach. IEEE Transactions on Information Technology
in Biomedicine, 4(4), 274-284.
Mitchell, T.M. (1997). Machine learning. New York:
McGraw Hill.
Pawlak, Z. (1991). Rough sets: Theoretical aspects of
reasoning about data. Boston, MA: Kluwer.
Quinlan, R. (1992). C 4.5 programs for machine learn-
ing. San Meteo, CA: Morgan Kaufmann.
Shah, S.C., & Kusiak, A. (2004). Data mining and genetic
algorithm based Gene/SNP selection. Artificial Intelli-
gence in Medicine, 31(3), 183-196.
Smith, M.W., & Joseph, G.J. (2003). Pharmacy data in
the VA health care system. Medical Care Research and
Review: MCRR, 60 (3 Suppl), 92S-123S.
Tham, C.K., Heng, C.K., & Chin, W.C. (2003). Predicting
risk of coronary artery disease from DNA microarray-
based genotyping using neural networks and other statis-
tical analysis tool. Journal of Bioinformatics and Compu-
tational Biology, 1(3), 521-539.
244
Data Mining and Warehousing in Pharma Industry
Tye, H. (2004). Application of statistical design of experi-
ments methods in drug discovery. Drug Discovery To-
day, 9(11), 485-491.
KEY TERMS
Adverse Effects: Any untoward medical occurrences
that may be life threatening and requires in-patient
hospitalization.
Clustering: Clustering algorithms discover simi-
larities and differences among groups of items. It di-
vides a dataset so that patients with similar content are
in the same group, and groups are as different as pos-
sible from each other.
Customized Protocols: A specific set of treatment
parameters and their values those are unique for indi-
vidual. The customized protocols are derived from dis-
covered knowledge patterns.
Data Visualization: The method or end result of
transforming numeric and textual information into a
graphic format. Visualizations are used to explore large
quantities of data holistically in order to understand
trends or principles.
Decision Trees: Decision-tree algorithm creates
rules based on decision trees or sets of if-then state-
ments to maximize interpretability.
Drug Discovery: It is a research process that identi-
fies molecules with desired biological effects so a to
develop new therapeutic drugs.
Feature Reduction Methods: The goal of feature
reduction method is to identify the minimum set of non-
redundant features (e.g., SNPs, genes) that are useful in
classification.
Knowledge Discovery: Knowledge discovery in
databases is the process of identifying valid, novel,
potentially useful, and ultimately understandable pat-
terns/models in data.
Neural Networks: Neural networks are a set of
simple units (neurons) that receive a number of real values
inputs, which are processed through the network to
produce a real value output.
OLAP: Online analytical processing is a category of
software tools that provides analysis of data stored in a
database.
TEAM LinG

Data Mining for Damage Detection in
Engineering Structures
RamdevKanapady
University of Minnesota, USA
Aleksandar Lazarevic
University of Minnesota, USA
INTRODUCTION
The process of implementing and maintaining a structural
health monitoring system consists of operational evalu-
ation, data processing, damage detection, and life predic-
tion of structures. This process involves the observation
of a structure over a period of time using continuous or
periodic monitoring of spaced measurements, the extrac-
tion of features from these measurements, and the analy-
sis of these features to determine the current state of
health of the system. Such health monitoring systems are
common for bridge structures, and many examples are
citied in (Maalej et al., 2002).
The phenomenon of damage in structures includes
localized softening or cracks in a certain neighborhood of
a structural component due to high operational loads, or
the presence of flaws due to manufacturing defects.
Methods that detect damage in the structure are useful for
non-destructive evaluations that typically are employed
in agile manufacturing and rapid prototyping systems. In
addition, there are a number of structures, such as turbine
blades, suspension bridges, skyscrapers, aircraft struc-
tures, and various structures deployed in space, for which
structural integrity is of paramount concern (Figure 1).
With the increasing demand for safety and reliability of
Figure 1. Damage detection is part of structural health
monitoring system
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
245
D
aerospace, mechanical and civilian structures damage
detection techniques become critical to reliable predic-
tion of damage in these structural systems.
The most currently used damage detection methods
are manual, such as tap test, visual, or specially localized
measurement techniques (Doherty, 1997). These tech-
niques require that the location of the damage be on the
surface of the structure. In addition, location of the
damage has to be known a priori, and these locations
have to be readily accessible. This makes current mainte-
nance procedure of large structural systems very time
consuming and expensive due to its heavy reliance on
human labor.
BACKGROUND
The damage in structures and structural systems is de-
fined through comparison of two different states of the
system; the first one is the initial undamaged state, and
the second one is the damaged state. Due to the changes
in the properties of the structure or quantities derived
from these properties, the process of damage detection
eventually reduces to a form of pattern recognition/data
mining problem. In addition, emerging continuous moni-
toring of an instrumented structural system often results
in the accumulation of a large amount of data that need to
be processed, analyzed, and interpreted for feature extrac-
tion in an efficient damage detection system. However,
the rate of accumulating such data sets far outstrips the
ability to analyze them manually. More specifically, there
is often information hidden in the data that cannot be
discovered manually. As a result, there is a need to
develop an intelligent data processing component that
can significantly improve the current damage detection
systems. An immediate alternative is to design data min-
ing techniques that can enable real-time prediction and
identification of damage for newly available test data,
once a sufficiently accurate model is developed.
In recent years, various data mining techniques, such
as artificial neural networks (ANNs) (Anderson, Lemoine
TEAM LinG
 Data Mining for Damage Detection in Engineering Structures
& Ambur, 2003; Lazarevic et al., 2004; Ni, Wang & Ko,
2002; Sandhu et al., 2001; Yun & Bahng, 2000; Zhao, Ivan
& DeWolf, 1998), support vector machines (SVMs) (Mita
& Hagiwara 2003), decision trees (Sandhu et al., 2001),
have been applied successfully to structural damage
detection problems, thus showing that they can be poten-
tially useful for such class of problems. This success can
be attributed to numerous disciplines integrated with data
mining, such as pattern recognition, machine learning,
and statistics. In addition, it is well known that data mining
techniques effectively can handle noisy, partially incom-
plete, and faulty data, which is particularly useful, since
in damage detection applications, measured data are ex-
pected to be incomplete, noisy, and corrupted.
The intent of this paper is to provide a survey of
emerging data mining techniques for damage detection
structures. Although the field of damage detection is very
broad and consists of vast literature that is not based on
data mining techniques, this survey will be focused pre-
dominantly on data mining techniques for damage detec-
tion based on changes in properties of the structure.
However, a large amount of literature applicable to fault
detection and diagnosis to application-specific system,
such as rotating machinery, is not within the scope of this paper.
CATEGORIZATION OF STRUCTURAL
DAMAGE
The damage in structures can be classified as linear or
nonlinear. Damage is considered as linear if the undam-
aged linear-elastic structure remains linear-elastic after
damage. However, if the initially linear-elastic structure
behaves in a nonlinear manner after the damage initiation,
then the damage is considered as nonlinear. However, it
is possible that the damage is linear at the damage initia-
tion phase, but after prolonged growth in time, it may
become nonlinear. For example, loose connections be-
tween the structures at the joints or the joints that rattle
(Sekhar, 2003) are considered non-linear damages. Ex-
amples of such non-linear damage detection systems are
described in Adams and Farrar (2002) and Kerschen and
Golinval (2004).
Most of the modal data in the literature are proposed
for the linear case. They are based on the following three
levels of damage identification: (1) Recognition-quali-
tative indication that damage might be present in the
structure; (2) Localizationinformation about the prob-
able location of the damage in the structure; and (3)
Assessmentestimate of the extent of severity of the
damage in the structure. Such linear damage detection
techniques can be found in Yun and Bahng (2000), Ni, et
al. (2002), and Lazarevic, et al. (2004).
246
CLASSIFICATION OF DAMAGE
DETECTION TECHNIQUES
We provide several different criteria for classification of
damage detection techniques based on data mining.
In the first classification, damage detection techniques
can be categorized into continuous (Keller & Ray, 2003)
and periodic (Patsias & Staszewski, 2002) damage detec-
tion systems. Continuous techniques usually employ an
integrated approach that consists of data acquisition
process, feature extraction from large amounts of data
collected from real-time sensors, and damage detection
process. In periodic techniques, feature extraction pro-
cess is optional, since the amount of data that need to be
processed is not large and does not necessarily require
data mining techniques for feature extraction.
In the second classification, we distinguish between
application-based and application-independent tech-
niques. Application-based techniques are generally ap-
plicable to a specific structural system, and they typically
assume that the monitored structure responds in some
predetermined manner that can be accurately modeled by
(i) numerical techniques such as finite element (Sandhu et
al., 2001) or boundary element analysis (Anderson, Lemoine
& Ambur, 2003) and/or (ii) behavior of the response of the
structures that are based on physics-based models (Keller
& Ray, 2003). Most of damage detection techniques that
exist in literature belong to the application-based ap-
proach, where the minimization of the residue between the
experimental and the analytical model is built into the
system. Often, this type of data is not available and can
render application-based methods impractical for certain
applications, particularly for structures that are designed
and commissioned without such models. On the other
hand, application-independent techniques do not de-
pend on specific structure, and they are generally appli-
cable to any structural system. However, the literature on
these techniques is very sparse, and the research in this
area is at a very nascent stage (Bernal & Gunes, 2000;
Zang, Friswell & Imregun 2004).
In the third classification, damage detection tech-
niques are split into signature-based and non-signature-
based methods. Signature-based techniques extensively
use signatures of known damages in the given structure
that are provided by human experts. These techniques
commonly fall into the category of recognition of damage
detection, which only provides the qualitative indication
that damage might be present in the structure (Friswell,
Penny & Wilson, 1994) and, to a certain extent, the
localization of the damage (Friswell, Penny & Garvey,
1997). Non-signature methods are not based on signa-
tures of known damages, and they not only recognize but
also localize and assess the extent of damage. Most of the
damage detection techniques in the literature fall into this
TEAM LinG
Data Mining for Damage Detection in Engineering Structures
category (Lazarevic et al., 2004; Ni et al., 2002; Yun &
Bahng, 2000).
In the fourth classification, damage detection tech-
niques are classified into local (Sekhar, 2003; Wang, 2003)
and global (Fritzen & Bohle, 2001) techniques. Typically,
the damage is initiated in a small region of the structure,
and, hence, it can be considered as local phenomenon. One
could employ local or global damage detection features
that are derived from local or global response or properties
of the structure. Although local features can detect the
damage effectively, these features are very difficult to
obtain from the experimental data, such as higher natural
frequencies and mode shapes of structure (Ni et al., 2002).
In addition, since the vicinity of damage is not known a
priori, the global methods that can employ only global
damage detection features, such as lower natural frequen-
cies of the structure (Lazarevic et al., 2004), are preferred.
Finally, damage detection techniques can be classified
as traditional and emerging data mining techniques. Tra-
ditional analytical techniques employ mathematical mod-
els to approximate the relationships between specific dam-
age conditions and changes in the structural response or
dynamic properties. Such relationships can be computed
by solving a class of so-called inverse problems. The major
drawbacks of the existing approaches are as follows: (i) the
more sophisticated methods involve computationally cum-
bersome system solvers, which are typically solved by
singular value decomposition techniques, non-negative
least-squares techniques, bounded variable least-squares
techniques, and so forth; and, (ii) all computationally
intensive procedures need to be repeated for any newly
available measured test data for a given structure. A brief
survey of these methods can be found in Doebling, et al.
(1996). On the other hand, data mining techniques consist
of application techniques to model an explicit inverse
relation between damage detection features and damage
by minimization of the residue between the experimental
and the analytical model at the training level. For example,
the damage detection features could be natural frequen-
cies, mode shapes, mode curvatures, and so forth. It
should be noted that data mining techniques also are
applied to detect features in large amounts of measurement
data. In the next few sections, we will provide a short
description of several types of data mining algorithms
used for damage detection.
Classification
Data mining techniques based on classification were suc-
cessfully applied to identify the damage in the structures.
For example, decision trees have been applied to detect
damage in an electrical transmission tower (Sandhu et al.,
2001). It has been found that in this approach, decision
trees can be easily understood, while many interesting
rules about the structural damage were found. In addi-
tion, a method using support vector machines (SVMs) ,
has been proposed to detect local damages in a building
structure (Mita & Hagiwara, 2003). The method is verified
to have the capability to identify not only the location of
damage, but also the magnitude of damage with satisfac-
tory accuracy, employing modal frequency patterns as
damage features.
Pattern Recognition
Pattern recognition techniques also are applied to dam-
age detection by various researchers. For example, the
statistical pattern recognition is applied to damage de-
tection employing relatively few measurements of modal
data collected from three scale model-reinforced con-
crete bridges (Haritos & Owen, 2004), but the method
only was able to indicate that damage had occurred. In
addition, independent component analysis, a multivari-
ate statistical method also known as proper orthogonal
decomposition, has been applied to damage detection
problems on time history data to capture essential pat-
tern of the measured vibration data (Zang, Friswell &
Imregun, 2004).
Neural Networks
Prediction-based techniques such as neural networks
(Lazarevic et al., 2004; Ni et al., 2002, Zhao et al., 1998) has
been successfully applied to detect the existence, loca-
tion, and quantification of the damage in the structure
employing modal data. Neural networks have been ex-
tremely popular in recent years due to their capabilities
as universal approximators.
In damage detection approaches based on neural
networks, the damage location and severity are simulta-
neously identified using a one-stage scheme, also called
the direct method (Zhao et al., 1998), where the neural
network is trained with different damage levels at each
possible damage location. However, these studies were
restricted to very small models with a small number of
target variables (order of 10), and the development of a
predictive model that could identify correctly the loca-
tion and severity of damage in practical large-scale com-
plex structures using this direct approach was a consid-
erable challenge. Increased geometric complexity of the
structure caused an increase in the number of target
variables, thus resulting in data sets with a large number
of target variables. Since the number of prediction mod-
els that needs to be built for each continuous target
variable increases, the number of training data records
required for effective training of neural networks also
increases, thus requiring more computational time for
training neural networks, but also more time for data
247
TEAM LinG
 Data Mining for Damage Detection in Engineering Structures
generation, since each damage state (data record) re-
quires an eigen solver to generate natural frequency and
mode shapes of the structure. The earlier direct approach,
employed by numerous researchers, required the predic-
tion of the material property; namely, the Youngs modu-
lus of elasticity considering all the elements in the domain
individually or simultaneously. However, this approach
does not scale to situations in which thousands of ele-
ments are present in the complex geometry of the structure
or when multiple elements in the structure have been
damaged simultaneously.
To reduce the size of the system under consideration,
several substructure-based approaches have been pro-
posed (Sandhu et al., 2002; Yun & Bahng, 2000). These
approaches partition the structure into logical substruc-
tures and then predict the existence of the damage in each
of them. However, pinpointing the location of damage and
extent of the damage is not resolved completely in these
approaches. Recently, these issues have been addressed
in two hierarchical approaches (Lazarevic et al., 2004; Ni
et al., 2002). In the former, neural networks are hierarchi-
cally trained using one-level damage samples to first
locate the position of the damage, and then the network
is retrained by an incremental weight update method
using additional samples corresponding to different dam-
age degrees, but only at the identified location at the first
stage. The input attributes of neural networks are de-
signed to depend only on damage location, and they
consisted of several natural frequencies and a few incom-
plete modal vectors. Since measuring mode shapes are
difficult, global methods based only on natural frequen-
cies are highly preferred. However, employing natural
frequencies as features traditionally has many drawbacks
(e.g., two symmetric damage locations cannot be distin-
guished using only natural frequencies). To overcome
these drawbacks, Lazarevic, et al. (2004) proposed hierar-
chical and localized clustering approaches based only on
natural frequencies as features where symmetrical dam-
age locations of damage as well as spatial characteristics
of structural systems are integrated in building the model.
Other Techniques
Other data mining based approaches also have been
applied to different problems in structural health monitor-
ing. For example, outlier-based analysis techniques
(Worden, Manson & Fieller, 2000) have been used to
detect the existence of damage; wavelet-based approaches
(Wang, 2003) have been used to detect damage features;
and a combination of independent component analysis
and artificial neural networks (Zang, Friswell & Imregun,
2004) have been applied successfully to detect damages
in structures.
248
FUTURE TRENDS
Damage detection is increasingly becoming an indispens-
able and integral component of any comprehensive struc-
tural health monitoring program for mechanical and large-
scale civilian, aerospace, and space structures. Although
a variety of techniques have been developed for detecting
damages, there is still a number of research issues con-
cerning the prediction performance and efficiency of the
techniques that needs to be addressed (Auwerarer &
Peeters, 2003; De Boe & Golinval, 2001).
CONCLUSION
In this paper, a survey of emerging data mining tech-
niques for damage detection in structures is provided.
This survey reveals that the existing data mining tech-
niques are based predominantly on changes in properties
of the structure to classify, localize, and predict the extent
of damage.
REFERENCES
Adams, D., & Farrar, C. (2002). Classifying linear and
nonlinear structural damage using frequency domain ARX
models. Structural Health Monitoring, 1(2), 185-201.
Anderson, T., Lemoine, G., & Ambur, D. (2003). An arti-
ficial neural network based damage detection scheme for
electrically conductive composite structures. Proceed-
ings of the 44th AIAA/ASME/ASCE/AHS/ASC Structures,
Structural Dynamics, and Materials Conference, Nor-
folk, Virginia.
Auwerarer, H., & Peeters, B. (2003). International research
projects on structural health monitoring: An overview.
Structural Health Monitoring, 2(4), 341-358.
Bernal, D., & Gunes, B. (2000). Extraction of system
matrices from state-space realizations. Proceedings of the
14 th Engineering Mechanics Conference, Austin, Texas.
De Boe, P., & Golinval, J.-C. (2001). Damage localization
using principal component analysis of distributed sensor
array. In F.K. Chang (Ed.), Structural health monitoring:
The demands and challenges (pp. 860-861). Boca Raton,
FL: CRC Press.
Doebling, S., Farrar, C., Prime, M., & Shevitz, D. (1996).
Damage identification and health monitoring of struc-
tural systems from changes in their vibration character-
istics: A literature review [Report LA-12767-MS]. Los
Alamos National Laboratory, Los Alamos, NM.
TEAM LinG
Data Mining for Damage Detection in Engineering Structures
Doherty, J. (1987). Nondestructive evaluation. In A.S.
Kobayashi (Ed.), Handbook on experimental mechanics
(ch. 12). Society of Experimental Mechanics, Inc, England
Cliffs, NJ.
Friswell, M., Penny J., & Garvey, S. (1997). Parameter
subset selection in damage location. Inverse Problems in
Engineering, 5(3), 189-215.
Friswell, M., Penny J., & Wilson, D. (1994). Using vibra-
tion data and statistical measures to locate damage in
structures. Modal Analysis: The International Journal of
Analytical and Experimental Modal Analysis, 9(4), 239-254.
Fritzen, C., & Bohle, K. (2001). Vibration based global
damage identificationA tool for rapid evaluation of
structural safety. In F.D. Chang (Ed.), Structural health
monitoring: The demands and challenges (pp. 849-859).
Boca Raton, FL: CRC Press.
Haritos, N., & Owen, J.S. (2004). The use of vibration data
for damage detection in bridges: A comparison of system
identification and pattern recognition approaches. Struc-
tural Health Monitoring, 3(2), 141-163.
Keller, E., & Ray, A. (2003). Real-time health monitoring of
mechanical structures. Structural Health Monitoring,
2(3), 191-203.
Kerschen, G., & Golinval, J-C. (2004). Feature extraction
using auto-associative neural networks. Smart Materials
and Structures, 13, 211-219.
Khoo, L., Mantena, P., & Jadhav, P. (2004). Structural
damage assessment using vibration modal analysis. Struc-
tural Health Monitoring, 3(2), 177-194.
Lazarevic, A., Kanapady, R., Tamma, K.K., Kamath, C., &
Kumar, V. (2003a). Localized prediction of continuous
target variables using hierarchical clustering. Proceed-
ings of the Third IEEE International Conference on Data
Mining, Florida.
Lazarevic, A., Kanapady, R., Tamma, K.K., Kamath, C., &
Kumar, V. (2003b). Damage prediction in structural me-
chanics using hierarchical localized clustering-based
approach. Proceedings of Data Mining and Knowledge
Discovery: Theory, Tools, and Technology V, Orlando,
Florida.
Lazarevic, A., Kanapady, R., Tamma, K.K., Kamath, C., &
Kumar, V. (2004). Effective localized regression for dam-
age detection in large complex mechanical structures.
Proceedings of the 9th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining,
Seattle, Washington.
Maalej, M., Karasaridis, A., Pantazopoulou, S., &
Hatzinakos, D. (2002). Structural health monitoring of ,
smart structures. Smart Materials and Structures, 11,
581-589.
Mita, A., & Hagiwara, H. (2003). Damage diagnosis of a
building structure using support vector machine and
modal frequency patterns. Proceedings of SPIE, 5057,
San Diego, CA.
Ni, Y., Wang, B., & Ko, J. (2002). Constructing input
vectors to neural networks for structural damage identi-
fication. Smart Materials and Structures, 11, 825-833.
Patsias, S., & Staszewski, W.J. (2002). Damage detection
using optical measurements and wavelets. Structural
Heath Monitoring, 1(1), 5-22.
Sandhu, S., Kanapady, R., Tamma, K.K., Kamath, C., &
Kumar, V. (2001). Damage prediction and estimation in
structural mechanics based on data mining. Proceedings
of the 7th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining/Fourth Work-
shop on Mining Scientific Datasets, San Francisco, Cali-
fornia.
Sandhu, S., Kanapady, R., Tamma, K.K., Kamath, C., &
Kumar, V. (2002). A sub-structuring approach via data
mining for damage prediction and estimation in complex
structures. Proceedings of the SIAM International Con-
ference on Data Mining, Arlington, Virginia.
Sekhar, S. (2003). Identification of a crack in rotor system
using a model-based wavelet approach. Structural Health
Monitoring, 2(4), 293-308.
Wang, W. (2003). An evaluation of some emerging tech-
niques for gear fault detection. Structural Health Moni-
toring, 2(3), 225-242.
Worden, K., Manson, G., & Fieller, N. (2000). Damage
detection using outlier analysis. Journal of Sound and
Vibration, 229(3), 647-667.
Yun, C., & Bahng, E.Y. (2000). Sub-structural identifica-
tion using neural networks. Computers & Structures, 77, 41-52.
Zang, C., Friswell, M.I., & Imregun, M. (2004). Structural
damage detection using independent component analy-
sis. Structural Health Monitoring, 3(1), 69-83.
Zhao, J., Ivan, J., & De Wolf, J. (1998). Structural damage
detection using artificial neural networks. Journal of
Infrastructure Systems, 4(3), 93-101.
249
TEAM LinG
 Data Mining for Damage Detection in Engineering Structures
KEY TERMS
Boundary Element Method: Numerical method to solve
the differential equations with boundary/initial condi-
tions over surface of a domain.
Finite Element Method: Numerical method to solve
the differential equations with boundary/initial condi-
tions over a domain.
Modal Properties: Natural frequency, mode shapes,
and mode curvatures constitutes modal properties.
250
Mode Shapes: Eigen vectors associated with the natu-
ral frequencies of the structure.
Natural Frequency: Eigen values of the mass and
stiffness matrix system of the structure.
Smart Structure: A structure with a structurally inte-
grated fiber optic sensing system.
Structures and Structural System: The word struc-
ture used has been employed loosely in the paper. The
structure is referred to as the continuum material, whereas
the structural system consists of structures that are
connected at joints.
TEAM LinG
 251

Data Mining for Intrusion Detection ,

Aleksandar Lazarevic
University of Minnesota, USA

INTRODUCTION nerabilities and they are usually not sufficient to ensure

Today computers control power, oil and gas delivery,
communication systems, transportation networks, bank-
ing and financial services, and various other infrastruc-
ture services critical to the functioning of our society.
However, as the cost of the information processing and
Internet accessibility falls, more and more organiza-
tions are becoming vulnerable to a wide variety of cyber
threats. According to a recent survey by CERT/CC (Com-
puter Emergency Response Team/Coordination Cen-
ter), the rate of cyber attacks has been more than dou-
bling every year in recent times (Figure 1). In addition,
the severity and sophistication of the attacks are also
growing. For example, Slammer/Sapphire Worm was
the fastest computer worm in history. As it began spread-
ing throughout the Internet, it doubled in size every 8.5
seconds and infected at least 75,000 hosts causing
network outages and unforeseen consequences such as
canceled airline flights, interference with elections,
and ATM failures (Moore, 2003).
It has become increasingly important to make our
information systems, especially those used for critical
functions in the military and commercial sectors, resis-
tant to and tolerant of such attacks. The conventional
approach for securing computer systems is to design
security mechanisms, such as firewalls, authentication
mechanisms, and Virtual Private Networks (VPN) that
create a protective shield around them. However, such
security mechanisms almost always have inevitable vul-
Figure 1. Growth rate of cyber incidents reported to
Computer Emergency Response Team/Coordination
Center (CERT/CC)
120000
100000
Number of cyber incidents
complete security of the infrastructure and to ward off
attacks that are continually being adapted to exploit the
systems weaknesses often caused by careless design
and implementation flaws. This has created the need for
security technology that can monitor systems and iden-
tify computer attacks. This component is called intru-
sion detection and is a complementary to conventional
security mechanisms. This article provides an overview
of current status of research in intrusion detection
based on data mining.
BACKGROUND
Intrusion detection includes identifying a set of mali-
cious actions that compromise the integrity, confiden-
tiality, and availability of information resources. An
Intrusion Detection System (IDS) can be defined as a
combination of software and/or hardware components
that monitors computer systems and raises an alarm
when an intrusion happens.
Traditional intrusion detection systems are based on
extensive knowledge of signatures of known attacks.
However, the signature database has to be manually
revised for each new type of intrusion that is discovered.
In addition, signature-based methods cannot detect
emerging cyber threats, since by their very nature these
threats are launched using previously unknown attacks.
Finally, very often there is substantial latency in deploy-
ment of newly created signatures. All these limitations
have led to an increasing interest in intrusion detection
techniques based upon data mining.
The tremendous increase of novel cyber attacks has
made data mining based intrusion detection techniques
extremely useful in their detection. Data mining tech-
niques for intrusion detection generally fall into one of
three categories; misuse detection, anomaly detection

80000 and summarization of monitored data.

60000

40000 MAIN THRUST

20000
Before applying data mining techniques to the problem
0
1 1991
1990 2 1992
3 1993
4 1994
5 6
1995 7
1996 8
1997 9
1998 10 2000
1999 11 2001
12 2002
13 2003
14
of intrusion detection, the data has to be collected.
Year Different types of data can be collected about informa-

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG

tion systems (e.g., tcpdump and netflow data for net-
work intrusion detection, syslogs or system calls for
host intrusion detection). However, such collected data
is often available in a raw format and needs to be
processed in order to be used in data mining techniques.
For example, in MADAM ID project (Lee, 2000, 2001)
at Columbia University, association rules and frequent
episodes were extracted from network connection
records to construct three groups of features: (i) con-
tent-based features that describe intrinsic characteris-
tics of a network connection (e.g., number of packets,
acknowledgments, data bytes from source to destina-
tion), (ii) time-based traffic features that compute the
number of connections in some recent time interval
(e.g., last few seconds) and (iii) connection based fea-
tures that compute the number of connections from a
specific source to a specific destination in the last N
connections (e.g., N = 1000).
When the feature construction step is complete, ob-
tained features may be used in any data mining technique.
Misuse Detection
In misuse detection based on data mining, each instance
in a data set is labeled as normal or attack/intrusion Anomaly Detection
and a learning algorithm is trained over the labeled data.
These techniques are able to automatically retrain intru-
sion detection models on different input data that in-
clude new types of attacks, as long as they have been
labeled appropriately. Unlike signature-based intrusion
detection systems, data mining based misuse detection
models are created automatically, and can be more
sophisticated and precise than manually created signa-
tures. In spite of the fact that misuse detection models
have high degree of accuracy in detecting known attacks
and their variations, their obvious drawback is the in-
ability to detect attacks whose instances have not yet
been observed. In addition, labeling data instances as
normal or intrusive may require enormous time for
many human experts.
Since standard data mining techniques are not di-
rectly applicable to the problem of intrusion detection
due to dealing with skewed class distribution (attacks/
intrusions correspond to a class of interest that is much
smaller, i.e., rarer, than the class representing normal
behavior) and learning from data streams (attacks/intru-
sions very often represent sequence of events), a num-
ber of researchers have developed specially designed
data mining algorithms that are suitable for intrusion
detection. Research in misuse detection has focused
mainly on classification of network intrusions using
various standard data mining algorithms (Barbara, 2001;
Ghosh, 1999; Lee, 2001; Sinclair, 1999), rare class
predictive models (Joshi, 2001) and association rules
(Barbara, 2001; Lee, 2000; Manganaris, 2000).
252
Data Mining for Intrusion Detection
MADAM ID (Lee, 2000, 2001) was one of the first
projects that applied data mining techniques to the intru-
sion detection problem. In addition to the standard
features that were available directly from the network
traffic (e.g., duration, start time, service), three groups
of constructed features were also used by the RIPPER
algorithm to learn intrusion detection rules from DARPA
1998 data set (Lippmann, 1999). Other classification
algorithms that are applied to the intrusion detection
problem include standard decision trees (Bloedorn,
2001; Sinclair, 1999), modified nearest neighbor algo-
rithms (Ye, 2001b), fuzzy association rules (Bridges,
2000), neural networks (Dao, 2002; Lippman, 2000a),
nave Bayes classifiers (Schultz, 2001), genetic algo-
rithms (Bridges, 2000), genetic programming
(Mukkamala, 2003a), and etcetera. Most of these ap-
proaches attempt to directly apply specified standard
techniques to publicly available intrusion detection data
sets (Lippmann, 1999, 2000b), assuming that the labels
for normal and intrusive behavior are already known.
Since this is not realistic assumption, misuse detection
based on data mining has not been very successful in
practice.
Anomaly detection creates profiles of normal legiti-
mate computer activity (e.g., normal behavior of users,
hosts, or network connections) using different tech-
niques and then uses a variety of measures to detect
deviations from defined normal behavior as potential
anomaly. Anomaly detection models often learn from a
set of normal (attack-free) data, but this also requires
cleaning data from attacks and labeling only normal data
records. Nevertheless, other anomaly detection tech-
niques detect anomalous behavior without using any
knowledge about the training data. Such models typi-
cally assume that the data records that do not belong to
the majority behavior correspond to anomalies.
The major benefit of anomaly detection algorithms
is their ability to potentially recognize unforeseen and
emerging cyber attacks. However, their major limita-
tion is potentially high false alarm rate, since deviations
detected by anomaly detection algorithms may not nec-
essarily represent actual attacks, but new or unusual, but
still legitimate, network behavior.
Anomaly detection algorithms can be classified into
several groups: (i) statistical methods; (ii) rule-based
methods; (iii) distance-based methods; (iv) profiling
methods; and (v) model-based approaches (Lazarevic,
2004). Although anomaly detection algorithms are quite
diverse in nature, and thus may fit into more than one
proposed category, most of them employ certain data
mining or artificial intelligence techniques.
TEAM LinG
Data Mining for Intrusion Detection
Statistical methods: Statistical methods monitor
the user or system behavior by measuring certain
variables over time (e.g., login and logout time of
each session). The basic models keep averages of
these variables and detect whether thresholds are
exceeded based on the standard deviation of the
variable. More advanced statistical models com-
pute profiles of long-term and short-term user
activities by employing different techniques, such
as Kolmogorov-Smirnov test (Cabrera, 2000), Chi-
square (c 2) statistics (Ye, 2001a), probabilistic
modeling (Yamanishi, 2000), and likelihood of
data distributions (Eskin, 2000).
Distance-based methods: Most statistical ap-
proaches have limitation when detecting outliers in
higher dimensional spaces, since it becomes in-
creasingly difficult and inaccurate to estimate the
multidimensional distributions of the data points.
Distance-based approaches attempt to overcome
limitations of statistical outlier detection ap-
proaches and they detect outliers by computing
distances among points. Several distance-based
outlier detection algorithms that have been re-
cently proposed for detecting anomalies in net-
work traffic (Lazarevic, 2003) are based on com-
puting the full dimensional distances of points
from one another using all the available features,
and on computing the densities of local neighbor-
hoods. Values of categorical features are converted
into the frequencies of their occurrences and they
are further considered as continuous ones. MINDS
(Minnesota Intrusion Detection System) (Ertoz,
2004) employs outlier detection algorithms to as-
sign an anomaly score to each network connection.
A human analyst then has to look at only the most
anomalous connections to determine if they are
actual attacks or other interesting behavior. Ex-
periments on live network traffic have shown that
MINDS is able to routinely detect various suspi-
cious behavior (e.g., policy violations), worms, as
well as various scanning activities.
In addition, several clustering based techniques, such
as fixed-width and canopy clustering (Eskin, 2002), have
been used to detect network intrusions in DARPA 1998
data sets as small clusters when compared to the large
ones that corresponded to the normal behavior. In an-
other interesting approach (Fan, 2001), artificial anoma-
lies in the network intrusion detection data are generated
around the edges of the sparsely populated data regions,
thus forcing the learning algorithm to discover the spe-
cific boundaries that distinguish these regions from the
rest of the data.
Rule-based systems: Rule-based systems were
used in earlier anomaly detection based IDSs to ,
characterize normal behavior of users, networks
and/or computer systems by a set of rules. Ex-
amples of such rule-based IDSs include
ComputerWatch (Dowell, 1990) and Wisdom &
Sense (Liepins, 1992).
Profiling methods: In profiling methods, pro-
files of normal behavior are built for different
types of network traffic, users, programs and
etcetera; and deviations from them are consid-
ered as intrusions. Profiling methods vary greatly
ranging from different data mining techniques to
various heuristic-based approaches.
For example, ADAM (Audit Data and Mining) (Bar-
bara, 2001) is a hybrid anomaly detector trained on
both attack-free traffic and traffic with labeled attacks.
The system uses a combination of association rule
mining and classification to discover novel attacks in
tcpdump data by using the pseudo-Bayes estimator.
Recently reported IDDM system (Abraham, 2001) rep-
resents an off-line IDS, where the intrusions are de-
tected only when sufficient amounts of data are col-
lected and analyzed. The IDDM system describes pro-
files of network data at different times, identifies any
large deviations between these data descriptions and
produces alarms in such cases. PHAD (packet header
anomaly detection) (Mahoney, 2002) monitors net-
work packet headers and builds profiles for 33 differ-
ent fields from these headers by observing attack free
traffic and building contiguous clusters for the values
observed for each field. ALAD (application layer
anomaly detection) (Mahoney, 2002) uses the same
method for calculating the anomaly scores as PHAD,
but it monitors TCP data and builds TCP streams when
the destination port is smaller than 1024.
Finally, there have also been several recently pro-
posed commercial products that use profiling-based
anomaly detection techniques. For example, Antura
from System Detection (System Detection, 2003) use
data mining based user profiling, while Mazu Profiler
from Mazu Networks (Mazu Networks, 2003) and
Peakflow X from Arbor networks (Arbor Networks,
2003) use rate-based and connection profiling anomaly
detection schemes.
Model-based approaches: Many researchers
have used different types of data mining models,
such as replicator neural networks (Hawkins,
2002) or unsupervised support vector machines
(Eskin, 2002; Lazarevic, 2003), to characterize
the normal behavior of the monitored system. In
253
TEAM LinG

the model-based approaches, anomalies are de-
tected as deviations for the model that represents
the normal behavior. Replicator four-layer feed-
forward neural network (RNN) reconstructs input
variables at the output layer during the training
phase, and then uses the reconstruction error of
individual data points as a measure of outlyingness,
while unsupervised support vector machines at-
tempt to find a small region where most of the data
lies, label these data points as a normal behavior,
and then detect deviations from learned models as
potential intrusions. In addition, standard neural
networks (NN) were also used in intrusion detec-
tion problems to learn a normal profile. For ex-
ample NNs were often used to model the normal
behavior of individual users (Ryan, 1997), to build
profiles of software behavior (Ghosh, 1999) or to
profile network packets and queue statistics (Lee,
2000).
Summarization
Summarization techniques use frequent itemsets or as-
sociation rules to characterize normal and anomalous
behavior in the monitored computer systems. For ex-
ample, association patterns generated at different times
were used to study significant changes in the network
traffic characteristics at different periods of time (Lee,
2001). Association pattern analysis has also been shown
to be beneficial in constructing profiles of normal
network traffic behavior (Manganaris, 1999). MINDS
(Ertoz, 2004) uses association patterns to provide high-
level summary of network connections that are ranked
highly anomalous in the anomaly detection module.
These summaries allow a human analyst to examine a
large number of anomalous connections quickly and to
provide templates from which signatures of novel at-
tacks can be built for augmenting the database of signa-
ture-based intrusion detection systems.
FUTURE TRENDS
Intrusion detection techniques have improved dramati-
cally over time, especially in the past few years. IDS
technology is developing rapidly and its near-term fu-
ture is very promising. Data mining techniques for
intrusion detection increasingly become an indispens-
able and integral component of any comprehensive en-
terprise security program, since they successfully
complement traditional security mechanisms.
254
Data Mining for Intrusion Detection
CONCLUSION
Although a variety of techniques have been developed
for detecting different types of computer attacks in
different computer systems, there are still a number of
research issues concerning the prediction performance,
efficiency and fault tolerance of IDSs that need to be
addressed. Signature analysis, the most common strat-
egy in the commercial domain until recently, is increas-
ingly integrated with different anomaly detection and
alert correlation techniques based on advanced data
mining and artificial intelligence techniques in order to
detect emerging and coordinated computer attacks.
REFERENCES
Abraham, T. (2001). IDDM: Intrusion detection using
data mining techniques. Australia Technical Report
DSTO-GD-0286. DSTO Electronics and Surveillance
Research Laboratory, Department of Defense.
Arbor Networks. (2003). Intelligent network manage-
ment with peakflow traffic. Retrieved from http://
www.arbornetworks.com/products_sp.php
Barbara, D., Wu, N., & Jajodia, S. (2001). Detecting
novel network intrusions using Bayes estimators. In
Proceedings of the First SIAM Conference on Data Min-
ing, Chicago, IL.
Bloedorn, E., Christiansen, A., Hill, W., Skorupka, C.,
Talbot, L., & Tivel, J. (2001). Data mining for network
intrusion detection: How to get started. MITRE Tech-
nical Report. Retrieved from www.mitre.org/work/
tech_papers/tech_papers_01/bloedorn_datamining
Bridges, S., & Vaughn, R. (2000). Fuzzy data mining and
genetic algorithms applied to intrusion detection. In
Proceedings of the 23rd National Information Sys-
tems Security Conference, Baltimore, MD.
Cabrera, J., Ravichandran, B., & Mehra, R. (2000). Statis-
tical traffic modeling for network intrusion detection. In
The Proceedings of 8th International Symposium on
Modeling, Analysis and Simulation of Computer and
Telecommunication System.
Dao, V., & Vemuri, R. (2002). Computer network intrusion
detection: A comparison of neural networks methods.
Differential Equations and Dynamical Systems, Special
Issue on Neural Networks.
TEAM LinG
Data Mining for Intrusion Detection
Dowell, C., & Ramstedt, P. (1990). The Computerwatch
data reduction tool. In Proceedings of the 13th Na-
tional Computer Security Conference, Washington, DC.
Ertoz, L., Eilertson, E., Lazarevic, A., Tan, P., Srivastava,
J., Kumar, V., & Dokas, P. (2004). The MINDS: Minne-
sota intrusion detection system. In A. Joshi, H. Kargupta,
K. Sivakumar, & Y. Yesha (Eds.), Next generation data
mining. Boston: Kluwer Academic Publishers.
Eskin, E. (2000). Anomaly detection over noisy data using
learned probability distributions. In Proceedings of the
International Conference on Machine Learning, Stanford
University, CA.
Eskin, E., Arnold, A., Prerau, M., Portnoy, L., & Stolfo, S.
(2002). A geometric framework for unsupervised anomaly
detection: Detecting intrusions in unlabeled data. In S.
Jajodia & D. Barbara (Eds.), Applications of data mining
in computer security, advances in information security.
Boston: Kluwer Academic Publishers.
Fan, W., Lee, W., Miller, M., Stolfo, S.J., & Chan, P.K.
(2001). Using artificial anomalies to detect unknown
and known network intrusions. In the Proceedings of
the First IEEE International Conference on Data Min-
ing, San Jose, CA.
Ghosh, A., & Schwartzbard, A. (1999). A study in using
neural networks for anomaly and misuse detection. In
Proceedings of the Eighth USENIX Security Sympo-
sium (pp. 141-151).
Hawkins, S., He, H., Williams, G., & Baxter, R. (2002).
Outlier detection using replicator neural networks. In
Proceedings of the 4th International Conference on
Data Warehousing and Knowledge Discovery (pp.
170-180). Lecture Notes in Computer Science 2454.
Aix-en-Provence, France.
Joshi, M., Agarwal, R., & Kumar, V. (2001). PNrule,
mining needles in a haystack: Classifying rare classes
via two-phase rule induction. In Proceedings of the
ACM SIGMOD Conference on Management of Data,
Santa Barbara, CA.
Lazarevic, A., Ertoz, L., Ozgur, A., Srivastava, J., & Kumar,
V. (2003). A comparative study of anomaly detection
schemes in network intrusion detection. In Proceedings
of the Third SIAM International Conference on Data
Mining, San Francisco, CA.
Lazarevic, A., Kumar, V., & Srivastava, J. (2004). Intrusion
detection: A survey. In V. Kumar, J. Srivastava, & A.
Lazarevic (Eds.), Managing cyber threats: Issues, ap-
proaches and challenges. Boston: Kluwer Academic Pub-
lishers.
Lee, W., & Stolfo, S.J. (2000). A framework for construct-
ing features and models for intrusion detection systems. ,
ACM Transactions on Information and System Security,
3(4), 227-261.
Lee, W., Stolfo, S.J., & Mok, K. (2001). Adaptive intrusion
detection: A data mining approach. Artificial Intelli-
gence Review, 14, 533-567.
Liepins, G., & Vaccaro, H. (1992). Intrusion detection: Its role
and validation. Computers and Security, 347-355.
Lippmann, R., & Cunningham, R. (2000a). Improving intru-
sion detection performance using keyword selection and
neural networks. Computer Networks, 34 (4), 597-603.
Lippmann, R., Haines, J.W., Fried, D.J., Korba, J., & Das,
K. (2000b). The 1999 DARPA off-line intrusion detection
evaluation. Computer Networks.
Lippmann, R.P., Cunningham, R.K., Fried, D.J., Graf, I.,
Kendall, K.R., Webster, S.E., & Zissman, M.A. (1999).
Results of the DARPA 1998 offline intrusion detection
evaluation. In Proceedings of Workshop on Recent Ad-
vances in Intrusion Detection.
Mahoney, M., & Chan, P. (2002). Learning nonstationary
models of normal network traffic for detecting novel
attacks. In Proceedings of the Eighth ACM Interna-
tional Conference on Knowledge Discovery and Data
Mining (pp. 376-385), Edmonton, Canada.
Manganaris, S., Christensen, M., Serkle, D., & Hermiz, K.
(2000). A data mining analysis of RTID alarms. Computer
Networks, 34(4), 571-577.
Mazu Networks. (2003). Mazu Profiler. An Overview.
www.mazunetworks.com/solutions/white_papers/down-
load/Mazu_Profiler.pdf
Moore, D., Paxson, V., Savage, S., Shannon, C., Staniford,
S., & Weaver, N. (2003). The spread of the Sapphire/
Slammer Worm. Retrieved from www.cs.berkeley.edu/
~nweaver/sapphire
Mukkamala, S., Sung, A., & Abraham, A. (2003a). A
linear genetic programming approach for modeling in-
trusion. In Proceedings of the IEEE Congress on Evo-
lutionary Computation, Perth, Australia.
Ryan, J., Lin, M-J., & Miikkulainen, R. (1997). Intrusion
detection with neural networks. In Proceedings of the
AAAI Workshop on AI Approaches to Fraud Detection
and Risk Management (pp. 72-77), Providence, RI.
Schultz, M., Eskin, E., Zadok, E., & Stolfo, S. (2001). Data
mining methods for detection of new malicious executables.
In Proceedings of the IEEE Symposium on Security and
Privacy (pp. 38-49), Oakland, CA.
255
TEAM LinG

Sinclair, C., Pierce, L., & Matzner, S. (1999). An application
of machine learning to network intrusion detection. In
Proceedings of the 15th Annual Computer Security Ap-
plications Conference (pp. 371-377).
System Detection. (2003). Anomaly detection: The Antura
difference. Retrieved from http://www.sysd.com/library/
anomaly.pdf
Yamanishi, K., Takeuchi, J., Williams, G., & Milne, P.
(2000). On-line unsupervised outlier detection using
finite mixtures with discounting learning algorithms. In
Proceedings of the Sixth ACM SIGKDD International
Conference on Knowledge Discovery and Data Min-
ing (pp. 320-324), Boston, MA.
Ye, N., & Chen, Q. (2001a). An anomaly detection tech-
nique based on a chi-square statistic for detecting intru-
sions into information systems. Quality and Reliability
Engineering International Journal, 17(2), 105-112.
Ye, N., & Li, X. (2001b, June). A scalable clustering
technique for intrusion signature recognition. In Pro-
ceedings of the IEEE Workshop on Information Assur-
ance and Security. United States Military Academy,
West Point, NY.
256
Data Mining for Intrusion Detection
KEY TERMS
Anomaly Detection: Analysis strategy that identi-
fies intrusions as unusual behavior that differs from the
normal behavior of the monitored system.
Intrusion: Malicious, externally induced, operational
fault in the computer system.
Intrusion Detection: Identifying a set of malicious
actions that compromise the integrity, confidentiality,
and availability of information resources.
Misuse Detection: Analysis strategy that looks for
events or sets of events that match a predefined pattern
of a known attack.
Signature-Based Intrusion Detection: Analysis
strategy where monitored events are matched against a
database of attack signatures to detect intrusions.
Tcpdump: Computer network debugging and secu-
rity tool that allows the user to intercept and display
TCP/IP packets being transmitted over a network to
which the computer is attached.
Worms: Self-replicating programs that aggressively
spread through a network by taking advantage of auto-
matic packet sending and receiving features found on
many computers.
TEAM LinG

Data Mining in Diabetes Diagnosis and
Detection
Indranil Bose
The University of Hong Kong, Hong Kong
INTRODUCTION
Diabetes is a disease worrying hundreds of millions of
people around the world. In the USA, the population of
diabetic patients is about 15.7 million (Breault et al., 2002).
It is reported that the direct and indirect cost of diabetes
in the USA is $132 billion (Diabetes Facts, 2004). Since
there is no method that is able to eradicate diabetes,
doctors are striving for ways to fight this doom. Research-
ers are trying to link the cause of diabetes with patients
lifestyles, inheritance information, age, and so forth in
order to get to the root of the problem. Due to the
prevalence of a large number of responsible factors and
the availability of historical data, data mining tools have
been used to generate inference rules on the cause and
effect of diabetes as well as to help in knowledge discov-
ery in this area. The goal of this chapter is to explain the
different steps involved in mining diabetes data and to
show, using case studies, how data mining has been
carried out for detection and diagnosis of diabetes in
Hong Kong, USA, Poland, and Singapore.
BACKGROUND
Diabetes is a severe metabolic disorder marked by high
blood glucose level, excessive urination, and persistent
thirst, caused by lack of insulin actions. There are usually
three forms of diabetesType 1, Type 2, and gestational.
It is believed that diabetes is a particularly opportune
disease for data mining technology for a number of rea-
sons (Breault, 2001):
There are many diabetic databases with historic
patient information.
New knowledge about treatment of diabetes can
help save money.
Diabetes can produce terrible complications like
blindness, kidney failure, and so forth, so physi-
cians need to know how to identify potential cases
quickly.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
257
,
The availability of historical data on diabetes natu-
rally leads to the application of data mining techniques to
discover interesting patterns (Apte et al., 2002; Hsu et al.,
2000). The objective is to find rules that help to under-
stand diabetes, facilitate early detection of diabetes, and
discover how diabetes may be associated with different
segments of the population.
The data mining process for diagnosis of diabetes can
be divided into five steps, though the underlying prin-
ciples and techniques used for data mining diabetic data-
bases may differ for different projects in different coun-
tries. Following is a brief description of the five steps.
Step 1: Data Cleaning
Before carrying out data mining on the diabetic patient
database, the data should be cleaned. Errors such as
missing values, typographical errors, or wrong informa-
tion are contained in the patient records, and, worse still,
many records are duplicate records. Two approaches can
be used to clean the data, namely standardized format
schema and sorted neighborhood method (Hsu et al.,
2000). By generating the standardized format schema, a
user defines mappings among attributes in different for-
mats, and each of the database files are modified into this
standardized format. The next task is to look for duplicate
records that have to be removed from the standardized
format using the sorted neighborhood method. Under this
scheme, the database is sorted on one or more fields that
uniquely identify each record, and the chosen fields of the
records are compared within a sliding window. When
duplicates are detected, the user is called in to verify it,
and the duplicates are removed from the database.
Step 2: Data Preparation
The importance of the data preparation step cannot be
overstated, since the success of data analysis depends on
it (Breault et al., 2002). A fundamental issue is whether to
use a relational database comprising multiple tables or a
flat file that is best suited for data mining (Breault, 2002).
TEAM LinG

Generally, either of the following two methods is adopted.
In the first method, a set of flat files is used to represent
all the data, then data mining tools are used separately on
each file, and the results obtained from the flat files are
linked together in some fashion. An alternative method is
to find a data reduction technique that allows fields in a
complex database to be transformed into vector scores
instead of considering them separately. For example, for
an Australian study, the 26 items recorded for a patient
from a diabetes database was converted to a four-compo-
nent vector <HgbA1c, Eye, Lipids, Microalbumin> by
assigning them to each component with an associated
weight (determined by domain experts) that indicated how
strongly the item related to the vector, and significant
data reduction was obtained.
Step 3: Data Analysis
Data mining tools are used to convert the data stored in
databases into useful knowledge. Different software pro-
grams using different models are run on the same data to
provide different results. Sometimes, the result may be
unexpected, but many of the rules and causal relation-
ships discovered can conform to the trends. A software
that is commonly used for data mining is Classification
and Regression Tree (CART). CART recursively parti-
tions the input variable space to maximize purity in the
terminal tree nodes (Breault et al., 2002).
Step 4: Knowledge Evaluation
The rules obtained from data mining may not be meaning-
ful or true. The experts have to evaluate the knowledge
before using it. For example, multiple random samples
should be used to evaluate the data mining tools used to
insure that results are not just by chance (Breault, 2001).
For diabetes, similar data mining studies in other geo-
graphic and cultural locations are needed to prove any
results that are suspected to be specific to one region or
segment of population.
Step 5: Knowledge Usage
After knowledge is extracted as a result of data mining, the
developers have to address the issue of cost effective-
ness of applying that knowledge for detection of diabe-
tes. As a result of Step 4, the data miners may obtain a
costly solution that can help a small group of people.
However, this may not prove helpful for diagnosing
diabetes for the entire population. It is the effective usage
of the knowledge gathered from data mining for real-life
application that will mark the success of the endeavor.
258
Data Mining in Diabetes Diagnosis and Detection
MAIN THRUST
Hong Kong CaseDiabetes Registry
and Statistical Analysis
In Hong Kong, around 10% of the population is suffering
from diabetes, of which 5% belong to Type 1 and 95%
belong to the Type 2 form of diabetes. The prevalence of
Type 2 diabetes is increasing at an alarming rate among
Chinese, and its development is believed to involve the
interplay between genetic and environmental factors. In
view of this, the major hospitals have established their
own diabetes registry for knowledge discovery. For in-
stance, the diabetes clinic at the Prince of Wales Hospital
of Hong Kong has adopted a structured diabetes care
protocol and has created a diabetes registry based on a
modified Europe DIAB-CARE format (Apte et al., 2002)
since 1995.
In September 2000, a diabetes data mining study was
conducted to investigate the patterns of diabetes and
their relationships with clinical characteristics in Hong
Kong Chinese patients with late-onset (over age 34) Type
2 diabetes (Lee et al., 2000). This study involved 2,310
patients selected from a hospital clinic-based diabetes
registry. A statistical analysis toolStatistical Package
for Social Sciences (SPSS)was used for conducting t-
test, Mann-Whitney U test, analysis of variance, and
2 test. Many useful results were generated, which can
be found on the two tables on page 1366 of the paper by
Lee, et al. (2000). For example, it was found that the
patients, irrespective of their sex, were more likely to have
a diabetic mother than a diabetic father. Also, female
patients with a diabetic mother were found to have higher
levels of plasma total cholesterol compared to those
having a diabetic father. In two-group comparisons, there
was also evidence that the male patients with a diabetic
father had higher body mass index (BMI) values than the
male patients with a diabetic mother. It also was shown
that both maternal and paternal factors may be respon-
sible for the development of Type 2 diabetes in the
Chinese population. All these rules greatly improved the
physicians understanding of diabetes.
United States CaseClassification and
Regression Trees (CART)
Diabetes is a major health problem in the United States.
According to current statistics, about 5.9% of the popu-
lation, or 16 million people in the USA, have diabetes. It
is estimated that the percentage will rise to 8.9% by 2025.
In the United States, there is a long history of diabetic
TEAM LinG
Data Mining in Diabetes Diagnosis and Detection
registries and databases with systematically collected
patient information. A large-scale research was carried out
by applying data mining techniques to a diabetic data
warehouse from an integrated health care system in the
New Orleans area with 30,383 diabetic patients (Breault et
al., 2002).
To prepare the data tables, structured query language
(SQL) statements were executed on the data warehouse to
form the flat file as input to data mining software. To do
data mining, CART software was used with a binary target
variable and ten predictors: age, sex, emergency depart-
ment visits, office visits, comorbidity index, dyslipidemia,
hypertension, cardiovascular disease, retinopathy, and
end-stage renal disease. The outcome showed that the
most important variable associated with bad glycemic
control is younger age, not the comorbiditity index or
whether patients have related diseases. The total classifi-
cation error (40.5%) was substantial.
PIMA Indian CaseMachine Learning
Mining Approach
Another special group in the USA that deserves a separate
mention because of the importance of the findings is the
PIMA Indian case. With a very high rate of diabetes for
Pima Indians, the Pima Indian Diabetes Database (PIDD)
with 768 diabetes patients has been established by the
National Institute of Health.
There have been many studies applying data mining
techniques to the PIDD. Some well-known examples of
data mining techniques used include multi-stream depen-
dency detection (MSDD) algorithm, Bayesian neural net-
works, and multiplier-free feed-forward neural networks.
Although the cited examples use somewhat different sub-
groups of the PIDD, accuracy for predicting diabetes
ranges from 66% to 81%. It is interesting to see that with
a wide variety of prediction tools available, efficiency and
accuracy can be improved greatly in diabetes data mining.
Recently, several modified data mining techniques have
been used successfully on the same database. This in-
cludes the usage of decision trees with an augmented
splitting criterion (Buja & Lee, 2001). The use of the fuzzy
Nave Bayes method yielded a best-case accuracy of
76.95% (Tang et al., 2002). The application of shunted
inhibitory neural networks, where the neurons can act as
adaptive non-linear filters, resulted in an accuracy of over
80% and performed better than multi-layer perceptrons, in
general (Arulampalam & Bouzerdoum, 2001).
Poland CaseRough Sets Technique
In Poland, more than 1.5 million people suffer from diabe-
tes, and more than 41,000 have Type 1 diabetes (Medtronic
Figure 1. Image of ROSETTA software on training and
test data sets ,
Limited, 2002). Rough sets have been applied to mine
data in a Polish diabetes database using the ROSETTA
software (hrn, 1997). The rough sets approach investi-
gates structural relationships in the data rather than
probability distributions and produces decision tables
rather than trees.
A recent study from a Polish medical school used a
dataset of 107 patients aged five to 22 who were suffering
from insulin-dependent diabetes. In this study, it was
found that the minimal subsets of attributes that are
efficient for rule making included age of disease diagno-
sis, microalbuminuria (yes/no) and disease duration in
years. With the use of rough sets techniques, decision
rules were generated to predict microalbuminuria. The
best predictor was age <7 predicting no microalbuminuria
at 83.3% accuracy.
Singapore CaseData Cleansing and
Interaction With Domain Experts
Around 10% of the population in Singapore is diabetic.
In the diabetes data mining exercise conducted in
Singapore, the objective was to find rules that could be
used by physicians to understand more about diabetes
and to find special patterns about a particular patient
population (Hsu et al., 2000). In order to deal with noisy
data, a semi-automatic data cleaning system was utilized
to reconcile format differences among tables with user
mapping input. A sorted neighborhood method was
used to remove duplicate records.
In a particular case at the National University of
Singapore, the researchers mined the data via a classifi-
cation with association rule tool, using thresholds of 1%
support and 50% confidence. They generated 700 rules.
The physicians were overwhelmed with the large number
and also wanted to know causal connections rather than
259
TEAM LinG

associations. It was realized that post-processing was
needed in order to make the results usable for the physicians.
The tree method was employed to generate general
rules giving the underlying trends in the data that the
physicians already knew and exception rules giving de-
viations to these trends. The physicians found the excep-
tion rules especially helpful in understanding how sub-
population trends differ from the main population. With
the help of the physicians domain knowledge, the data
mining process was optimized by reducing the size of data
and hypothesis space and by removing unnecessary
query operations (Owrang, 2000).
FUTURE TRENDS
An important step in using data mining for diagnosing
diabetes is the creation of a centralized database that can
store most diabetic patients health data. The larger the
data pool, the more accurate the results from data mining
are likely to be. The experiences and the many challenges
faced in building a data warehouse for a not-for-profit
organization called Christiana Care in the state of Dela-
ware, USA, is described by Ewen, et al., and it is reported
that the creation of the data warehouse was responsible
for a gain in operating revenue in 1997 (Ewen et al., 1998).
Though the task is not easy, for successful data mining,
it is almost imperative to create data warehouses for
sharing of diabetes-related information among physi-
cians across the world. The major obstacle is that there is
only a small number of diabetic patients in this world that
have access to shared care, while many others remain
undiagnosed, untreated, or suboptimally treated (Chan,
2000). This is a major challenge that has to be overcome
in the future.
The case studies clearly indicate that there is no single
method that proves to be effective for mining diabetes
databases. Many tools and techniques, such as statistical
methods, machine learning algorithms, neural networks,
Bayesian neural sets, and rough sets, have been em-
ployed to study and figure out useful rules and patterns
to help physicians combat the deadly disease. There is
further need in the future to use other techniques like
case-based reasoning for assessment of risk of complica-
tions for individual diabetes patients (Armengol et al.,
2004; Montani et al., 2003) or hybrid techniques like rule
induction, using simulated annealing for discovering
associations between observations made of patients on
their first visit and early mortality (Richards et al., 2001).
Moreover, besides doing data mining in diabetes
databases, it is also important that the same work be
carried out for studying the complications of diabetes and
understanding its relationship to other diseases. By ap-
plying different data-mining techniques and tools, more
260
Data Mining in Diabetes Diagnosis and Detection
knowledge about associations between diabetes and
other diseases have to be dug out in order to improve
public health in the future. Though Type 1 and Type 2
diabetes have been studied frequently, more effort is
needed in the future to study the diagnosis and detection
of gestational diabetes.
CONCLUSION
The occurrence of diabetes is increasing at an alarming
rate all over the world, and its development is believed to
involve the interplay among many unknown and mysteri-
ous reasons such as genetic and environmental factors.
In view of this, data-mining technology can play a very
important role in analyzing existing diabetes databases
and identifying useful rules that help diabetes prevention
and control.
It is worthwhile to recognize that successful applica-
tion of data-mining technologies to diabetes prevention
and control requires: (1) preparing a comprehensive dia-
betes database for input into data mining software to
avoid garbage in and garbage out; this will require data
cleansing and transformations from a relational data ware-
house to a data mining data table that is useable by data
mining tools; (2) selecting and skillfully applying the
appropriate data-mining software and techniques; (3)
intelligently sifting through the software output to priori-
tize the areas that will provide the most cost savings or
outcomes improvement; and (4) interacting with domain
experts to select the best-fit rules and patterns that opti-
mize the efficiency and effectiveness of the data-mining
process. In this paper, we have studied how the above
steps are applied for mining diabetes databases in gen-
eral, and, in particular, we have discussed the various
methods adopted by researchers in different countries for
diagnosis of diabetes.
REFERENCES
Apte, C., Liu, B., Pednault, E.P.D., & Smyth, P. (2002).
Business applications of data mining. Communications of
the ACM, 45(8), 49-53.
Armengol, E., Palaudaries, A., & Plaza, E. (2004). Indi-
vidual prognosis of diabetes long-term risks: A CBR
approach. Technical Report IIIA.
Arulampalam, G., & Bouzerdoum, A. (2001). Application
of shunting inhibitory artificial neural networks to medical
diagnosis. Proceedings of the Seventh Australian and
New Zealand Intelligent Information Systems Confer-
ence.
TEAM LinG
Data Mining in Diabetes Diagnosis and Detection
Breault, J.L. (2001). Data mining diabetic databases: Are
rough sets a useful addition? [Electronic version]. Com-
puting Science and Statistics, 33.
Breault, J.L. (2002). Mathematical challenges of variable
transformations in data mining diabetic data ware-
houses. Retrieved from http://www.ipam.ucla.edu/pub-
lications/sdm2002/sdm2002_jbreault_poster.pdf
Breault, J.L., Goodall C.R., & Fos, P.J. (2002). Data mining
a diabetic data warehouse. Artificial Intelligence in
Medicine, 26, 37-54.
Buja, A., & Lee, Y-S. (2001). Data mining criteria for tree-
based regression and classification. San Francisco, CA:
KDD 2001.
Chan, J.C.N. (2000). Heterogeneity of diabetes mellitus in
the Hong Kong Chinese population. Hong Kong Medical
Journal, 6(1), 77-84.
Diabetes Facts. (2004). Retrieved from http://
diabetes.mdmercy.com/about_diabetes/facts.html
Ewen, E.F. et al. (1998). Data warehousing in an inte-
grated health system: Building the business case. Wash-
ington, D.C.: DOLAP 98.
Hsu, W., Lee, M.L., Liu, B., & Ling, T.W. (2000). Explora-
tion mining in diabetic patients databases: Findings and
conclusions. Proceedings of the Sixth ACM SIGKDD
International Conference on Knowledge Discovery and
Data Mining. Boston, Massachusetts.
Lee, S.C. et al. (2000). Diabetes in Hong Kong Chinese:
Evidence for familial clustering and parental effects. Dia-
betes Care, 23, 1365-1368.
Medtronic Limited. (2002). Diabetes overview. Retrieved
from http://www.medtronic.com/UK/health/diabetes/
diabetes_overview.html
Montani, S. et al. (2003). Integrating model-based deci-
sion support in a multi-modal reasoning systems for
managing type 1 diabetic patients. Artificial Intelligence
in Medicine, 29, 131-151.
hrn, A., & Komorowski, J. (1997). ROSETTA: A rough
set toolkit for analysis of data. Proceedings of the Joint
Conference of Information Sciences, Durham, North Carolina.
Owrang, M.M. (2000). Using domain knowledge to opti-
mize the knowledge discovery process in databases.
International Journal of Intelligent Systems,15, 45-60.
Richards, G., Rayward-Smith, V.J., Sonksen, P.H., Carey,
S., & Weng, C. (2001). Data mining for indicators of early ,
mortality in a database of clinical records. Artificial Intel-
ligence in Medicine, 22, 215-231.
Tang, Y., Pan, W., Qiu, X, & Xu, Y. (2002). The identifica-
tion of fuzzy weighted classification system incorporated
with fuzzy Nave Bayes from data. Proceedings of the
IEEE International Conference on Systems, Man and
Cybernetics.
KEY TERMS
Body Mass Index (BMI): A measure of mass of an
individual that is calculated as weight divided by height
squared.
Data Warehouse: A database, frequently very large,
that can access all of a companys information. It contains
data about how the warehouse is organized, where the
information can be found, and any connections between
existing data.
Gestational Diabetes: This form of diabetes develops
in 2% to 5% of all pregnancies but disappears when a
pregnancy is over.
Neural Networks: Computer processors or software
based on the human brains mesh-like neuron structure.
Neural networks can learn to recognize patterns and
programs to solve related problems on their own.
Rough Sets: A method of representation of uncer-
tainty in the membership of a set. It is related to fuzzy sets
and is a popular data mining technique in medicine and
finance.
Type 1 Diabetes: This insulin-dependent diabetes
mellitus includes risk factors that are less well defined.
Autoimmune, genetic, and environmental factors are in-
volved in the development of this type of diabetes.
Type 2 Diabetes: This non-insulin-dependent diabe-
tes mellitus accounts for about 90% to 95% of all diag-
nosed cases of diabetes. Risk factors include older age,
obesity, family history, prior history of gestational diabe-
tes, impaired glucose tolerance, physical inactivity, and
race/ethnicity.
261
TEAM LinG
262
Data Mining in Human Resources
Marvin D. Troutt
Kent State University, USA
Lori K. Long
Kent State University, USA
INTRODUCTION
In this paper, we briefly review and update our earlier work
(Long & Troutt, 2003) on the topic of data mining in the
human resources area. To gain efficiency, many organiza-
tions have turned to technology to automate many HR
processes (Hendrickson, 2003). As a result of this auto-
mation, HR professionals are able to make more informed
strategic HR decisions (Bussler & Davis, 2002). While HR
professionals may no longer need to manage the manual
processing of data, they should not abandon their ties to
data collected on and about the organizations employ-
ees. Using HR data in decision-making provides a firm
with the opportunity to make more informed strategic
decisions. If a firm can extract useful or unique informa-
tion on the behavior and potential of their people from HR
data, they can contribute to the firms strategic planning
process. The challenge is identifying useful information
in vast human resources databases that are the result of
the automation of HR related transaction processing.
Data mining is essentially the extracting of knowledge
based on patterns of data in very large databases and is
an analytical technique that may become a valuable tool
for HR professionals. Organizations that employ thou-
sands of employees and track employment related infor-
mation might find valuable information patterns con-
tained within their databases to provide insights in such
areas as employee retention and compensation planning.
To develop an understanding of the potential of data
mining HR information in a firm, we will identify opportu-
nities as well as concerns in applying data mining tech-
niques to HR Information Systems.
BACKGROUND
In this section, we review existing work on the topic. The
human resource information systems (HRIS) of most
organizations today feature relational database systems
that allow data to be stored in separate files that can be
linked by common elements such as name or identification
number. The relational database provides organizations
with the ability to keep a virtually limitless amount of data
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
on employees. It also allows organizations to access the
data in a variety of ways. For example, a firm can retrieve
data on a particular employee or they can retrieve data on
a certain group of employees through conducting a search
based on a specific parameter such as job classification.
The development of relational databases in organizations
along with advances in storage technology has resulted
in organizations collecting a large amount of data on
employees.
While these calculations are helpful to quantify the
value of some HR practices, the bottom-line impact of HR
practices is not always so clear. One can evaluate the cost
per hire, but does that information provide any reference
to the value of that hire? Should a greater value be
assessed to an employee who stays with the organization
for an extended period of time? Most data analysis re-
trieved from HRIS does not provide an opportunity to
seek out additional relationships beyond those that the
system was originally designed to identify.
Traditional data analysis methods often involve
manual work and interpretation of data that is slow,
expensive and highly subjective (Fayyad, Piatsky-Shapiro,
& Smyth, 1996a). For example, if an HR professional is
interested in analyzing the cost of turnover, they might
have to extract data from several different sources such as
accounting records, termination reports and personnel
hiring records. That data is then combined, reconciled and
evaluated. This process creates many opportunities for
errors. As business databases have grown in size, the
traditional approach has grown more impractical.
Data mining has been used successfully in many
functional areas such as finance and marketing. HRIS
applications in many organizations provide an as yet
unexplored opportunity to apply data mining techniques
(Patterson & Lindsay, 2003). While most applications
provide opportunities to generate ad-hoc or standardized
reports from specific sets of data, the relationships be-
tween the data sets are rarely explored. It is this type of
relationship that data mining seeks to discover.
The blind application of data mining techniques can
easily lead to the discovery of meaningless and invalid
patterns. If one searches long enough in any data set, it
is likely possible to find patterns that appear to hold but
TEAM LinG
Data Mining in Human Resources
are not necessarily statistically significant or useful
(Fayyad et al., 1996a). There has not been any specific
exploration of applying these techniques to human re-
source applications; however, there are some guidelines
in the process that are transferable to an HRIS. Feelders,
Daniels and Holsheimer (2000) outline six important steps
in the data mining process: 1) problem definition, 2)
acquisition of background knowledge, 3) selection of
data, 4) pre-processing of data, 5) analysis and interpre-
tation, and 6) reporting and use. At each of these steps,
we will look at important considerations as they relate to
data mining human resources databases. Further, we will
examine some specific legal and ethical considerations of
data mining in the HR context.
The formulation of the questions to be explored is an
important aspect of the data mining process. As men-
tioned earlier, with enough searching or application of
sufficiently many techniques, one might be able to find
useless or ungeneralizable patterns in almost any set of
data. Therefore, the effectiveness of a data mining project
is improved through establishing some general outlines
of inquiry prior to start the project. To this extent, data
mining and the more traditional statistical studies are
similar. Thus, careful attention to the scientific method
and sound research methods are to be followed. A widely
respected source of guidelines on research methods is the
book by Kerlinger and Lee (2000).
A certain level of expertise is necessary to carefully
evaluate questions posed in a data mining project. Obvi-
ously, a requirement is data mining and statistical exper-
tise, but one must also have some intimate understanding
of the data that is available, along with its business
context. Furthermore, some subject matter expertise is
needed to determine useful questions, select relevant
data and interpret results (Feelders et al., 2000). For
example, a firm with interest in evaluating the success of
an affirmative action program needs to understand the
Equal Employment Opportunity (EEO) classification sys-
tem to know what data is relevant.
Another important consideration in the process of
developing a question to look at is the role of causality
(Feelders et al., 2000). A subject matter experts involve-
ment is important in interpreting the results of the data
analysis. For example, a firm might find a pattern indicat-
ing a relationship between high compensation levels and
extended length of service. The question then becomes,
do employees stay with the company longer because they
receive high compensation? Or do employees receive
higher compensation if they stay longer with the com-
pany? An expert in the area can take the relationship
discovered and build upon it with additional information
available in the organization to help understand the cause
and effect of the specific relationship identified.
Selecting and preparing the data is the next step in the
data mining process. Some organizations have indepen- ,
dent Human Resource Information Systems that feature
multiple databases that are not connected to each other.
This type of system is sometimes selected to offer greater
flexibility to remote organizational locations or sub-groups
with unique information needs (Anthony et al., 1996). The
possible inconsistency of the design of the databases
could make data mining difficult when multiple databases
exist. Data warehousing can prevent this problem and an
organization may need to create a data warehouse before
they begin a data-mining project. The advantage gained
in first developing the data warehouse or mart is that most
of the data editing is effectively done in advance.
Another challenge in mining data is dealing with the
issues of missing or noisy data. Data quality may be
insufficient if data is collected without any specific analy-
sis in mind (Feelders et al., 2000). This is especially true
for human resource information. Typically when HR data
is collected, the purpose is some kind of administrative
need such as payroll processing. The need of data for the
required transaction is the only consideration in the type
of data to collect. Future analysis needs and the value in
the data collected is not usually considered. Missing data
may also be a problem, especially if the system adminis-
trator does not have control over data input. Many orga-
nizations have taken advantage of web-based technology
to allow for employee input and updating of their own data
(Hendrickson, 2003). Employees may choose not to enter
certain types of data resulting in missing data. However,
a data warehouse or datamart may help to prevent or
systemized the handling of many of these problems.
There are many types of algorithms in use in data
mining. The choice of the algorithm depends on the
intended use of the extracted knowledge (Brodley, Lane,
& Stough, 1999). The goals of data mining can be broken
down into two main categories. Some applications seek to
verify the hypothesis formulated by the user. The other
main goal is the discovery or uncovering new patterns
systematically (Fayyad et al., 1996a). Within discovery,
the data can be used to either predict future behavior or
describe patterns in an understandable form. A complete
discussion of data mining techniques is beyond the scope
of this paper. However, the following techniques have the
potential to be applicable for data mining of human re-
sources information.
Clustering and classification is an example of a set of
data mining techniques borrowed from classical statisti-
cal methods that can help describe patterns in informa-
tion. Clustering seeks to identify a small set of exhaustive
and mutual exclusive categories to describe the data that
is present (Fayyad et al., 1996a). This might be a useful
application to human resource data if you were trying to
263
TEAM LinG

identify a certain set of employees with consistent at-
tributes. For example, an employer may want to find out
what the main categories of top performers are that
employees fall into with an eye towards tailoring various
programs to the groups or further study of such groups.
One category may be more or less appropriate for one type
of training program. A difficulty with clustering tech-
niques is that no normative techniques are known that
specify the correct number of clusters that should be
formed. In addition, there exist many different logics that
may be followed in forming the clusters. Therefore, the art
of the analyst is critical.
Similarly, classification is a data mining technique that
maps a data item into one of several pre-defined classes
(Fayyad et al., 1996a). Classification may be useful in
human resources to classify trends of movement through
the organization for certain sets of successful employees.
A company is at an advantage when recruiting if it can
point out some realistic career paths for new employees.
Being able to support those career paths with information
reflecting employee success can make this a strong re-
source for those charged with hiring in an organization.
Decision Tree Analysis, also called tree or hierarchical
partitioning, is a somewhat related technique but follows
a very different logic and can be rendered somewhat more
automatic. Here, a variable is chosen first in such a way as
to maximize the difference or contrast formed by splitting
the data into two groups. One group consists of all obser-
vations having a value higher on a certain value of the
variable, such as the mean. Then the complement, namely
those lower than that value, becomes the other group.
Then each half can be subjected to successive further
splits with possibly different variables becoming impor-
tant to different halves. For example, employees might first
be split into two groups above and below average tenure
with the firm. Then the statistics of the two groups can be
compared and contrasted to gain insights about employee
turnover factors. A further split of the lower tenure group,
say based on gender, may help prioritize those most likely
to need special programs for retention. Thus, clusters or
categories can be formed by binary cuts, a kind of divide
and conquer approach. In addition, the order of variables
can be chosen differently to make the technique more
flexible. For each group formed, summary statistics can be
presented and compared. This technique is a rather pure
form of data mining and can be performed in the absence
of specific questions or issues. It might be applied as a way
of seeking interesting questions about a very large datamart.
Regression and related models, also borrowed from
classical statistics, permits estimating a linear function of
independent variables that best explains or predicts a
given dependent variable. Since this technique is gener-
ally well known, we will not dwell on the details here.
However, data warehouses and datamarts may be so large
264
Data Mining in Human Resources
that direct use of all available observations is imprac-
tical for regression and similar studies. Thus, random
sampling may be necessary to use regression analysis.
Various nonlinear regression techniques are also avail-
able in commercial statistical packages and can be used
in a similar way for data mining. Recently, a new model
fitting technique was proposed in Troutt, et al. (2001).
In this approach the objective is to explain the highest
or lowest performers, respectively, as a function of
one or more independent variables.
The final step in the process emphasizes the value of
the use of the information. The information extracted
must be consolidated and resolved with previous infor-
mation and then shared and acted upon (Fayyad, Piatsky-
Shapiro, Smyth, & Uthurusamy, 1996b). Too often, orga-
nizations go through the effort and expense of collecting
and analyzing data without any idea of how to use the
information retrieved. Applying data mining techniques
to an HRIS can help support the justification of the
investment in the system. Therefore, the firm should
have some expected use for the information retrieved in
the process.
One use of human resource related information is to
support decision-making in the organization. The results
obtained from data mining may be used for a full range of
decision-making steps. It can be used to provide infor-
mation to support a decision, or can be fully integrated
into an end-user application (Feelders et al., 2000). For
example, a firm might be able to set up decision rules
regarding employees based on the results of data mining.
They might be able to determine when an employee is
promotion eligible or when a certain work group should
be eligible for additional company benefits.
Organizational leaders must be aware of legislation
concerning legal and privacy issues when making deci-
sions about using personal data collected from individu-
als in organizations (Hubbard, Forcht, & Thomas, 1998).
By their nature, systems that collect employee informa-
tion run the risk of invading the privacy of employees by
allowing access to the information to others within the
organization. Although there is no explicit constitutional
right to privacy, certain amendments and federal laws
have relevance to this issue as they provide protection
for employees from invasion of privacy and defamation
(Fisher, Schoenfeldt, & Shaw, 1999). Further, recent
epidemics of identity theft create a need for organiza-
tions to monitor access to employee information (Carlson,
2004). Organizations can protect themselves from these
employee concerns by having solid business reasons for
any data collected from employees and ensuring access
to this data is restricted.
There are also some potential legal issues if a firm
uses inappropriate information extracted from data min-
ing to make employment related decisions. Even if a
TEAM LinG
Data Mining in Human Resources
manager has an understanding of current laws, they
could still face challenges as laws and regulations con-
stantly change (Ledvinka & Scarpello, 1992). An ex-
treme example that may violate equal opportunity laws
is a decision to hire only females in a job classification
because the data mining uncovered that females were
consistently more successful.
One research study found that an employees ability
to authorize disclosure of personal information affected
their perceptions of fairness and invasion of privacy
(Eddy & Stone-Romero, 1999). Therefore, it is recom-
mended that firms notify employees upon hire that the
information they provide may be used in data analyses.
Another recommendation is to establish a committee or
review board to monitor any activities relating to analysis
of personal information (Osborn, 1978). This committee
can review any proposed research and ensure compliance
with any relevant employment or privacy laws. Often the
employees reaction to the use of their information is
based upon their perceptions. If there is a perception that
the company is analyzing the data to take negative actions
against employees, employees are more apt to object to
the use. However, if the employer takes the time to notify
employees and obtain their permission, the perception of
negativity may be removed. Even if there may be no legal
consequences, employee confidence is something that
employers need to maintain.
UPDATES AND ISSUES
As of this writing, there are still surprisingly few reported
industry experiences with DM in HR. Of course, the
newness of the area will require a time lag. In addition,
many HR groups will not yet have the requisite experts
and information systems capabilities. However, there are
several forces at work that should improve the situation
in the near future.
First, DM for the human resources area is rapidly
becoming of interest to industry associations and soft-
ware vendors. Human Resources Benchmarking
Association(http://www.hrba.org/roundtable.pdf) now
provides links related to DM. In fact, this association is
affiliated with and linked to the Data Mining Benchmarking
Association.
The former offers a variety of services related to DM,
such as
Consortium studies with costs divided
HR benchmarking efforts
Data collection and database access
Benchmarking studies of important data mining
processes.
Similarly, Dynamic Health Strategies (DHS, http://
www.dhsgroup.com/) is an association that concentrates ,
on group health benefit issues. It combines the use of
proprietary technology, audit discipline, analytical soft-
ware, and bio-statistical evaluation techniques to assess
and improve the quality and performance of existing
health care providers, health plans, and health systems.
DHS performs analysis services for self-insured corpora-
tions, universities, government entities and group health
management. The process allows DHS to pinpoint spe-
cific measures that enable clients to reduce costs, reduce
healthcare risk exposure and improve quality of care.
Group health analysis allows the monitoring of cost,
quality and utilization of health care prior to problems
becoming major issues. Clients and consultants are then
able to apply the findings and recommendations to realize
benefits such as: targeting of specific health and wellness
issues, monitoring of utilization and cost over time, as-
sessment of effectiveness and efficiency of health and
wellness initiatives, design health plan benefits to meet
specific needs, application of risk models and cost projec-
tions for budgeting and financial strategies.
Evidently, the group health insurance benefits area is
spearheading interest in DM. We may postulate several
reasons for this. First, this area represents one of consid-
erable importance in terms of financial impacts. Next, the
existence of the Health Insurance Portability and Ac-
countability Act of 1996 (HIPAA) (http://
www.hipaadvisory.com/) created a requirement for ad-
ministrative simplification, compliance, and reporting
within healthcare entities. Third, the National Committee
for Quality Assurance (NCQA) (http://www.ncqa.org/
index.asp) has established a quality assurance system
called the Health Plan Employer Data and Information Set
(HEDIS). Information Systems for reporting related to
HIPAA and HEDIS serve as a ready data source for DM.
Software vendors have been developing new DM
tools at an accelerating pace. At least one vendor,
Lawson (http://www.bitpipe.com/), has developed a prod-
uct specialized to provide HR reporting, monitoring, and
decision support. The Defense Software Collaborators
(DACS, http://www.dacs.dtic.mil/) website has collected
links to a very large number of DM tools and vendors.
Many of these purport to be designed for the nontechni-
cal business user.
FUTURE TRENDS
The increasing interest in DM by HR related associations
will likely continue and should provide a strong impetus
and guidance for individual member firms. These associa-
tions make it possible for member firms to pool their
databases. Such pooled information gives members a
265
TEAM LinG

statistical leverage in that their own data may be insuf-
ficient for particular studies. However, when their par-
ticular data are viewed in the context of the larger
database, Bayesian methods might be brought to bear to
make better inferences that may not be reliable with
smaller data sets (Bickell & Doksum, 1977).
In addition to more firm-specific experience studies,
evaluation studies and comparisons are needed for the
various DM tools becoming available. Potential adopters
need assessments of both the effectiveness and ease of use.
CONCLUSION
The use of HR data beyond administrative purposes can
provide the basis for a competitive advantage by allowing
organizations to strategically analyze one of their most
important assets, their employees. Organizations must be
able to transform the data they have collected into useful
information. Data mining provides an attractive opportu-
nity that has not yet been adequately exploited. The
application of DM techniques to HR requires organiza-
tional expertise and work to prepare the system for mining.
In particular, a datamart for HR is a useful first step. With
proper preparation and consideration, HR databases to-
gether with data mining create an opportunity for organi-
zations to develop their competitive advantage through
using that information for strategic decision-making. A
number of current influences promise to increase the
interest of firms in the application of DM to the Human
Resources area. Trade association and software vendor
activities should also facilitate an increase acceptance
and willingness to adopt DM.
REFERENCES
Bickell, P.J., & Doksum, K. A. (1977). Mathematical sta-
tistics: Basic ideas and selected topics. San Francisco:
Holden Day, Inc.
Brodley, C.E., Lane, T., & Stough, T.M. (1999). Knowledge
discovery and data mining. American Scientist, 87, 54-61.
Bussler, L., & Davis, E. (2002). Information systems: The
quite revolution in human resource management. Journal
of Computer Information Systems, 17-20.
Carlson, L. (2004). Employers offering identity theft
protection. Employee Benefit News, 18(3), 50-51.
Eddy, E.R., Stone, D.L., & Stone-Romero, E.F. (1999). The
effects of information management policies on reac-
tions to human resource information systems: An integra-
266
Data Mining in Human Resources
tion of privacy and procedural justice perspectives. Per-
sonnel Psychology, 52(2), 335-358.
Fayyad, U.M., Piatsky-Shapiro, G., & Smyth, P. (1996a).
From data mining to knowledge discovery in databases.
AI Magazine, (7), 37-54.
Fayyad, U.M., Piatsky-Shapiro, G., Smyth, P., &
Uthurusamy, R. (1996b). Advances in knowledge discov-
ery and data mining. California: American Association
for Artificial Intelligence.
Feelders, A., Daniels, H., & Holsheimer, M. (2000). Meth-
odological and practical aspects of data mining. Informa-
tion & Management, (37), 271-281.
Fisher, C.D., Schoenfeldt, L.F., & Shaw, J.B. (1999). Hu-
man resource management. Boston: Houghton Mifflin
Company.
Hendrickson, A. (2003). Human resource information sys-
tems: Backbone technology of contemporary human
resources. Journal of Labor Research, 24(3) 381-394.
Hubbard, J.C., Forcht, K.A., & Thomas, D.S. (1998). Hu-
man resource information systems: An overview of cur-
rent ethical and legal issues. Journal of Business Ethics,
(17), 1319-1323.
Kerlinger, F. N., & Lee, H. B. (2000). Foundations of
Behavioral Research (3rd ed.). Orlando: Harcourt, Inc.
Ledvinka, J., & Scarpello, V.G. (1992). Federal regulation
of personnel and human resource management. Belmont:
Wadsworth Publishing Company.
Long, L. K., & M.D. Troutt. (2003). Data mining human
resource information systems. In J. Wang (Ed.), Data
mining: Opportunities and challenges (pp. 366-381).
Hershey: Idea Group Publishing.
Osborn, J.L. (1978). Personal information: Privacy at the
workplace. New York: AMACOM.
Patterson, B., & Lindsey, S. (2003). Mining the gold: Gain
competitive advantage through HR data analysis. HR
Magazine, 48(9), 131-136.
SAS Institute Inc. (2001). John Deer harvests HR records
with SAS. Retrieved from http://www.sas.com/news/suc-
cess/johndeere.html
Townsend, A., & Hendrickson, A. (1996). Recasting HRIS
as an information resource. HR Magazine, 41(2), 91-96.
Troutt, M.D., Hu, M., Shanker, M., & Acar, W. (2003).
Frontier versus ordinary regression models for data min-
ing. In P.C. Pendharker (Ed.), Managing data mining
TEAM LinG
Data Mining in Human Resources
technologies in organizations: Techniques and appli-
cations (pp. 21-31). Hershey: Idea Group Publishing.
KEY TERMS
Benchmarking: To identify the Best in Class of
business processes, which might then be implemented
or adapted for use by other businesses.
Enterprise Resource Planning System: An inte-
grated software system processing data from a variety of
functional areas such as finance, operations, sales, hu-
man resources and supply-chain management.
Equal Employment Opportunity Classification: A job
classification system set forth by the Equal Employment
Opportunity Commission (EEOC) for demographic report-
ing requirements. ,
HEDIS : Health Plan Employer Data and Information
Set, a quality assurance system established by the Na-
tional Committee for Quality Assurance (NCQA).
HIPAA: the Health Insurance Portability and Ac-
countability Act of 1996.
Human Resource Information System: An integrated
system used to gather and store information regarding an
organizations employees.
Subject Matter Expert: A person who is knowledge-
able about the skills and abilities required for a specific
domain such as Human Resources.
267
TEAM LinG
268
Data Mining in the Federal Government
Les Pang
National Defense University, USA
INTRODUCTION
Data mining has been a successful approach for improv-
ing the level of business intelligence and knowledge
management throughout an organization. This article
identifies lessons learned from data mining projects
within the federal government including military ser-
vices. These lessons learned were derived from the
following project experiences:
Defense Medical Logistics Support System Data
Warehouse Program
Department of Defense (DoD) Defense Financial
and Accounting Service (DFAS) Operation Mon-
goose
DoD Computerized Executive Information Sys-
tem (CEIS)
Department of Transportation (DOT) Executive
Reporting Framework System
Federal Aviation Administration (FAA) Aircraft
Accident Data Mining Project
General Accounting Office (GAO) Data Mining of
DoD Purchase and Travel Card Programs
U.S. Coast Guard Executive Information System
Veteran Administrations (VA) Demographics System
BACKGROUND
Data mining involves analyzing diverse data sources in
order to identify relationships, trends, deviations and
other relevant information that would be valuable to an
organization. This approach typically examines large
single databases or linked databases that are dispersed
throughout an organization. Pattern recognition tech-
nologies and statistical and mathematical techniques
are often used to perform data mining. By utilizing this
approach, an organization can gain a new level of corpo-
rate knowledge that can be used to address its business
requirements.
Many agencies in the federal government have ap-
plied a data mining strategy with significant success.
This chapter aims to identify the lessons gained as a
result of these many data mining implementations within
the federal sector. Based on a thorough literature re-
view, these lessons were uncovered and selected by the
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
author as being critical factors which led toward the
success of the real-world data mining projects. Also,
some of these lessons reflect novel and imaginative
practices.
MAIN THRUST
Each lesson learned (indicated in boldface) is listed
below. Following each practice is a description of illus-
trative project or projects (indicated in italics), which
support the lesson learned.
Avoid the Privacy Trap
DoD Computerized Executive Information System -
Patients as well as the system developers indicate their
concern for protecting the privacy of individuals their
medical records need safeguards. Any kind of large
database like that where you talk about personal info
raises red flags, said Alex Fowler, a spokesman for the
Electronic Frontier Foundation. There are all kinds of
questions raised about who accesses that info or protects
it and how somebody fixes mistakes (Hamblen, 1998).
Proper security safeguards need to be implemented
to protect the privacy of those in the mined databases.
Vigilant measures are needed to ensure that only autho-
rized individuals have the capability of accessing, view-
ing and analyzing the data. Efforts should also be made
to protect the data through encryption and identity man-
agement controls.
Evidence of the publics high concern for privacy
was the demise of the Pentagons $54 million Terrorist
Information Awareness (originally, Total Information
Awareness) effort the program in which government
computers were to be used to scan an enormous array of
databases for clues and patterns related to criminal or
terrorist activity. To the dismay of privacy advocates,
many government agencies are still mining numerous
databases (General Accounting Office, 2004; Gillmor,
2004). Data mining can be a useful tool for the govern-
ment, but safeguards should be put in place to ensure that
information is not abused, stated the chief privacy
officer for the Department of Homeland Security
(Sullivan, 2004). Congressional concerns on privacy
are so high that the body is looking at introducing
TEAM LinG
Data Mining in the Federal Government
legislation that would require agencies to report to Con-
gress on data mining activities to support homeland
security purposes (Miller, 2004).
Steer Clear of the Guns Drawn
Mentality if Data Mining Unearths a
Discovery
DoD Defense Finance & Accounting Services Opera-
tion Mongoose was a program aimed to discover billing
errors and fraud through data mining. About 2.5 million
financial transactions were searched to locate inaccu-
rate charges. This approach detected data patterns that
might indicate improper use. Examples include pur-
chases made on weekends and holidays, entertainment
expenses, highly frequent purchases, multiple purchases
from a single vendor and other transactions that do not
match with the agencys past purchasing patterns. It
turned up a cluster of 345 cardholders (out of 400,000)
who had made suspicious purchases.
However, the process needs some fine-tuning. As an
example, buying golf equipment appeared suspicious
until it was learned that a manager of a military recre-
ation center had the authority to buy the equipment.
Also, casino-related expense revealed to be a common-
place hotel bill. Nevertheless, the data mining results
have shown sufficient potential that data mining will
become a standard part of the Departments efforts to
curb fraud.
Create a Business Case Based on Case
Histories to Justify Costs
FAA Aircraft Accident Data Mining Project involved
the Federal Aviation Administration hiring MITRE Cor-
poration to identify approaches it can use to mine vol-
umes of aircraft accident data to detect clues about their
causes and how those clues could help avert future
crashes (Bloedorn, 2000). One significant data mining
finding was that planes with instrument displays that can
be viewed without requiring a pilot to look away from
the windshield were damaged a smaller amount in run-
way accidents than planes without this feature.
On the other hand, the government is careful about
committing significant funds to data mining projects.
One of the problems is how do you prove that you kept
the plane from falling out of the sky, said Trish Carbone,
a technology manager at MITRE. It is difficult to justify
data mining costs and relate it to benefits (Matthews,
2000).
One way to justify data mining program is to look at
past successes in data mining. Historically, fraud detec-
tion has been the highest payoff in data mining, but other
areas have also benefited from the approach such as in
sales and marketing in the private sector. Statistics ,
(dollars recovered) from efforts such as this can be used
to support future data mining projects.
Use Data Mining for Supporting
Budgetary Requests
Veterans Administration Demographics System pre-
dicts demographic changes based on patterns among its
3.6 million patients as well as data gathered from insur-
ance companies. Data mining enables the VA to provide
Congress with much more accurate budget requests. The
VA spends approximately $19 billion a year to provide
medical care to veterans. All government agencies such
as the VA are under increasing scrutiny to prove that they
are operating effectively and efficiently. This is particu-
larly true as a driving force behind the Presidents Man-
agement Agenda (Executive Office of the President,
2002). For many, data mining is becoming the tool of
choice to highlight good performance or dig out waste.
United States Coast Guard developed an executive
information system designed for managers to see what
resources are available to them and better understand
the organizations needs. Also, it is also used to identify
relationships between Coast Guard initiatives and sei-
zures of contraband cocaine and establish tradeoffs
between costs of alternative strategies. The Coast Guard
has numerous databases; content overlap each other; and
only one employee understands each database well
enough to extract information. In addition, field offices
are organized geographically but budgets are drawn up
by programs that operate nationwide. Therefore, there is
a disconnect between organizational structure and ap-
propriations structure. The Coast Guard successfully
used their data mining program to overcome these is-
sues (Ferris, 2000).
DOT Executive Reporting Framework (ERF) aims
to provide complete, reliable and timely information in
an environment that allows for the cross-cutting identi-
fication, analysis, discussion and resolution of issues.
The ERF also manages grants and operations data. Grants
data shows the taxes and fees that DOT distributes to the
states highway and bridge construction, airport devel-
opment and transit systems. Operations data covers
payroll, administrative expenses, travel, training and
other operations cost. The ERF system accesses data
from various financial and programmatic systems in use
by operating administrators.
Before 1993 there was no financial analysis system
to compare the departments budget with congressional
appropriations. There was also no system to track per-
formance against the budget or how they were doing
with the budget. The ERF system changed this by track-
269
TEAM LinG

ing of the budget and providing the ability to correct any
areas that have been over planned. Using ERF, adjust-
ments were made within the quarter so the agency did not
go over budget. ERF is being extended to manage budget
projections, development and formulation. It can be used
as a proactive tool, which allows an agency to be more
dynamically to project ahead. The system has improved
financial accountability in the agency (Ferris, 1999).
Give Users Continual Computer-Based
Training
DoD Medical Logistics Support System (DMLSS) built
a data warehouse with a front-end decision support/data
mining tools to help manage the growing costs of health
care, enhance health care delivery, enhance health deliv-
ery in peacetime and promote wartime readiness and
sustainability. DMLSS is responsible for the supply of
medical equipment and medicine worldwide for DoD
medical care facilities. The system received recognition
for reducing the inventory in its medical depot system by
80 percent and reducing supply request response time
from 71 to 15 days (Government Computer News, 2001).
One major challenge faced by the agency was the diffi-
culty in keeping up on the training of users because of the
constant turnover of military personnel. It was deter-
mined that there is a need to provide quality computer-
based training on a continuous basis (Olsen, 1997).
Provide the Right Blend of Technology,
Human Capital Expertise and Data
Security Measures
General Accounting Office used data mining to identify
numerous instances of illegal purchases of goods and
services from restaurants, grocery stores, casinos, toy
stores, clothing retailers, electronics stores, gentlemens
clubs, brothels, auto dealers and gasoline service sta-
tions. This was all part of their effort to audit and inves-
tigate federal government purchase and travel card and
related programs (General Accounting Office, 2003).
Data mining goes beyond using the most effective
technology and tools. There must be well-trained indi-
viduals involved who know about the process, proce-
dures and culture of the system being investigated. They
need to understand the capabilities and limitation of data
mining concepts and tools. In addition, these individual
must recognize the data security issues associated with
the use of large, complex and detailed databases.
270
Data Mining in the Federal Government
FUTURE TRENDS
Despite the privacy concerns, data mining continues to
offer much potential in identifying waste and abuse,
potential terrorist and criminal activity, and identify
clues to improve efficiency and effectiveness within
organizations. This approach will become more perva-
sive because of its integration with online analytical
tools, the improved ease of use in utilizing data mining
tools and the appearance of novel visualization tech-
niques for reporting results. Also, the emergence of a
new branch of data mining called text mining to help
improve the efficiency of searching on the Web. This
approach transforms textual data into a useable format
that facilitates classifying documents, finds explicit
relationships or associations among documents, and
clusters documents into categories (SAS, 2004).
CONCLUSION
These lessons learned may or may not fit in all environ-
ments due to cultural, social and financial consider-
ations. However, the careful review and selection of
relevant lessons learned could result in addressing the
required goals of the organization by improving the
level of corporate knowledge.
A decision maker needs to think outside the box
and move away from the traditional approaches to suc-
cessfully implement and manage their programs. Data
mining poses a challenging but highly effective ap-
proach to improve business intelligence within ones
domain.
REFERENCES
Bloedorn, E. (2000). Data mining for aviation safety.
MITRE Publications.
Executive Office of the President, Office of Manage-
ment and Budget. (2002). Presidents Management
Agenda. Fiscal Year 2002.
Ferris, N. (1999). 9 Hot Trends for 99. Government
Executive.
Ferris, N. (2000). Information is power. Government
Executive.
General Accounting Office. (2003). Data mining:
Results and challenges for government program au-
dits and investigations. GAO-03-591T.
TEAM LinG
Data Mining in the Federal Government
General Accounting Office. (2004). Data mining: Federal
efforts cover a wide range of uses. GAO-04-584.
Gillmor, D. (2004). Data mining by government ram-
pant. eJournal.
Government Computer News. (2001). Ten agencies
honored for innovative projects. Government Com-
puter News.
Hamblen, M. (1998). Pentagon to deploy huge medical
data warehouse. Computer World.
Matthews, W. (2000). Digging Digital Gold. Federal
Computer Week.
Miller, J. (2004). Lawmakers renew push for data-
mining law. Government Computer News.
Olsen, F. (1997). Health record project hits pay dirt.
Government Computer News.
SAS. (2004). SAS Text Miner.
Schwartz, A. (2000). Making the Web safe. Federal
Computer Week.
Sullivan, A. (2004). U.S. Government still data mining.
Reuters.
KEY TERMS
Computer-Based Training: A recent approach in-
volving the use of microcomputer, optical disks such as
compact disks, and/or the Internet to address an
organizations training needs.
Executive Information System: An application
designed for the top executives that often features a
dashboard interface, drill-down capabilities and trend
analysis.
Federal Government: The national government of the
United States, established by the Constitution, which ,
consists of the executive, legislative, and judicial
branches. The head of the executive branch is the Presi-
dent of the United States. The legislative branch con-
sists of the United States Congress, and the Supreme
Court of the United States is the head of the judicial
branch.
Legacy System: Typically, a database management
system in which an organization has invested consider-
able time and money and resides on a mainframe or
minicomputer.
Logistics Support System: A computer package
that assists in the planning and deploying the movement
and maintenance of forces in the military. The package
could deals with the design and development, acquisi-
tion, storage, movement, distribution, maintenance,
evacuation and disposition of material; movement, evacu-
ation, and hospitalization of personnel; acquisition of
construction, maintenance, operation and disposition
of facilities; and acquisition of furnishing of services.
Purchase Cards: Credit cards used in the federal
government used by authorized government official for
small purchases, usually under $2,500.
Travel Cards: Credit cards issued to federal em-
ployees to pay for costs incurred on official business
travel.
NOTE
The views expressed in this article are those of the
author and do not reflect the official policy or position
of the National Defense University, the Department of
Defense or the U.S. Government.
271
TEAM LinG
272
Data Mining in the Soft Computing Paradigm
Pradip Kumar Bala
IBAT, Deemed University, India
Shamik Sural
Indian Institute of Technology, Kharagpur, India
Rabindra Nath Banerjee
Indian Institute of Technology, Kharagpur, India
INTRODUCTION
Data mining is a set of tools, techniques and methods that
can be used to find new, hidden or unexpected patterns
from a large volume of data typically stored in a data
warehouse. Results obtained from data mining help an
organization in more effective individual and group deci-
sion-making. Regardless of the specific technique, data
mining methods can be classified by the function they
perform or by their class of application.
Association rule is a type of data mining that corre-
lates one set of items or events with another set of items
or events. It employs association or linkage analysis,
searching transactions from operational systems for in-
teresting patterns with a high probability of repetition.
Classification techniques include mining processes in-
tended to discover rules that define whether an item or
event belongs to a particular predefined subset or a class
of data. This category of techniques is probably the most
broadly applicable to different types of business prob-
lems. In some cases, it is difficult to define the parameters
of a class of data to be analyzed. When parameters are
elusive, clustering methods can be used to create parti-
tions so that all members of each set are similar according
to a specified set of metrics. Summarization describes a set
of data in compact form. Regression techniques are used
to predict a continuous value. The regression can be
linear or non-linear with one predictor variable or more
than one predictor variables, known as multiple regres-
sion.
Soft computing, which includes application of fuzzy
logic, neural network, rough set and genetic algorithm, is
an emerging area in data mining. By studying combina-
tions of variables and how different combinations affect
data sets, we develop neural network, a non-linear predic-
tive model that learns. Machine learning techniques,
such as genetic algorithms and fuzzy logic, can derive
meaning from complicated and imprecise data. They can
extract patterns from and detect trends within the data that
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
are far too complex to be noticed by either humans or more
conventional automated analysis techniques. Because of
this ability, neural computing and machine learning tech-
nologies demonstrate broad applicability in the world of
data mining and, thus, to a wide variety of complex
business problems. Rough set is the approximation of an
imprecise and uncertain set by pair of precise concepts,
called the lower and upper approximations.
Each soft computing technique addresses problems in
its domain using a distinct methodology. However, they
are not substitute of each other. In fact, these soft com-
puting tools work in a cooperative manner, rather than
being competitive. This has led to the development of
hybridization of soft computing tools for data mining
applications (Mitra et al., 2002). It should, however, be
kept in mind that soft computing techniques have been
traditionally developed to handle small data sets. Extend-
ing the soft computing paradigm for processing large
volumes of data is itself a challenging task. In the next
section, we give a brief background of the various soft
computing techniques.
BACKGROUND
Fuzzy rules offer an attractive trade-off between the need
for accuracy and compactness on one hand, and scalability
on the other, when reasoning systems within a particular
knowledge domain become quite complex. Fuzzy rules
generalize the concept of categorization because, by
definition, the same object can belong to multiple sets
with different degrees of membership. In this sense, fuzzy
logic eliminates the problems associated with borderline
cases: where, for example, a value of degree of member-
ship with 0.9 may cause a rule to fire but a value 0.899 may
not. The net result is that fuzzy systems tend to provide
greater accuracy than traditional rule-based systems when
continuous variables are involved.
Neural network, which draws its inspiration from neu-
roscience, attempts to mirror the way a human brain works
TEAM LinG
Data Mining in the Soft Computing Paradigm
in recognizing patterns by developing mathematical struc-
tures with the ability to learn. An artificial neural network
(ANN) learns through training. These are simple com-
puter-based programs whose primary function is to con-
struct models of a problem space based on trial and error.
The process of training a neural net to associate certain
input patterns with correct output responses involves the
use of repetitive examples and feedback, much like the
training of a human being.
Rough set theory finds application in studying impre-
cision, vagueness, and uncertainty in data analysis and
is based on the establishment of equivalence classes
within a given training data. A rough set gives an approxi-
mation of a vague concept by two precise concepts, called
the lower and upper approximations. These two approxi-
mations are a classification of the domain of interest into
disjoint categories. The lower approximation is a descrip-
tion of the domain objects known with certainty to belong
to the subset of interest and upper approximation is a
description that may possibly belong to the subset.
Genetic algorithms (GAs) are computational models
used in efficient and global search methods for optimality
in problem solving. These search algorithms are based on
the mechanics of natural genetics theory combined with
Darwins theory of survival of the fittest and are par-
ticularly suitable for solving complex optimization prob-
lems as well as applications that require adaptive problem-
solving strategies. In data mining, GA finds application in
hypothesis testing and refinement.
With this background, we next present how soft com-
puting techniques can be applied to specific data mining
problems.
MAIN THRUST
Association Rule Mining: Following are some of the
applications of soft computing tools in association rule
mining.
Fuzzy Logic: A generalized association rule may
involve binary, quantitative or categorical data and
hierarchical relation. In quantitative or categorical
association rule mining, irrespective of the method-
ology used, sharp boundaries remain a problem
which under-estimates or over-emphasizes the ele-
ments near the boundaries. This, may, therefore,
lead to an inaccurate representation of semantics.
To deal with the problem, fuzzy sets and fuzzy items,
usually in the form of labels or linguistic terms, are
used and defined onto the domains (Chien et al.,
2001). In the fuzzy framework, conventional notions
of support and confidence could be extended as
well. The partial belongingness of an item in a
subset is taken into account while computing the
degree of support and the degree of confidence. The ,
measures are similar in spirit to the count operator
used for fuzzy cardinality. Subsequently, with these
extended measures incorporated, several mining
algorithms have been developed (Gyenesei, 2000;
Gyenesei & Teuhola, 2001; Shu et al., 2001). Instead
of dividing quantitative attributes into fixed inter-
vals, linguistic terms can be used to represent the
regularities and exceptions the way in which hu-
mans perceive the reality. Chen et al. (2002) have
developed an algorithm for fuzzy association rules
in dealing with partitioning quantitative data do-
mains. Wei & Chen (1999) extended generalized
association rules with fuzzy taxonomies, by which
partial belongings could be incorporated. Further-
more, a recent effort has been made which incorpo-
rates linguistic hedges on existing fuzzy taxonomies
(Chen et al., 1999; Chen et al., 2002a). Several fuzzy
extensions have been made on interestingness
measures. A measure called Interestingness De-
gree has been proposed which can be seen as the
increase in probability of an event Y caused by the
occurrence of another event X. Attempts have been
made to introduce thresholds for filtering databases
in dealing with very low membership degrees
(Hullermeier, 2001).
Genetic Algorithm: Min et al. (2001) have used a
GA-based data mining approach in e-commerce to
find association rules of IF-THEN form for adopters
and non-adopters of e-purchasing. Association
rules of the form IF-THEN form can also be mined,
which provides a high degree of accuracy and cov-
erage (Lopes et al., 1999).
Clustering: Following are some of the applications
of soft computing tools in clustering.
Fuzzy Logic: A fuzzy clustering algorithm makes an
attempt to group the prospects into categories based
on their identifying characteristics. For example, for
prospective customers of any business, the key
attributes can include geographic data, psycho-
graphic data and others. Clusters expressed in lin-
guistic terms can be easily handled using fuzzy sets.
Using fuzzy sets, we can also find dependencies
between data expressed in qualitative format. Use of
fuzzy logic can help in avoiding searching for less
important, trivial or meaningless patterns in data-
bases. Fuzzy clustering algorithms have been de-
veloped for mining telecommunications customer
and prospect database to gain customer information
for deciding a marketing strategy (Russell et al., 1999).
Neural Network: Self Organizing Map (SOM) is one
of the most widely used unsupervised neural net-
work models that employ competitive learning steps.
273
TEAM LinG

An important data mining task is organizing data
points with single or multiple dimensions in their
natural cluster. Kohonen et al. (2000) has demon-
strated the applicability of self-organizing map where
large data sets can be partitioned in stages. A two-
phase method can be used where in the first phase,
the step-wise strategy of SOM is used, and then in
the second phase, the resulting prototypes of the
SOM are clustered by an agglomerative clustering
method or by k-means clustering (Vesanto et al.,
2000). A dimension-independent algorithm has been
developed which allows hierarchical clustering of
SOMs, based on a spread factor. This can be used as
a controlling measure for generating maps with dif-
ferent dimensionality (Alahakoon et al., 2000).
Classification and Rule Extraction: Following are
some of the applications of soft computing tools in
classification and rule extraction.
Neural Network: For the purpose of classification or
rule extraction, ANN is used in the supervised learn-
ing paradigm. The most common supervised learn-
ing paradigm is error back-propagation. Here, a neu-
ral network receives an input example, generates a
guess, and compares that guess with the expected
result. The error between the guess and the desired
result is fed back to improve the guess in an iterative
manner. In this sense, the ANN is being supervised
by the feedback, which shows the network where it
made the mistakes and how the correct result should
look. The most common form of the back-propaga-
tion algorithm uses a sum-of-squared-errors approach
to generate an aggregate measure of the difference
error. A general framework for classification rule
mining, called NEUCRUM (NEUral Classification RUle
Mining) has been developed. It has two compo-
nents, one is a specific neural classifier named FANNC
and the other is a novel rule extraction approach
named STARE (Zhou et al., 2000).
Rough Set: Piasta (1999) has presented an approach
called the ProbRough system to analyze business
databases based on rule induction. The ProbRough
system can induce decision rules from databases
with a very high number of objects and attributes.
Based on rough set theory, another approach in the
selection of attributes for construction of decision
tree has been developed (Wei, 2003). According to
Han & Kamber (2003), rough set theory can be
applied for classification to discover structural rela-
tionships in imprecise or noisy data. It can be applied
to discrete-valued attributes and hence, continu-
ous-valued attributes must be discretized prior to its
use. A classifier can be trained using rough set
learning algorithm for rule extraction in IF-THEN
form, from a decision table.
274
Data Mining in the Soft Computing Paradigm
Neuro-Fuzzy Computing: Neuro-Fuzzy comput-
ing combines strong features of neural network
and fuzzy approach together. To deal with numeri-
cal and linguistic data and granular knowledge, a
granular neural network can be designed (Zhang et
al., 2000). High-level granular knowledge in the
form of rules is generated by compressing the low-
level granular data. An algorithm has been devel-
oped to mine mixed fuzzy rules involving both
numeric and categorical attributes. Fuzzy set-driven
computational techniques for data mining have
been discussed by Pedrycz (1998), establishing the
relationship between data mining and fuzzy model-
ing.
Other Hybrid Approaches: Hybrid prediction sys-
tem based on neural network with its learning
based on memory-based reasoning can be de-
signed for classification. It can also learn the dy-
namic behavior of a system over a period of time.
It has been established by experimentation that a
hybrid system has a high potential in solving data
mining problems (Shin et al., 2000). The concepts
of fuzzy logic and rough set can be applied in Multi-
layer Perceptron (MLP) neural network to extract
rules from crude domain knowledge. Appropriate
number of the hidden nodes is automatically deter-
mined and the dependency factors are used in the
initial weight encoding.
Other Data Mining Applications: The soft com-
puting paradigm has also been extended to some
other types of data mining as discussed below.
Genetic algorithms have been used in regression
analysis. One basic assumption in traditional regression
models is that there is no interaction amongst the at-
tributes. To learn non-linear multi-regression from a set
of training data, an adaptive GA can be used. A genetic
algorithm can handle attribute interactions efficiently.
GA has been used to discover interesting rules in a
dependency-modeling task (Noda et al., 1999). The sys-
tem developed by Shin et al. (2000) for classification, as
discussed above, can be applied for regression analysis also.
Fuzzy functional dependencies are extensions of clas-
sical functional dependencies, aimed at dealing with
fuzziness in databases and reflecting the semantics that
close values of a collection of certain items are depen-
dent on close values of a collection of a set of different
items. Generally, fuzzy functional dependencies have
different forms depending on the different aspects of
integrating fuzzy logic in classical functional dependen-
cies. Fuzzy inference generalizes both imprecise and
precise inference. An attempt has been made by Yang &
Singhal (2001) to develop a framework of linking fuzzy
TEAM LinG
Data Mining in the Soft Computing Paradigm
functional dependencies and fuzzy association rules in a
closer manner.
Discovering relationships among time series is an
interesting application since time series patterns reflect
the evolution of changes in item values with sequential
factors like time. The value of each time series item is
viewed as a pattern over time, and the similarity between
any two patterns is measured by pattern matching. Chen
et al. (2001) have presented a method based on Dynamic
Time Warping (DTW) to discover pattern associations.
Summarization is one of the major components of data
mining. Lee & Kim (1997) have proposed an interactive
top-down summary discovery process which utilizes fuzzy
ISA hierarchies as domain knowledge. They have defined
a generalized tuple as a representational form of a data-
base summary including fuzzy concepts. By virtue of
fuzzy ISA hierarchies, where fuzzy ISA relationships
common in actual domains are naturally expressed, the
discovery process comes up with more accurate database
summaries. They have also presented an informative-
ness measure for distinguishing generalized tuples that
deliver more information to users, based on Shannons
information theory.
GA-based approach can be used for discovering tem-
poral trends by synthesizing Bayesian Networks
(Novobilski & Kamangar, 2002). Bonaventura et al. (2003)
have developed a hybrid model for the prediction of the
linguistic origin of the surnames. It is a neural network
module combining the results provided both by the lexical
rule module and by the statistical module and used to
compute the evidence for the classes.
FUTURE DIRECTIONS
In this paper, we have discussed various soft computing
methods used in data mining. Our focus has been on four
primary soft computing techniques, namely, fuzzy logic,
neural network, rough set and genetic algorithm. Al-
though these techniques have not yet attained maturity
to the extent the conventional data mining techniques
have, it is expected that these techniques will mature well
enough to be dealt with as independent areas of data
mining very soon.
CONCLUSION
Hybridization of fuzzy, neural and genetic algorithms in
solving data mining problems seems to be an upcoming
area in the field of data mining. However, more stress
needs to be given to the domain of improving the effi-
ciency of these soft computing techniques when applied
to the data mining problems. Processing of tera bytes of
data with a reasonable time response is a primary require-
ment of any kind of data mining algorithm. Since soft ,
computing techniques typically require more processing
than the traditional techniques, it remains to be seen how
well they can be adopted successfully to the interesting
and challenging field of data mining.
REFERENCES
Alahakoon, D., Halgamuge, S.K., & Srinivasan, B. (2000).
Dynamic self-organizing maps with controlled growth for
knowledge discovery. IEEE Transactions on Neural
Networks, 11(3), 601-614.
Bonaventura, P., Marco, G., Marco, M., Franco, S., &
Sheng, J. (2003). A hybrid model for the prediction of the
linguistic origin of surnames. IEEE Transactions on Knowl-
edge and Data Engineering, 15(3), 760-763.
Chen, G.Q., Wei, Q., & Kerre, E.E. (1999). Fuzzy data
mining: Discovery of fuzzy generalized association rules.
In Recent Research Issues on Management of Fuzziness
in Databases. Berlin: Springer-Verlag.
Chen, G.Q., Yan, P., & Kerre, E.E. (2002a). Mining fuzzy
implication-based association rules in quantitative data-
bases. In International FLINS Conference on Computa-
tional Intelligent Systems for Applied Research, Bel-
gium.
Chen, G.Q., Wei, Q., Liu, D., & Wets, G. (2002). Simple
association rules (SAR) and the SAR-based rule discov-
ery. Journal of Computer & Industrial Engineering, 43,
721-733.
Chen, G.Q., Wei, Q., & Zhang, H. (2001). Discovering
similar time-series patterns with fuzzy clustering and
DTW methods. In International Fuzzy Systems Associa-
tion Conference, Vancouver, BA, Canada.
Chien, B.C., Lin, Z.L., & Hong, T.P. (2001). An efficient
clustering algorithm for mining fuzzy quantitative asso-
ciation rules. In Ninth International Fuzzy Systems Asso-
ciation World Congress (pp. 1306-1311), Vancouver,
Canada.
Gyenesei, A. (2000). A fuzzy approach for mining quan-
titative association rules. TUCS Technical Reports 336.
Department of Computer Science, University of Turku,
Finland.
Gyenesei, A., & Teuhola, J. (2001). Interestingness mea-
sures for fuzzy association rules. In Principles and Prac-
tice of Knowledge Discovery in Databases, Freiburg,
Germany.
275
TEAM LinG

Han, J., & Kamber, M. (2003). Data mining: Concepts and
techniques. San Francisco: Morgan Kaufmann.
Hullermeier, E. (2001). Fuzzy association rules: Semantics
issues and quality measures. In Lecture Notes in Com-
puter Science 2206 (pp. 380-391). Berlin & Heidelberg:
Springer.
Kohonen, T., Kaski, S., Lagus, K., Salojarvi, J., Honkela,
J., Paatero, V., & Saarela, A. (2000). Self organization of a
massive document collection. IEEE Transactions on
Neural Networks, 11(3), 574-585.
Lee, D.H., & Kim, M.H. (1997). Database summarization
using fuzzy ISA hierarchies. IEEE Transactions on Sys-
tems, Man and Cybernetics Part B: Cybernetics, 27 (1).
Lopes, C., Pacheco, M., Vellasco, M., & Passos, E. (1999).
Rule-Evolver: An evolutionary approach for data mining.
In Seventh International Workshop on Rough Sets, Fuzzy
Sets, Data Mining and Granular-Soft Computing (pp.
458-462), Yamaguchi, Japan.
Min, H., Smolinski, T., & Boratyn, G.A. (2001). A genetic
algorithm-based data mining approach to profiling the
adopters and non-adopters of e-purchasing. In Third
International Conference on Information Reuse and
Integration, Las Vegas, USA.
Mitra, S., Pal, S.K., & Mitra, P. (2002). Data mining in soft
computing framework: A survey. IEEE Transactions on
Neural Networks, 13, 3-14.
Noda, E., Freitas, A.A., & Lopes, H.S. (1999). Discovering
interesting prediction rules with a genetic algorithm. In
IEEE Congress on Evolutionary Computing (pp. 1322-1329).
Novobilski, A., & Kamangar, F. (2002). A genetic algo-
rithm based approach for discovering temporal trends
using Bayesian networks. In Sixth World Conference on
Systemics, Cybernetics, and Informatics.
Pedrycz, W. (1998). Fuzzy set technology in knowledge
discovery. Fuzzy Sets and Systems, 98, 279-290.
Piasta, Z. (1999). Analyzing business databases with the
ProbRough rule induction system. In Workshop on Data
Mining in Economics, Marketing and Finance (pp. 22-
29), Chania, Greece.
Russell, S., & Lodwick, W. (1999). Fuzzy clustering in data
mining for telco database marketing campaigns. In North
Atlantic Fuzzy Information Processing Symposium (pp.
720-726), New York.
Shin,C.K., Tak Yun, U., Kang Kim, H., & Chan Park, S.
(2000). A hybrid approach of neural network and memory-
based learning to data mining. IEEE Transactions on
Neural Networks, 11(3), 637-646.
276
Data Mining in the Soft Computing Paradigm
Shu, J.Y., Tsang, E.C.C., Daniel, & Yeung, S. (2001). Query
fuzzy association rules in relational database. In Ninth
International Fuzzy Systems Association World Con-
gress, Vancouver, Canada.
Vesanto, J., & Alhoniemi, E. (2000). Clustering of the self-
organizing map. IEEE Transactions on Neural Networks,
11(3), 586-600.
Wei, J.M. (2003). Rough set based approach to selection
of node. International Journal of Computational Cogni-
tion, 1(2).
Wei, Q., & Chen, G.Q. (1999). Mining generalized associa-
tion rules with fuzzy taxonomic structures. In Eighteenth
International Conference of North Atlantic Fuzzy Informa-
tion Processing Systems (pp. 477-481), New York, NY, USA.
Yang, Y., & Singhal, M. (2001). Fuzzy functional depen-
dencies and fuzzy association rules. In First Interna-
tional Conference on Data Warehouse and Knowledge
Discovery (pp. 229-240), Florence, Italy.
Zhang, Y.Q., Fraser, M.D., Gagliano, R.A., & Kandel, A.
(2000). Granular neural networks for numerical-linguistic
data fusion and knowledge discovery. IEEE Transac-
tions on Neural Networks, 11, 658-667.
Zhou, Z.H., Yuan, J., & Chen, S.F. (2000). A general neural
framework for classification rule mining. International
Journal of Computers, Systems, and Signals, 1(2), 154-168.
KEY TERMS
Data Mining: A set of tools, techniques and methods
used to find new, hidden or unexpected patterns from a
large collection of data typically stored in a data ware-
house.
Fuzzy Set: A set that captures the different degrees of
belongingness of different objects in the universe instead
of a sharp demarcation between objects that belong to a
set and those that does not.
Genetic Algorithm: A search and optimization tech-
nique that uses the concept of survival of genetic mate-
rials over various generations of populations much like
the theory of natural evolution.
Hybrid Technique: A combination of two or more soft
computing techniques used for data mining. Examples are
neuro-fuzzy, neuro-genetic, and etcetera.
Neural Network: A connectionist model that can be
trained in supervised or unsupervised mode for learning
patterns in data.
TEAM LinG
Data Mining in the Soft Computing Paradigm

Rough Set: A method of modeling impreciseness and Soft Computing: Collection of methods and tech-
vagueness in data through two sets representing the niques like fuzzy set, neural network, rough set and ,
upper bound and lower bound of the data set. genetic algorithm for solving complex real world prob-
lems.

277

TEAM LinG
278
Data Mining Medical Digital Libraries
Colleen Cunningham
Drexel University, USA
Xiaohua Hu
Drexel University, USA
INTRODUCTION
Given the exponential growth rate of medical data and
the accompanying biomedical literature, more than
10,000 documents per week (Leroy et al., 2003), it has
become increasingly necessary to apply data mining
techniques to medical digital libraries in order to assess
a more complete view of genes, their biological func-
tions and diseases. Data mining techniques, as applied to
digital libraries, are also known as text mining.
BACKGROUND
Text mining is the process of analyzing unstructured
text in order to discover information and knowledge that
are typically difficult to retrieve. In general, text mining
involves three broad areas: Information Retrieval (IR),
Natural Language Processing (NLP) and Information
Extraction (IE). Each of these areas are defined as
follows:
Natural Language Processing: a discipline that
deals with various aspects of automatically pro-
cessing written and spoken language.
Information Retrieval: a discipline that deals
with finding documents that meet a set of specific
requirements.
Information Extraction: a sub-field of NLP that
addresses finding specific entities and facts in
unstructured text.
MAIN THRUST
The current state of text mining in digital libraries is
provided in order to facilitate continued research, which
subsequently can be used to develop large-scale text
mining systems. Specifically, an overview of the pro-
cess, recent research efforts and practical uses of min-
ing digital libraries, future trends and conclusions are
presented.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Text Mining Process
Text mining can be viewed as a modular process that
involves two modules: an information retrieval module
and an information extraction module. presents the
relationship between the modules and the relationships
between the phases within the information retrieval
module. The former module involves using NLP tech-
niques to pre-process the written language and using
techniques for document categorization in order to find
relevant documents. The latter module involves finding
specific and relevant facts within text. NLP consists of
three distinct phases: (1) tokenization, (2) parts of
speech (PoS) tagging and (3) parsing. In the tokenization
step, the text is decomposed into its subparts, which are
subsequently tagged during the second phase with the
part of speech that each token represents (e.g., noun,
verb, adjective, etc.). It should be noted that generating
the rules for PoS tagging is a very manual and labor-
intensive task. Typically, the parsing phase utilizes shal-
low parsing in order to group syntactically related words
together because full parsing is both less efficient (i.e.,
very slow) and less accurate (Shatkay & Feldman, 2003).
Once the documents have been pre-processed, then they
can be categorized.
There are two approaches to document categoriza-
tion: Knowledge Engineering (KE) and Machine Learn-
ing (ML). Knowledge Engineering requires the user to
manually define rules, which can consequently be used
to categorize documents into specific pre-defined cat-
egories. Clearly, one of the drawbacks of KE is the time
that it would take a person (or group of people) to
manually construct and maintain the rules. ML, on the
other hand, uses a set of training documents to learn the
rules for classifying documents. Specific ML tech-
niques that have successfully been used to categorize
text documents include, but are not limited to, Decision
Trees, Artificial Neural Networks, Nearest Neighbor
and Support Vector Machines (SVM) (Stapley et al.,
2002). Once the documents have been categorized, then
documents that satisfy specific search criteria can be
retrieved.
TEAM LinG
Data Mining Medical Digital Libraries

Figure 1. Overview of text mining process

D
Text
Knowledge Enginee ring Boolean
Tokenization Phase Technique Technique

Decomposed, untagged
tokens (i.e. words)
OR OR

Part of Speech Phase

Machine Learning
N A Tokens (i.e. words) tagge d
Technique s Vector
V N with part of speech (e.g.
(e.g. Decision Trees, Technique s
noun, verb, etc.)
Artificial Neural Networks,
Parsing Phase Nearest Neighbor, Support
Vector Machines, etc.)
Adjecti
Verbs
ves Groups of sy ntactically-
related words
Nouns Adverbs

NLP techniques to pre-process Document Categorization Document Retrieval

written languag e Techniques Technique s

Information Retrieval Module

Retrieved Documents

Information Extraction Module

There are several techniques for retrieving documents
that satisfy specific search criteria. The Boolean ap-
proach returns documents that contain the terms (or
phrases) contained in the search criteria; whereas, the
vector approach returns documents based upon the term
frequency-inverse document frequency (TF x IDF) for
the term vectors that represent the documents. Varia-
tions of clustering and clustering ensemble algorithms
(Iliopoulous et al., 2001; Hu, 2004), classification al-
gorithms (Marcotte et al., 2001) and co-occurrence
vectors (Stephens et al., 2001) have been successfully
used to retrieve related documents. An important point
to mention is that the terms that are used to represent the
search criteria as well as the terms used to represent the
documents are critical to successfully and accurately
returning related documents. However, terms often have
multiple meanings (i.e., polysemy) and multiple terms
can have the same meaning (i.e., synonyms). This repre-
sents one of the current issues in text mining, which will
be discussed in the next section.
The last part of the text mining process is informa-
tion extraction, of which the most popular technique is
co-occurrence (Blaschke & Valencia, 2002; Jenssen et
al., 2001). There are two disadvantages to this approach,
each of which creates opportunities for further re-
search. First, this approach depends upon assumptions
regarding sentence structure, entity names, and etcetera
that do not always hold true (Pearson, 2001). Furthermore,
this approach relies heavily on completeness of the list of
gene names and synonyms and summarizes the modular
process of text mining.
Research to Address Issues in Mining
Digital Libraries
The issues in mining digital libraries, specifically medi-
cal digital libraries, include scalability, ambiguous En-
glish and biomedical terms, non-standard terms and
structure and inconsistencies between medical reposi-
tories (Shatkay & Feldman, 2003). Most of the current
text mining research focuses on automating informa-
tion extraction (Shatkay & Feldman, 2003). The
scalability of the text mining approaches is of concern
because of the rapid rate of growth of the literature. As
such, while most of the existing methods have been
applied to relatively small sample sets, there has been an
increase in the number of studies that have been focused
on scaling techniques to apply to large collections
(Pustejovsky et al., 2002; Jenssen et al., 2002). One
exception to this is the study by Jenssen et al. (2001) in
which the authors used a predefined list of genes to
retrieve all related abstracts from PubMed that con-
tained the genes on the predefined list.

279

TEAM LinG

Table 1. General text mining process
General Step General Purpose
Identify and retrieve
IR
relevant docs
Find specific and
relevant facts within
text
[e.g., find specific gene
IE
entities (entity
extraction) &
relationships between
specific genes
(relationship
extraction)]
Since mining digital libraries relies heavily on the abil-
ity to accurately identify terms, the issues of ambiguous
terms, special jargon and the lack of naming conventions
are not trivial. This is particularly true in the case of digital
libraries where the issue is further compounded by non-
standard terms. In fact, a lot of effort has been dedicated
to building ontologies to be used in conjunction with text
mining techniques (Boeckmann et al., 2003; HUGO, 2003;
Liu et al., 2001; NLM, 2003; Oliver et al., 2002; Pruitt &
Maglott, 2001; Pustejovsky et al., 2002). Manually build-
ing and maintaining ontologies, however, is a time con-
suming effort. In light of that, there have been several
efforts to find ways of automatically extracting terms to
incorporate into and build ontologies (Nenadic et al., 2002;
Ono et al., 2001). The ontologies are subsequently used to
match terms. For instance, Nenadic et al. (2002) developed
the Tagged Information Management System (TIMS), which
is an XML-based Knowledge Acquisition system that uses
ontology for information extraction over large collections.
Uses of Text Mining in Medical Digital
Libraries
There are many uses for mining medical digital libraries
that range from generating hypotheses (Srinivasan, 2004)
Table 2. Some uses of text mining medical digital libraries
Application Technique
Building gene networks Co-occurrence
Unsupervised cluster learning and
Discovering protein association vector classification
Discovering gene interactions Co-occurrence
Discovering uses for thalidomide Mapping phrases to UMLS concepts
Extracting and combining relations Rule-based parser and co-occurrence Leroy et al., 2003
Incorporating ontologies (e.g., mapping
Generating hypotheses terms to MeSH)
Identifying biological virus weapons
280
Data Mining Medical Digital Libraries
General Issues General Solutions
aliases,
synonyms &
homonyms
Controlled vocabularies
(ontologies)
aliases,
[e.g., gene terms and
synonyms &
synonyms: LocusLink (Pruitt
homonyms
& Maglott, 2001), SwissProt
(Boeckmann et al., 2003),
HUGO (HUGO, 2003),
National Library of Medicine's
MeSH (NLM, 2003)]
to discovering protein associations (Fu et al., 2003). For
instance, Srinivasan (2004) developed MeSH-based text
mining methods that generate hypotheses by identifying
potentially interesting terms related to specific input.
Further examples include, but are not limited to: uncov-
ering uses for thalidomide (Weeber et al., 2003), discov-
ering functional connections between genes (Chaussabel
& Sher, 2002) and identifying viruses that could be used
as biological weapons (Swanson et al., 2001). He summa-
rizes some of the recent uses of text mining in medical
digital libraries.
FUTURE TRENDS
The large volume of genomic data resulting and the
accompanying literature from the Human Genomic
project is expected to continue to grow. As such, there
will be a continued need for research to develop scal-
able and effective data mining techniques that can be
used to analyze the growing wealth of biomedical data.
Additionally, given the importance of gene names in
the context of mining biomedical literature and the fact
that there are a number of medical sources that use
different naming conventions and structures, research
Supporting Literature
Jenssen et al., 2001; Stapley & Benoit,
2000; Adamic et al., 2002
Fu et al., 2003
Stephens et al., 2001; Chaussabel &
Sher, 2002
Weeber et al., 2001
Srinivasan, 2004; Weeber et al., 2003
Swanson et al., 2001
TEAM LinG
Data Mining Medical Digital Libraries
to further develop ontology will play an important part in
mining medical digital libraries. Finally, it is worth mentioning
that there has been some effort to link the unstructured text
documents within medical digital libraries with their related
structured data in data repositories.
CONCLUSION
Given the practical applications of mining digital librar-
ies and the continued growth of available data, mining
digital libraries will continue to be an important area
that will help researchers and practitioners gain invalu-
able and undiscovered insights into genes, their rela-
tionships, biological functions, diseases and possible
therapeutic treatments.
REFERENCES
Blaschke, C., & Valencia, A. (2002). The frame-based
module of the SUISEKI information extraction system.
IEEE Intelligent Systems, Special Issue on Intelligent
Systems in Biology, 17(2), 14-20.
Boeckmann, B., Bairoch, A., Apweiler, R., Blatter, M.C.,
Estreicher, A., Gasteiger, E., Martin, M.J., Michoud, K.,
ODonovan, C., Phan, I., Pilbout, S., & Schneider, M.
(2003). The SWISS-PROT Protein Knowledgebase and
its Supplement TrEMBL in 2003. Nucleic Acids Re-
search, 31(1), 365-370.
Chaussabel, D., & Sher, A. (2002). Mining microarray
expression data by literature profiling. Genome Biol-
ogy, 3(10), research0055.1-0055.16.
De Bruijn, B., & Martin, J. (2002). Getting to the core of
knowledge: Mining biomedical literature. International
Journal of Medical Informatics, 67, 7-18.
Fu, Y., Mostafa, J., & Seki, K. (2003). Protein association
discovery in biomedical literature. In Proceedings of the
Third ACM/IEEE-CS Joint Conference on Digital Li-
braries (pp.113-115).
Hu, X. (2004). Integration of cluster ensemble and text
summarization for gene expression. In Proceedings of
the 2004 IEEE Symposium of Bioinformatics and
Bioengineering.
HUGO. (2003). HUGO (The Human Genome Organi-
zation) Gene Nomenclature Committee. Retrieved from
http://www.gene.ucl.ac.uk/nomenclature
Iliopolous, I., Enright, A.J., & Ouzounis, C.A. (2001).
Textquest: Document clustering of Medline abstracts for
concept discovery in molecular biology. In Proceedings
of the Pacific Symposium on Biocomputing (PSB) (pp. ,
384-395).
Jenssen, T.K., Laegrid, A., Komorowski, J., & Hovig, E.
(2001). A literature network of human genes for high-
throughput analysis of gene expression. Nature Genet-
ics, 28(1), 21-28.
Leroy, G., Chen, H., Martinez, J.D., Eggers, S., Flasey, R.R.,
Kislin, K.L., Huang, Z., Li, J., Xu, J., McDonald, D.M., &
Ng, G. (2003). Genescene: Biomedical text and data mining.
In Proceedings of the Third ACM/IEEE-CS joint confer-
ence on Digital Libraries (pp. 116-118).
Liu, H. Lussier, Y.A., & Friedman, C. (2001). Diambiguating
ambiguous biomedical terms in biomedical narrative text:
an unsupervised method-abstract. Journal of Biomedi-
cal Informatics, 34(4), 249-261.
Marcotte, E.M, Xenarios, I., & Eisenberg, D. (2001). Min-
ing literature for protein-protein interactions.
Bioinformatics, 17(4), 359-363.
NLM. (2003). MeSH: Medical Subject Headings. Re-
trieved from http://www.nlm.nih.gov/mesh/
Oliver, D.E., Rubin, D.L., Stuart, J.M., Hewett, M., Klein,
T.E., & Altman, R.B. (2002). Ontology development for a
pharmacogenetics knowledge base. Proceedings of Pa-
cific Symposium on Biocomputing (PSB)-2002, 7, 65-76.
Pearson, H. (2001). Biologys name game. Nature,
411(6838), 631-632.
Pruitt, K.D., & Maglott, D.R. (2001). RefSeq and
LocusLink:NCBI gene-centered resources. Nucleic Acids
Research, 29(1), 137-140.
Pustejovsky, J., Castano, J., Zhang, J., Kotecki, M., &
Cochran, B. (2002). Robust relational parsing over bio-
medical literature: Extracting inhibit relations. Pro-
ceedings of Pacific Symposium on Biocomputing (PSB)-
2002, 7, 362-373.
Shatkay, H., & Feldman, R. (2003). Mining the biomedi-
cal literature in the genomic era: An overview. Journal
of Computational Biology, 10(6), 821-855.
Srinivasan, P. (2004). Text Mining: Generating hypoth-
eses from MEDLINE. Journal of the American Society
for Information Science and Technology, 55(5), 396-413.
Stapley, B.J., Kelley, L.A., & Sternberg, M.J. (2002). Predict-
ing the sub-cellular location of proteins from text using
support vector machines. Proceedings of the Pacific Sym-
posium on Biocomputing (PSB), 7, 374-385.
281
TEAM LinG

Stephens, M., Palakal, M., Mukhopadhyay, S., Raje, R., &
Mostafa, J. (2001). Detecting gene relations from Medline
abstracts. In Proceedings of the Pacific Symposium on
Biocomputing (PSB) (pp. 483-496).
Swanson, D.R., Smalheiser, N.R., & Bookstein, A. (2001).
Information discovery from complementary literatures:
Categorizing viruses as potential weapons. Journal of the
American Society for Information Science, 52(10), 797-
812.
Weeber, M., Klein, H., Berg, L., & Vos, R. (2001). Using
concepts in literature-based discovery: Simulating
Swansons Raynaud-Fish Oil and Migraine-Magnesium
discoveries. Journal of the American Society for Informa-
tion Science, 52(7), 548-557.
Weeber, M., Vos, R., Klein, H., de Jong-Van den Berg,
L.T.W., Aronson, A., & Molema, G. (2003). Generating
hypotheses by discovering implicit associations in the
literature: A case report for new potential therapeutic
uses for Thalidomide. Journal of the American Medi-
cal Informatics Association, 10(3), 252-259.
KEY TERMS
Bibliomining: Data mining applied to digital li-
braries to discover patterns in large collections.
282
Data Mining Medical Digital Libraries
Bioinformatics: Data mining applied to medical digi-
tal libraries.
Clustering: An algorithm that takes a dataset and
groups the objects such that objects within the same
cluster have a high similarity to each other, but are
dissimilar to objects in other clusters.
Information Extraction: A sub-field of NLP that ad-
dresses finding specific entities and facts in unstructured
text.
Information Retrieval: A discipline that deals with
finding documents that meet a set of specific require-
ments.
Machine Learning: Artificial intelligence meth-
ods that use a dataset to allow the computer to learn
models that fit the data.
Natural Language Processing: A discipline that
deals with various aspects of automatically processing
written and spoken language.
Supervised Learning: A machine learning tech-
nique that requires a set of training data, which consists
of known inputs and a priori desired outputs (e.g., clas-
sification labels) that can subsequently be used for
either prediction or classification tasks.
Unsupervised Learning: A machine learning tech-
nique, which is used to create a model based upon a
dataset; however, unlike supervised learning, the de-
sired output is not known a priori.
TEAM LinG

Data Mining Methods for Microarray
Data Analysis
Lei Yu
Arizona State University, USA
Huan Liu
Arizona State University, USA
INTRODUCTION
The advent of gene expression microarray technology
enables the simultaneous measurement of expression
levels for thousands or tens of thousands of genes in a
single experiment (Schena, et al., 1995). Analysis of gene
expression microarray data presents unprecedented op-
portunities and challenges for data mining in areas such
as gene clustering (Eisen, et al., 1998; Tamayo, et al., 1999),
sample clustering and class discovery (Alon, et al., 1999;
Golub, et al., 1999), sample class prediction (Golub, et al.,
1999; Wu, et al., 2003), and gene selection (Xing, Jordan,
& Karp, 2001; Yu & Liu, 2004). This article introduces the
basic concepts of gene expression microarray data and
describes relevant data-mining tasks. It briefly reviews
the state-of-the-art methods for each data-mining task
and identifies emerging challenges and future research
directions in microarray data analysis.
BACKGROUND AND MOTIVATION
The rapid advances of gene expression microarray tech-
nology have provided scientists, for the first time, the
opportunity of observing complex relationships among
various genes in a genome by simultaneously measuring
the expression levels of thousands of genes in massive
experiments. In order to extract biologically meaning-
ful insights from a plethora of data generated from
microarray experiments, advanced data analysis tech-
niques are in demand. Data-mining methods, which dis-
cover patterns, statistical or predictive models, and
relationships among massive data, are effective tools
for microarray data analysis.
Gene expression microarrays are silicon chips that
simultaneously measure the expression levels of thou-
sands of genes. The description of technologies for
constructing these chips and measuring gene expres-
sion levels is beyond the scope of this article (refer to
Draghici, 2003, for an introduction). Each expression
level of a specific gene among thousands of genes
measured in an experiment is eventually recorded as a
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
283
,
numerical value. Expression levels of the same set of
genes under study are normally accumulated through
multiple experiments on different samples (or the same
sample under different conditions) and recorded in a
data matrix. In data mining, data are often stored in the
form of a matrix, of which each column is described by
a feature or attribute and each row consists of feature
values and forms an instance, also called a record or
data point, in a multidimensional space defined by the
features. Figure 1 illustrates two ways of representing
microarray data in a matrix form. In Figure 1a, each
feature is a sample (S) and each instance is a gene (G).
Each genes expression levels are measured across all
the samples (or conditions), so fij is the measurement of
the expression level of the ith gene for the jth sample,
where i = 1,..., n and j = 1,..., m. In Figure 1b, the data
matrix is the transpose of the one in Figure 1a, in which
features are genes, and instances are samples. Some-
times, data in Figure 1b may have class labels ci for each
instance, represented in the last column. The class la-
bels can be different types of diseases or phenotypes of
the underlying samples. A typical microarray data set
may contain thousands of genes but only a small number
of samples (often less than 100). The number of samples
is likely to remain small at least for the near future
due to the expense of collecting microarray samples
(Dougherty, 2001).
The two different forms of data shown in Figure 1
have different data-mining tasks. When instances are
genes (Figure 1a), gene clustering can be performed to
find similarly expressed genes across many samples.
When instances are samples (Figure 1b), three different
tasks can be performed: sample clustering, which in-
volves grouping similar samples together to discover
classes or subclasses of samples; sample class predic-
tion, which involves predicting diseases or phenotypes
of novel samples based on patterns learned from train-
ing samples with known class labels; and gene selection,
which involves selecting a small number of genes from
thousands of genes to reduce the dimensionality of the
data and improve the performance of classification and
clustering methods.
TEAM LinG

Figure 1. Microarray data matrix
a b
MAJOR LINES OF RESEARCH AND
DEVELOPMENT
In this part, we briefly review methods for each of the
data-mining tasks identified earlier: gene clustering,
sample clustering, sample class prediction, and gene
selection. We discuss gene clustering and sample clus-
tering together, for these two tasks are common; how-
ever, they are applied on microarray data from different
directions.
Clustering
Clustering is a process of grouping similar samples,
objects, or instances into clusters. Many clustering
methods exist (for a review, see Jain, Murty, & Flynn,
1999; Parson, Ehtesham, & Liu, 2004). They can be
applied to microarray data analysis for clustering genes
or samples. In this article, we present three groups of
frequently used clustering methods.
The first use of hierarchical clustering in gene
clustering is first described in Eisen et al. (1998). Each
instance forms a cluster in the beginning, and the two
most similar clusters are merged until all instances are
in one single cluster. The clustering results in the form
of a tree structure, called dendrogram, which can be
broken at different levels by using domain knowledge.
Tree structures are easy to understand and can reveal
close relationships among resulting clusters, but they
do not provide a unique partition among all the in-
stances, because different ways to determine a basic
level in the dendrogram can result in different cluster-
ing results.
Unlike hierarchical clustering methods, partition-
based clustering methods divide the whole data into a
fixed number of clusters. Examples are K-means (Herwig,
et at., 1999), self-organizing maps (Tamayo, et al., 1999),
and graph-based partitioning (Xu, Y., Olman, & Xu, D.,
284
Data Mining Methods for Microarray Data Analysis
2002). The methods of K-means often require specifica-
tion of the number of clusters, K, and the selection of K
instances as the initial clusters. All instances are then
partitioned into the K clusters, optimizing some objec-
tive function (e.g., inner-cluster similarity) by assign-
ing each instance to the most similar cluster, which is
determined by the distance between the instance and the
mean of each cluster in the current iteration. Self-
organizing maps (SOMs) are variations of K-means
methods and require specification of the initial topol-
ogy of K nodes to construct the map. In graph-based
partitioning methods, a Minimum Spanning Tree (MST)
is often constructed, and the clusters are generated by
deleting the MST edges with the largest lengths. Graph-
based partitioning methods do not heavily depend on the
regularity of the geometric shape of cluster boundaries,
as K-means and SOMs do.
Traditional clustering methods require that each
instance belong to a single cluster, even though some
instances may be only slightly relevant for the biologi-
cal significance of their assigned clusters. Fuzzy C-
means (Dembele & Kastner, 2003) apply a fuzzy parti-
tioning method that assigns cluster membership values
to instances; this process is called fuzzy clustering. It
links each instance to all clusters via a real-value vector
of indexes. The value of each index lies between 0 and 1,
where a value close to 1 indicates a strong association to
the corresponding cluster, while a value close to 0
indicates no association. The vector of indexes thus
defines the membership of an instance with respect to
the various clusters.
Sample Class Prediction
Apart from clustering methods, which do not require a
priori knowledge about the classes of available instances,
a classification method requires training instances with
labeled classes, leans patterns that discriminate be-
TEAM LinG
Data Mining Methods for Microarray Data Analysis
tween various classes, and ideally, correctly predicts the
classes of unseen instances. Many classification meth-
ods can be applied to predict diseases or phenotypes of
novel samples from microarray data. We present four
commonly used methods in this section.
Linear discriminative analysis (LDA) For an m x n
gene expression matrix X (m is the number of samples, and
n is the number of genes), Linear discriminative analysis
(LDA) seeks linear combinations, xa, of sample vectors xi
= (xi1,,xin) with large ratios of between-class to within-
class sums of squares. In other words, it tries to maximize
the ratio aTBa/aTWa, where B and W denote, respectively,
the n x n matrices of between-class and within-class sums
of squares (Dudoit, Fridlyand, & Speed, 2000).
Nearest neighbor (NN) usually does not learn during
the training phase. Only when it is required to classify a
new sample does NN search the data to find the nearest
neighbor for the new sample, using the class label of the
nearest neighbor to predict the class label of the new
sample. K-NN makes the prediction for a new sample
based on the most common class label among the K
training samples most similar to the new sample. Ex-
amples can be found in Pomeroy, et. al, 2002).
Decision trees classify samples by building a tree-
like structure. Specifically, they recursively split samples
into two child branches based on the values of a selected
feature, starting with all the samples. Each leaf node of
the tree is pure in terms of classes, and the resulting
partition corresponds to a classifier. By limiting the
number of consecutive branches, they can produce more
generalized classifiers. Different forms of trees exist.
In Wu et al. (2003), classification and regression trees
are applied for sample classification.
Support vector machines (SVMs) have also been
shown effective in sample classification (Brown, et al.,
2000). They try to separate a set of training samples of two
different classes with a hyperplane in an n-dimensional
space defined by n features (genes). If no separating
hyperplane exists in the original space, a kernel function
is used to map the samples into a higher dimensional
space where a separating hyperplane exists. Complex
kernel functions that provide nonlinear mappings result
in nonlinear classifiers. SVMs avoid overfitting by se-
lecting a hyperplane that is maximally distant from the
training samples of two different classes, called maxi-
mum margin separating hyperplane, from among many
hyperplanes that can separate the two classes.
Gene Selection
The nature of relatively high dimensionality but a small
sample size of data in sample classification and cluster-
ing can cause the problem of curse of dimensionality
and overfitting of the training data (Dougherty, 2001).
Therefore, selecting a small number of discriminative
genes from thousands of genes is essential for suc- ,
cessful sample classification and clustering. Feature
selection methods (for a review, see Blum & Langley,
1997; Liu & Motoda, 1998) can be applied in microarray
data analysis for gene selection.
Among gene selection methods, earlier methods
often evaluate genes in isolation without considering
gene-to-gene correlation. They rank genes according
to their individual relevance or discriminative power to
the targeted classes and select top-ranked genes. Some
methods based on statistical tests or information gain
have been employed in Golub et al. (1999) and Model,
(2001). However, a number of studies (Ding & Peng,
2003; Xion, Fang, & Zhao, 2001) point out that simply
combining a highly ranked gene with another highly
ranked gene often does not form a good gene set,
because some highly correlated genes could be redun-
dant. Removing redundant genes among selected ones
can achieve a better representation of the characteris-
tics of the targeted classes and lead to improved clas-
sification accuracy. Methods that handle gene redun-
dancy based on pair-wise correlation analysis among
genes can be found in Ding and Peng (2003); Xing,
Jordan, and Karp (2001); and Yu and Liu (2004). A gene
selection method for unlabeled samples is also pro-
posed and is shown effective for sample clustering in
Xing and Karp (2001).
FUTURE TRENDS
Traditional data-mining methods are often designed
for data where the number of instances is significantly
larger than the number of features. In microarray data
analysis, however, the number of features (genes) is
huge, and the number of instances (samples) is rela-
tively small, for tasks of sample clustering or classifi-
cation. This unique characteristic of microarray data
presents a challenge to the scalability of current data-
mining methods to high dimensionality. In addition, the
relative shortage of instances in the context of high
dimensionality often causes many methods to overfit
the training data. Therefore, besides improving current
data-mining methods, substantial research efforts are
needed to come up with new methods specifically
designed for microarray data.
CONCLUSION
Gene expression microarrays are a revolutionary tech-
nology with great potential to provide accurate medical
diagnostics, develop cures for diseases, and produce a
285
TEAM LinG

detailed genome-wide molecular portrait of cellular states
(Piatetsky-Shapiro & Tamayo, 2003). Data-mining meth-
ods are effective tools to turn massive raw data from
microarray experiments into biologically important in-
sights. In this article, we provide a brief introduction to
microarray data analysis and a concise review of various
data-mining methods for microarray data.
REFERENCES
Alon, U. (1999). Broad patterns of gene expression re-
vealed by clustering analysis of tumor and normal colon
tissues probed by oligonucleotide arrays. Proceedings
of the National Acad. Sci., 96 (pp. 6745-6750), USA.
Blum, A. L., & Langley, P. (1997). Selection of relevant
features and examples in machine learning. Artificial
Intelligence, 97, 245-271.
Brown, M. et al. (2000). Knowledge-based analysis of
microarray gene expression data by using support vec-
tor machines. Proceedings of the National Acad. Sci., 97
(pp. 262-267), USA.
Dembele, D., & Kastner, P. (2003). Fuzzy C-means method
for clustering microarray data. Bioinformatics, 19(8), 973-
980.
Ding, C., & Peng, H. (2003). Minimum redundancy feature
selection from microarray gene expression data. Proceed-
ings of the Computational Systems Bioinformatics Con-
ference (pp. 523-529).
Dougherty, E. R. (2001). Small sample issue for microarray-
based classification. Functional Genomics, 2, 28-34.
Draghici, S. (2003). Data analysis tools for DNA
microarrays. Chapman & Hall/CRC.
Dudoit, S., Fridlyand, J., & Speed, T. P. (2000). Com-
parison of discrimination methods for the classifica-
tion of tumors using gene expression data (Tech. Rep.
No. 576). Berkeley, CA: University of California at
Berkeley, Department of Statistics.
Eisen, M. (1998). Clustering analysis and display of
genome-wide expression patterns. Proceedings of the
National Acad. Sci., 95 (pp. 14863-14868), USA.
Golub, T. (1999). Molecular classification of cancer: Class
discovery and class prediction by gene expression moni-
toring. Science, 286, 531-537.
Herwig, R., (1999). Large-scale clustering of cDNA finger-
prints. Genomics, 66, 249-256.
286
Data Mining Methods for Microarray Data Analysis
Jain, A. K., Murty, M. N., & Flynn, P. J. (1999). Data
clustering: A review. ACM Computing Surveys, 31(3),
264-323.
Liu, H., & Motoda, H. (1998). Feature selection for knowl-
edge discovery and data mining. Boston: Kluwer Aca-
demic.
Model, F. (2001). Feature selection for DNA methylation
based cancer classification. Bioinformatics, 17, 154-164.
Parson, L., Ehtesham, H., & Liu, H. (2004). Subspace
clustering for high dimensional data: A review. ACM
SIGKDD Explorations, 6(1), 90-105.
Piatetsky-Shapiro, G., & Tamayo, P. (2003). Microarray
data mining: Facing the challenges. SIGKDD Explora-
tions, 5(2), 1-5.
Pomeroy, S. L. (2002). Prediction of central nervous sys-
tem embryonal tumor outcome based on gene expression.
Nature, 415, 436-442.
Schena, M. (1995). Quantitative monitoring of gene ex-
pression patterns with a complementary DNA microarray.
Science, 270, 467-470.
Tamayo, P. (1999). Interpreting patterns of gene expres-
sion with self-organizing maps: Methods and application
to hematopoietic differentiation. Proceedings of the Na-
tional Acad. Sci., 96 (pp. 2907-2912), USA.
Wu, B., (2003). Comparison of statistical methods for
classification of ovarian cancer using mass spectrom-
etry data. Bioinformatics, 19, 1636-1643.
Xing, E., Jordan, M., & Karp, R. (2001). Feature selection
for high-dimensional genomic microarray data. Proceed-
ings of the 18th International Conference on Machine
Learning (pp. 601-608).
Xing, E., & Karp, R. (2001). CLIFF: Clustering of high-
dimensional microarray data via iterative feature filtering
using normalized cuts. Bioinformatics, 17, S306-S315.
Xion, M., Fang, Z. & Zhao, J. (2001). Biomarker identifica-
tion by feature wrappers. Genome Research, 11, 1878-
1887.
Xu, Y., Olman, V., & Xu, D. (2002). Clustering gene expres-
sion data using a graph-theoretic approach: An applica-
tion of minimum spanning trees. Bioinformatics, 18(4),
536-545.
Yu, L., & Liu, H. (2004). Redundancy based feature selec-
tion for microarray data. Proceedings of the 10th ACM
SIGKDD Conference on Knowledge Discovery and Data
Mining.
TEAM LinG
Data Mining Methods for Microarray Data Analysis
KEY TERMS
Classification: The process of predicting the classes
of unseen instances based on patterns learned from
available instances with predefined classes.
Clustering: The process of grouping instances into
clusters so that instances are similar to one another
within a cluster but dissimilar to instances in other
clusters.
Data Mining: The application of analytical methods
and tools to data for the purpose of discovering patterns,
statistical or predictive models, and relationships among
massive data.
Feature Selection: A process of choosing an optimal
subset of features from original features, according to a ,
certain criterion.
Gene: A hereditary unit consisting of a sequence of
DNA that contains all the information necessary to
produce a molecule that performs some biological func-
tion.
Gene Expression Microarrays: Silicon chips that
simultaneously measure the expression levels of thou-
sands of genes.
Genome: All the genetic information or hereditary
material possessed by an organism.
287
TEAM LinG
288
Data Mining with Cubegrades
Amin A. Abdulghani
Quantiva, USA
INTRODUCTION
Much interest has been expressed in database mining
by using association rules (Agrawal, Imielinski, & Swami,
1993). In this article, I provide a different view of the
association rules, which are referred to as cubegrades
(Imielinski, Khachiyan, & Abdulghani, 2002) .
An example of a typical association rule states that,
say, in 23% of supermarket transactions (so-called market
basket data) customers who buy bread and butter also
buy cereal (that percentage is called confidence) and that
in 10% of all transactions, customers buy bread and butter
(this percentage is called support). Bread and butter
represent the body of the rule, and cereal constitutes the
consequent of the rule. This statement is typically repre-
sented as a probabilistic rule. But association rules can
also be viewed as statements about how the cell repre-
senting the body of the rule is affected by specializing it
with the addition of an extra constraint expressed by the
rules consequent. Indeed, the confidence of an associa-
tion rule can be viewed as the ratio of the support drop,
when the cell corresponding to the body of a rule (in this
case, the cell of transactions including bread and butter)
is augmented with its consequent (in this case, cereal).
This interpretation gives association rules a dynamic
flavor reflected in a hypothetical change of support af-
fected by specializing the body cell to a cell whose
description is a union of body and consequent descrip-
tors. For example, the earlier association rule can be
interpreted as saying that the count of transactions in-
cluding bread and butter drops to 23% of the original
when restricted (rolled down) to the transactions includ-
ing bread, butter, and cereal. In other words, this rule
states how the count of transactions supporting buyers
of bread and butter is affected by buying cereal as well.
With such interpretation in mind, a much more general
view of association rules can be taken, when support
(count) can be replaced by an arbitrary measure or aggre-
gate, and the specialization operation can be substituted
with a different delta operation. Cubegrades capture
this generalization. Conceptually, this is very similar to
the notion of gradients used in calculus. By definition, the
gradient of a function between the domain points x1 and
x2 measures the ratio of the delta change in the function
value over the delta change between the points. For a
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
given point x and function f(), it can be interpreted as a
statement of how a change in the value of x (x) affects a
change in value in the function ( f(x)).
From another viewpoint, cubegrades can also be con-
sidered as defining a primitive for cubes. An n-dimen-
sional cube is a group of k-dimensional (k<=n) cuboids
arranged by the dimensions of the data. A cell represents
an association of a measure m (e.g., total sales) with a
member of every dimension. The scope of interest in
Online Analytical Processing (OLAP) is to evaluate one
or more measure values of the cells in the cube. Cubegrades
allow a broader, more dynamic view. In addition to evalu-
ating the measure values in a cell, they evaluate how the
measure values change or are affected in response to a
change in the dimensions of a cell. Traditionally, OLAP
has had operators such as drill downs, rollups defined,
but the cubegrade operator differs from them as it returns
a value measuring the effect of the operation. Additional
operators have been proposed to evaluate/measure cell
interestingness (Sarawagi, 2000; Sarawagi, Agrawal, &
Megiddo, 1998). For example, Sarawagi et al. computes
anticipated value for a cell by using the neighborhood
values, and a cell is considered an exception if its value is
significantly different from its anticipated value. The
difference is that cubegrades perform a direct cell-to-cell
comparison.
BACKGROUND
An association or propositional rule can be defined in
terms of cube cells. It can be defined as a quadruple (body,
consequent, support, confidence) where body and conse-
quent are cells over disjoint sets of attributes, support is
the number of records satisfying the body, and confi-
dence is the ratio of the number of records that satisfy the
body and the consequent to the number of records that
satisfy just the body. You can also consider an associa-
tion rule as a statement about a relative change of
measure, COUNT, when specializing or drilling down the
cell denoted by the body to the cell denoted by the body
+ consequent. The confidence of the rule measures how
the consequent affects the support when drilling down
the body. These association rules can be generalized in
two ways:
TEAM LinG
Data Mining with Cubegrades

By allowing relative changes in other measures, Figure 1. Cubegrade: specialization, generalization,

instead of just confidence, to be returned as part of and mutation ,
the rule.
By allowing cell modifications to be able to occur in A=a1, B=b1 [Count=150, Avg
different directions instead of just specializations (M1)=25]
(or drill-downs).

These generalized cell modifications are denoted as Generalization

cubegrades. A cubegrade expresses how a change in the

structure of a given cell affects a set of predefined mea- Mutation
sures. The original cell being modified is referred to as the A=a1, B=b1, C=c1 A=a1, B=b1, C=c2
source, and the modified cell as target. [Count=100, Avg(M1)=20] [Count=70, Avg(M1)=30]

More formally, a cubegrade is a 5-tuple (source, tar-

get, measures, value, delta-value) where
source and target are cube cells
measures is the set of measures that are evaluated
both in the source as well as in the target
value is a function, value: measures R, that
evaluates measure m measures in the source
delta-value is also a function, delta-value: mea-
sures R, that computes the ratio of the value of m
measures in the target versus the source
A cubegrade can visually be represented as a rule
form:
Source target, [measures, value, delta-value]
Define a descriptor to be an attribute value pair of the
form dimension=value if the dimension is a discrete
attribute, or dimension = [lo, hi] if the attribute is a
dimension attribute. The cubegrades are distinguished as
three types:
Specializations: A cubegrade is a specialization if
the set of descriptors of the target are a superset of
those in the source. Within the context of OLAP, the
target cell is termed a drill-down of source.
Generalizations: A cubegrade is a generalization
if the set of descriptors of the target cell are a subset
of those in the source. Here, in OLAP, the target cell
is termed a roll-up of source.
Mutations: A cubegrade is a mutation if the target
and source cells have the same set of attributes but
differ on the descriptor values (they are union
compatible, so to speak, as the term has been used
in relational algebra).
Figure 1 illustrates the operations of these cubegrades.
Following, I illustrate some specific examples to explain
the use of these cubegrades:
Specialization
A=a1, B=b1, C=c1, D=d1
[Count=50, Avg(M1)=18]
(Specialization Cubegrade). The average age of
buyers who purchase $20 to $30 worth of milk
monthly drops by 10% among buyers who also buy
cereal.
(salesMilk=[$20,$30]) (salesMilk=[$20,$30],
salesCereal=[$1,$5])
[AVG(Age), AVG(Age) = 23, DeltaAVG(Age) = 90%]
(Mutation Cubegrade). The average amount spent
on milk drops by 30% when moving from suburban
buyers to urban buyers.
(areaType=suburban) (areaType=urban)
[ AVG(salesMilk), AVG(salesMilk) = $12.40,
DeltaAVG(salesMilk)= 70%]
MAIN THRUST
Similar to association rules (Agrawal & Srikant, 1994), the
generation of cubegrades can be divided into two phases:
(a) generation of significant cells (rather than frequent
sets) satisfying the source cell conditions and (b) compu-
tation of cubegrades from the source (instead of comput-
ing association rules from frequent sets) satisfying the
joint conditions between source and target and target
conditions.
The first task is similar to the computation of iceberg
cube queries (Beyer & Ramakrishnan, 1999; Han, Pei,
Dong, & Wang, 2001; Xin, Han, Li, & Wah, 2003). The
fundamental property that allows for pruning in these
computations is called monotonicity of the query: Let D

289

TEAM LinG

be a database and XD be a cell. A query is monotonic at
X if the condition Q(X) is FALSE implies that Q(X) is
FALSE for any X`X.
However, as described by Imielinski et al. (2002), deter-
mining whether a query Q is monotonic in terms of this
definition is an NP-hard problem for many simple classes
of queries. To work around this problem, the authors
introduced another notion of monotonicity, referred to as
view monotonicity of a query. Suppose you have cuboids
defined on a set S of dimension and measures. A view V on
S is an assignment of values to the elements of the set. If
the assignment holds for the dimensions and measures in
a given cell X, then V is a view for X on the set S. So, for
example, if in a cell of rural buyers, the average sales of
bread for 20 buyers is 15, then the view on the set {areaType,
COUNT(), AVG(salesBread)} for the cell is
{areaType=rural, COUNT()=20, AVG(salesBread)=15}.
Extending the definition, a view on a query is an assign-
ment of values for the set of dimension and measure
attributes of the query expression. A query Q()view mono-
tonic on view V if for any cell X in any database D such that
V is the view for X, the condition Q is FALSE, for X implies
Q is FALSE for all X'X. An important property of view
monotonicity is that the time and space required for check-
ing it for a query depends on the number of terms in the
query, not on the size of the database or the number of its
attributes. Because most of the queries typically have few
terms, it would be useful in many practical situations. The
method presented can be used for checking for view
monotonicity for queries that include constraints of type
(Agg {<, >, =, !=} c), where c is a constant and Agg can be
MIN, SUM, MAX, AVERAGE, COUNT, an aggregate that
is a higher order moment about the origin, or an aggregate
that is an integral of a function on a single attribute.
Consider a hypothetical query asking for cubes with
1000 or more buyers and with total milk sales less than
$50,000. In addition, the average milk sales per customer
should be between $20 and $50, with maximum sales greater
than $75. This query can be expressed as follows:
COUNT(*)>=1000 and AVG(salesMilk)>=20 and
AVG(salesMilk)<50 and
MAX(salesMilk)>=75 and SUM(saleMilk)<50K
Suppose, while performing bottom-up cube computa-
tion, you have a cell C with the following view V (Count
=1200; AVG(salesMilk) = 50; MAX(salesMilk) = 80;
MIN(salesMilk) = 30; SUM(salesMilk) =60000). Using the
method for checking view monotonicity, it can be shown
that some cell C of C can exist (though this subcell is not
guaranteed to exist in this database) with 1000<= count
<1075 for which this query can be satisfied. Thus, this
query cannot be pruned on the cell.
290
Data Mining with Cubegrades
However, if the view for C is (Count = 1200;
AVG(salesMilk) = 57; MAX(salesMilk) =80;
MIN(salesMilk) = 30; SUM(salesMilk) = 68400), then it
can be shown that there cannot exist any subcell C of C
in any database for which the original query can be
satisfied. Thus, the query can be pruned on cell C.
After the source cells have been computed, the next
task is to compute the set of target cells. This is done by
performing a set of query conversions which make it
possible to reduce cubegrade query evaluation to ice-
berg queries. Given a specific candidate source cell C,
define Q[C] as the query which results from Q by source
substitution. Here, Q is transformed by substituting into
its where clause all the values of the measures, and the
descriptors of the source cell C as well, by performing the
delta elimination step, which replaces all the relative
delta-values (expressed as fractions) by the regular less
thangreater than conditions. This is possible because
the values for the measures of C are known. With this, the
delta-values can now be expressed as conditions on
values of the target. For example, if AVG(Salary)=40K in
cell C and the condition on the DeltaAVG(Salary) is of the
form DeltaAVG(Salary) >1.10, this can be translated now
to AVG(Salary) >44K, where AVG(Salary) references the
target cell. The final step in source substitution is join
transformation, where the join conditions (specializa-
tions, generalizations, and mutations) in Q are trans-
formed into the target conditions because the source cell
is known. Notice, thus, that Q[C] is the cube query
specifying the target cell.
Dong, Han, Lam, Pei, and Wang (2001) present an
optimized version of target generation that is particularly
useful for the case where the number of source cells are
few in number. The ideas in the algorithm include the
following steps:
Perform for the set of identified source cells the
lowest common delta elimination such that the
resulting target condition does not exclude any
possible target cells.
Perform a bottom-up iceberg query for the target
cells based on the target condition. Define
LiveSet(T) of a target cell T as the candidate set of
source cells that can possibly match or join with the
target. A target cell, T, may identify a source, S, in
its LiveSet to be prunable based on its monotonic-
ity and thus removable from its LiveSet. In such a
case, all descendants of T would also not include
S in their LiveSets.
Perform a join of the target and each of its LiveSets
source cells and for the resulting cubegrade; check
whether it satisfies the join criteria and the delta-
value condition for the query.
TEAM LinG
Data Mining with Cubegrades
In a typical scenario, it is not expected that users
would be asking for cubegrades per se. Rather, it may be
more likely that they pose a query on how a given delta
change affects a set of cells, which cells are affected by
a given delta change, or what delta changes affect a set of
cells in a prespecified manner.
Further, more complicated sets of applications can be
implemented by using cubegrades (Abdulghani, 2001).
For example, one may be interested in finding cells that
remain stable and are not significantly affected by gener-
alization, specialization, or mutation. An illustration for
such a situation would be to find cells that remain stable
on the blood pressure measure and are not affected by a
different specialization on age or area demographics.
Another application could be to find effective factors, a
set of specialization, generalization, or mutation descrip-
tors that are effective in changing a measure value m by
a significant ratio. For example, one may want to find
effective factors in decreasing a cholesterol level across
a set of selected cells.
FUTURE TRENDS
A major challenge for cubegrade processing is its compu-
tational complexity. Potentially, an exponential number of
source/target cells can be generated. A positive develop-
ment in this direction is the work done on Quotient Cubes
(Lakshmanan, Pei, & Han, 2002). This work provides a
method for partitioning and compressing the cells of a
data cube into equivalent classes such that the resulting
classes have cells covering the same set of tuples and
preserving the cubes semantic roll-up/drill-down. In this
context, we can reduce the number of cells generated for
the source and target. Further, the pairings for the
cubegrade can be reduced by restricting the source and
target to different classes.
Another related challenge for cubegrades is to iden-
tify the set of interesting cubegrades (cubegrades that are
somewhat surprising). Insights to this problem can be
obtained from similar work done in the context of associa-
tion rules (Bayardo & Agrawal, 1999; Liu, Ma, &Yu, 2001).
The main difference being that for cubegrades, measures
(possibly a combination of them) other than COUNT are
involved, but for association rules, the interesting func-
tions are based on the count function. In addition,
cubegrades have cell modification in multiple directions,
but association rules are restricted to specializations.
As cubegrades are better understood, wider applica-
tions of the concept to various domains are expected to
be seen. Association rules have been applied with suc-
cess to such areas as intrusion detection (Lee, Stolfo, &
Mok, 1998), and microarray data (Tuzhilin & Adomavicius,
2002). Applying the cubegrade paradigm in such domains
provides an opportunity for richer mining results. ,
CONCLUSION
In this article, I look at a generalization of association rules
referred to as cubegrades. These generalizations include
allowing the evaluation of relative changes in other mea-
sures, rather than just confidence, to be returned as well
as allowing cell modifications to occur in different direc-
tions. The additional directions that I consider here in-
clude generalizations, which modify cells towards the
more general cell with fewer descriptors, and mutations,
which modify the descriptors of a subset of the attributes
in the original cell definition with the others remaining the
same. The paradigm allows you to ask queries that were
not possible through association rules. The downside is
that it comes with the price of relatively increased compu-
tation/storage costs that need to be tackled with innova-
tive methods.
REFERENCES
Abdulghani, A. (2001). Cubegrades-generalization of
association rules to mine large datasets. Doctoral disser-
tation, Rutgers University, New Brunswick, NJ. Disserta-
tion Abstracts International, DAI-B 62/10, UMI Number
3027950.
Agrawal, R., Imielinski, T., & Swami, A. N. (1993). Mining
association rules between sets of items in large data-
bases. Proceedings of the ACM SIGMOD Conference (pp.
207-216), USA.
Agrawal, R., & Srikant, R. (1994). Fast algorithms for
mining association rules in large databases. Proceedings
of the International Conference on Very Large Data
Bases (pp. 487-499), Chile.
Bayardo, R., & Agrawal, R. (1999). Mining the most
interesting rules. Proceedings of the ACM International
Conference on Knowledge Discovery and Data Mining
(pp. 145-154), USA.
Beyer, K. S., & Ramakrishnan, R. (1999). Bottom-up com-
putation of sparse and iceberg CUBEs. Proceedings of
the ACM SIGMOD Conference (pp. 359-370), USA.
Dong, G., Han, J., Lam, J. M., Pei, J., & Wang, K. (2001).
Mining multi-dimensional constrained gradients in data
cubes. Proceedings of the International Conference on
Very Large Data Bases (pp. 321-330), Italy.
291
TEAM LinG

Gray, J., Bosworth, A., Layman, A., & Pirahesh, H. (1996).
Data cube: A relational aggregation operator generalizing
group-by, cross-tab, and sub-total. Proceedings of the
International Conference on Data Engineering (pp. 152-
159), USA.
Han, J., Pei, J., Dong, G., & Wang, K. (2001). Efficient
computation of iceberg cubes with complex measures.
Proceedings of the ACM SIGMOD Conference (pp. 1-12),
USA.
Imielinski, T., Khachiyan, L., & Abdulghani, A. (2002).
Cubegrades: Generalizing association rules. Journal of
Data Mining and Knowledge Discovery, 6(3), 219-257.
Lakshmanan, L. V. S., Pei, J., & Han, J. (2002). Quotient
cube: How to summarize the semantics of a data cube.
Proceedings of the International Conference on Very
Large Data Bases (pp. 778-789), China.
Lee, W., Stolfo, S. J., & Mok, K. W. (1998). Mining audit
data to build intrusion detection models. Proceedings of
the ACM International Conference on Knowledge Dis-
covery and Data Mining (pp. 66-72), USA.
Liu, B., Ma, Y., & Yu, S. P. (2001). Discovering unexpected
information from your competitors Web sites. Proceed-
ings of the ACM International Conference on Knowl-
edge Discovery and Data Mining (pp. 144-153), USA.
Sarawagi, S. (2000). User-adaptive exploration of multidi-
mensional data. Proceedings of the International Con-
ference on Very Large Data Bases (pp. 307-316).
Sarawagi, S., Agrawal, R., & Megiddo, N.(1998). Discov-
ery driven exploration of OLAP data cubes. Proceedings
of the International Conference on Extending Database
Technology (pp. 168-182), Spain.
Tuzhilin, A., & Adomavicius, G. (2002). Handling very
large numbers of association rules in the analysis of
microarray data. Proceedings of the ACM International
Conference on Knowledge Discovery and Data Mining
(pp. 396-404), Canada.
Xin, D., Han, J., Li, X., & Wah, B. W.(2003). Star-cubing:
Computing iceberg cubes by top-down and bottom-up
integration. Proceedings of the International Confer-
ence on Very Large Data Bases (pp. 476-487), Germany.
292
Data Mining with Cubegrades
KEY TERMS
Cubegrade: A cubegrade is a 5-tuple (source, target,
measures, value, delta-value) where
source and target are cells
measures is the set of measures that are evaluated
both in the source as well as in the target
value is a function, value: measures R, that
evaluates measure m measures in the source
delta-value is a function, delta-value: measures
R, that computes the ratio of the value of m
measures in the target versus the source
Drill-Down: A cube operation that allows users to
navigate from summarized cells to more detailed cells.
Generalizations: A cubegrade is a generalization if
the set of descriptors of the target cell are a subset of the
set of attribute-value pairs of the source cell.
Iceberg Cubes: The set of cells in a cube that satisfies
an iceberg query.
Iceberg Query: A query on top of a cube that asks for
aggregates above a certain threshold.
Mutations: A cubegrade is a mutation if the target and
source cells have the same set of attributes but differ on
the values.
Query Monotonicity: () is monotonic at a cell X if the
condition Q(X) is FALSE implies that Q(X) is FALSE for
any cell X`X.
Roll-Up: A cube operation that allows users to aggre-
gate from detailed cells to summarized cells.
Specialization: A cubegrade is a specialization if the
set of attribute-value pairs of the target cell is a superset
of the set of attribute-value pairs of the source cell.
View: A view V on S is an assignment of values to the
elements of the set. If the assignment holds for the
dimensions and measures in a given cell X, then V is a view
for X on the set S.
View Monotonicity: A query Q is view monotonic on
view V if for any cell X in any database D such that V is
the view for X for query Q , the condition Q is FALSE for
X implies that Q is FALSE for all X` X.
TEAM LinG

Data Mining with Incomplete Data
Hai Wang
Saint Marys University, Canada
Shouhong Wang
University of Massachusetts Dartmouth, USA
INTRODUCTION
Survey is one of the common data acquisition methods
for data mining (Brin, Rastogi & Shim, 2003). In data
mining one can rarely find a survey data set that contains
complete entries of each observation for all of the
variables. Commonly, surveys and questionnaires are
often only partially completed by respondents. The
possible reasons for incomplete data could be numer-
ous, including negligence, deliberate avoidance for pri-
vacy, ambiguity of the survey question, and aversion.
The extent of damage of missing data is unknown when
it is virtually impossible to return the survey or ques-
tionnaires to the data source for completion, but is one
of the most important parts of knowledge for data min-
ing to discover. In fact, missing data is an important
debatable issue in the knowledge engineering field
(Tseng, Wang, & Lee, 2003).
In mining a survey database with incomplete data,
patterns of the missing data as well as the potential
impacts of these missing data on the mining results
constitute valuable knowledge. For instance, a data miner
often wishes to know how reliable a data mining result
is, if only the complete data entries are used; when and
why certain types of values are often missing; what
variables are correlated in terms of having missing
values at the same time; what reason for incomplete data
is likely, etc. These valuable pieces of knowledge can be
discovered only after the missing part of the data set is
fully explored.
BACKGROUND
There have been three traditional approaches to handling
missing data in statistical analysis and data mining. One
of the convenient solutions to incomplete data is to
eliminate from the data set those records that have
missing values (Little & Rubin, 2002). This, however,
ignores potentially useful information in those records.
In cases where the proportion of missing data is large,
the data mining conclusions drawn from the screened
data set are more likely misleading.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
293
,
Another simple approach of dealing with missing
data is to use generic unknown for all missing data
items. However, this approach does not provide much
information that might be useful for interpretation of
missing data.
The third solution to dealing with missing data is to
estimate the missing value in the data item. In the case of
time series data, interpolation based on two adjacent
data points that are observed is possible. In general
cases, one may use some expected value in the data item
based on statistical measures (Dempster, Laird, & Rubin,
1997). However, data in data mining are commonly of
the types of ranking, category, multiple choices, and
binary. Interpolation and use of an expected value for a
particular missing data variable in these cases are gener-
ally inadequate. More importantly, a meaningful treat-
ment of missing data shall always be independent of the
problem being investigated (Batista & Monard, 2003).
More recently, there have been mathematical meth-
ods for finding the salient correlation structure, or
aggregate conceptual directions, of a data set with miss-
ing data (Aggarwal & Parthasarathy, 2001; Parthasarathy
& Aggarwal, 2003). These methods make themselves
distinct from the traditional approaches of treating miss-
ing data by focusing on the collective effects of the
missing data instead of individual missing values. How-
ever, these statistical models are data-driven, instead of
problem-domain-driven. In fact, a particular data mining
task is often related to its specific problem domain, and
a single generic conceptual construction algorithm is
insufficient to handle a variety of data mining tasks.
MAIN THRUST
There have been two primary approaches of data mining
with incomplete data: conceptual construction and en-
hanced data mining.
Conceptual Construction with
Incomplete Data
Conceptual construction with incomplete data reveals
the patterns of the missing data as well as the potential
TEAM LinG

impacts of these missing data on the mining results
based only on the complete data. Conceptual construc-
tion on incomplete data is a knowledge development
process. To construct new concepts on incomplete
data, the data miner needs to identify a particular prob-
lem as a base for the construction. According to (Wang,
S. & Wang, H., 2004), conceptual construction is car-
ried out through two phases. First, data mining tech-
niques (e.g., cluster analysis) are applied to the data set
with complete data to reveal the unsuspected patterns of
the data, and the problem is then articulated by the data
miner. Second, the incomplete data with missing values
related to the problem are used to construct new con-
cepts. In this phase, the data miner evaluates the impacts
of missing data on the identification of the problem and
develops knowledge related to the problem. For ex-
ample, suppose a data miner is investigating the profile
of the consumers who are interested in a particular
product. Using the complete data, the data miner has
found that variable i (e.g., income) is an important factor
of the consumers purchasing behavior. To further verify
and improve the data mining result, the data miner must
develop new knowledge through mining the incomplete
data. Four typical concepts as results of knowledge
discovery in data mining with incomplete data are de-
scribed as follows:
(1) Reliability: The reliability concept reveals the
scope of the missing data in terms of the problem
identified based only on complete data. For in-
stance, in the above example, to develop the reli-
ability concept, the data miner can define index
VM(i)/VC(i) where VM(i) is the number of missing
values in variable i, and V C(i) is the number of
samples used for the problem identification in
variable i. Accordingly, the higher VM(i)/V C(i) is,
the lower the reliability of the factor would be.
(2) Hiding: The concept of hiding reveals how likely
an observation with a certain range of values in one
variable is to have a missing value in another
variable. For instance, in the above example, the
data miner can define index VM(i)|x(j)(a,b) where
VM(i) is the number of missing values in variable i,
x(j) is the occurrence of variable j (e.g., education
years), and (a,b) is the range of x(j); and use this
index to disclose the hiding relationships between
variables i and j, say, more than two thousand records
have missing values in variable income given the
value of education years ranging from 13 to 19.
(3) Complementing: The concept of complementing
reveals what variables are more likely to have miss-
ing values at the same time; that is, the correlation
of missing values related to the problem being
investigated. For instance, in the above example,
294
Data Mining with Incomplete Data
the data miner can define index VM(i,j)/VM(i) where
VM(i,j) is the number of missing values in both
variables i and j, and VM(i) is the number of missing
values in variable i. This concept discloses the
correlation of two variables in terms of missing
values. The higher the value VM(i,j)/VM(i) is, the
stronger the correlation of missing values would be.
(4) Conditional Effects: The concept of conditional
effects reveals the potential changes to the under-
standing of the problem caused by the missing
values. To develop the concept of conditional
effects, the data miner assumes different possible
values for the missing values, and then observe the
possible changes of the nature of the problem. For
instance, in the above example, the data miner can
define index P|z(i)=k where P is the change of
the size of the target consumer group perceived by
the data miner, z(i) represents all missing values
of variable i, and k is the possible value variable i
might have for the survey. Typically, k={max, min,
p} where max is the maximal value of the scale, min
is the minimal value of the scale, and p is the random
variable with the same distribution function of the
values in the complete data. By setting different
possible values of k for the missing values, the data
miner is able to observe the change of the size of the
consumer group and redefine the problem.
Enhanced Data Mining with Incomplete
Data
The second primary approach to data mining with in-
complete data is enhanced data mining, in which incom-
plete data are fully utilized. Enhanced data mining is
carried out through two phases. In the first phase, obser-
vations with missing data are transformed into fuzzy
observations. Since missing values make the observa-
tion fuzzy, according to fuzzy set theory (Zadeh, 1978),
an observation with missing values can be transformed
into fuzzy patterns that are equivalent to the observa-
tion. For instance, suppose there is an observation
A=X(x1, x2, . . . xc . . .xm) where x c is the variable with
missing value, and xc{ r1, r 2 ... rp } where rj (j=1, 2, ... p)
is the possible occurrence of xc. Let j = P j(xc = rj), the
fuzzy membership (or possibility) that x c belongs to rj ,
(j=1, 2, ...p), and j j = 1. Then, j [X |(xc= rj)] (j=1, 2,
... p) are fuzzy patterns that are the equivalence to the
observation A.
In the second phase of enhanced data mining, all
fuzzy patterns, along with the complete data, are used
for data mining using tools such as self-organizing maps
(SOM) (Deboeck & Kohonen, 1998; Kohonen, 1989;
Vesanto & Alhoniemi, 2000) and other types of neural
TEAM LinG
Data Mining with Incomplete Data
networks (Wang, 2000, 2002). These tools used for en-
hanced data mining are different from the original ones in
that they are capable of retaining information of fuzzy
membership for each fuzzy pattern. Wang (2003) has devel-
oped a SOM-based enhanced data mining model to utilize all
fuzzy patterns and the complete data for knowledge discov-
ery. Using this model, the data miner is allowed to compare
SOM based on complete data and fuzzy SOM based on all
incomplete data to perceive covert patterns of the data set.
It also allows the data miner to conduct what-if trials by
including different portions of the incomplete data to dis-
close more accurate facts. Wang (2005) has developed a
Hopfield neural network based model (Hopfield & Tank,
1986) for data mining with incomplete survey data. The
enhanced data mining method utilizes more information
provided by fuzzy patterns, and thus makes the data mining
results more accurate. More importantly, it produces rich
information about the uncertainty (or risk) of the data mining
results.
FUTURE TRENDS
Research into data mining with incomplete data is still in
its infancy. The literature on this issue is still scarce.
Nevertheless, the data miners endeavor for knowing
what we do not know will accelerate research in this
area. More theories and techniques of data mining with
incomplete data will be developed in the near future,
followed by comprehensive comparisons of these theo-
ries and techniques. These theories and techniques will
be built on the combination of statistical models, neural
networks, and computational algorithms. Data manage-
ment systems on large-scale database systems for data
mining with incomplete data will be available for data
mining practitioners. In the long term, techniques of
dealing with missing data will become a prerequisite of
any data mining instrument.
CONCLUSION
Generally, knowledge discovery starts with the original
problem identification. Yet the validation of the problem
identified is typically beyond the database and generic
algorithms themselves. During the knowledge discovery
process, new concepts must be constructed through
demystifying the data. Traditionally, incomplete data in
data mining are often mistreated. As explained in this
chapter, data with missing values must be taken into
account in the knowledge discovery process.
There have been two major non-traditional approaches
that can be effectively used for data mining with incom-
plete data. One approach is conceptual construction on
incomplete data. It provides effective techniques for
knowledge development so that the data miner is allowed ,
to interpret the data mining results based on the particu-
lar problem domain and his/her perception of the missing
data. The other approach is fuzzy transformation. Ac-
cording to this approach, observations with missing
values are transformed into fuzzy patterns based on
fuzzy set theory. These fuzzy patterns along with obser-
vations with complete data are then used for data mining
through, for examples, data visualization and classifica-
tion.
The inclusion of incomplete data for data mining
would provide more information for the decision maker
in identifying problems, verifying and improving the
data mining results derived from observations with
complete data only.
REFERENCES
Aggarwal, C.C., & Parthasarathy, S. (2001). Mining
massively incomplete data sets by conceptual recon-
struction. Proceedings of the 7th ACM SIGKDD Inter-
national Conference on Knowledge Discovery and
Data Mining (pp. 227-232). New York: ACM Press.
Batista, G., & Monard, M. (2003). An analysis of four
missing data treatment methods for supervised learning.
Applied Artificial Intelligence, 17(5/6), 519-533.
Brin, S., Rastogi, R., & Shim, K. (2003). Mining optimized
gain rules for numeric attributes. IEEE Transactions on
Knowledge & Data Engineering, 15(2), 324-338.
Deboeck, G., & Kohonen, T. (1998). Visual explora-
tions in finance with self-organizing maps. London,
UK: Springer-Verlag.
Dempster, A.P., Laird, N.M., & Rubin, D.B. (1997).
Maximum likelihood from incomplete data via the EM
algorithm. Journal of the Royal Statistical Society,
B39(1), 1-38.
Hopfield, J.J., & Tank, D. W. (1986). Computing with
neural circuits. Sciences, 233, 625-633.
Kohonen, T. (1989). Self-organization and associa-
tive memory (3rd ed.). Berlin: Springer-Verlag.
Little, R.J.A., & Rubin, D.B. (2002). Statistical analy-
sis with missing data (2nd ed.). New York: John Wiley
and Sons.
Parthasarathy, S., & Aggarwal, C.C. (2003). On the use
of conceptual reconstruction for mining massively in-
complete data sets. IEEE Computer Society, 15(6),
1512-1521.
295
TEAM LinG

Tseng, S., Wang, K., & Lee, C. (2003). A pre-processing
method to deal with missing values by integrating cluster-
ing and regression techniques. Applied Artificial Intelli-
gence, 17(5/6), 535-544.
Vesanto, J., & Alhoniemi, E. (2000). Clustering of the self-
organizing map. IEEE Transactions on Neural Networks,
11(3), 586-600.
Wang, S. (2000). Neural networks. In M. Zeleny (Ed.),
IEBM handbook of IT in business (pp. 382-391). London,
UK: International Thomson Business Press.
Wang, S. (2002). Nonlinear pattern hypothesis genera-
tion for data mining. Data & Knowledge Engineering,
40(3), 273-283.
Wang, S. (2003). Application of self-organizing maps
for data mining with incomplete data Sets. Neural Com-
puting & Application, 12(1), 42-48.
Wang, S. (2005). Classification with incomplete survey
data: A Hopfield neural network approach, Computers
& Operational Research, 32(10), 2583-2594.
Wang, S., & Wang, H. (2004). Conceptual construction
on incomplete survey data. Data and Knowledge Engi-
neering, 49(3), 311-323.
Zadeh, L.A. (1978). Fuzzy sets as a basis for a theory of
possibility. Fuzzy Sets and Systems, 1, 3-28.
KEY TERMS
Aggregate Conceptual Direction: Aggregate con-
ceptual direction describes the trend in the data along
which most of the variance occurs, taking the missing
data into account.
Conceptual Construction with Incomplete Data:
Conceptual construction with incomplete data is a knowl-
edge development process that reveals the patterns of
the missing data as well as the potential impacts of these
missing data on the mining results based only on the
complete data.
296
Data Mining with Incomplete Data
Enhanced Data Mining with Incomplete Data: Data
mining that utilizes incomplete data through fuzzy trans-
formation.
Fuzzy Transformation: The process of transform-
ing an observation with missing values into fuzzy pat-
terns that are equivalent to the observation based on
fuzzy set theory.
Hopfield Neural Network: A neural network with
a single layer of nodes that have binary inputs and
outputs. The output of each node is fed back to all other
nodes simultaneously, and each of the node forms a
weighted sum of inputs and passes the output result
through a nonlinearity function. It applies a supervised
learning algorithm, and the learning process continues
until a stable state is reached.
Incomplete Data: The data set for data mining con-
tains some data entries with missing values. For in-
stance, when surveys and questionnaires are partially
completed by respondents, the entire response data
becomes incomplete data.
Neural Network: A set of computer hardware and/
or software that attempt to emulate the information
processing patterns of the biological brain. A neural
network consists of four main components:
(1) Processing units (or neurons); and each of them
has a certain output level at any point in time.
(2) Weighted interconnections between the various
processing units which determine how the output
of one unit leads to input for another unit.
(3) An activation rule which acts on the set of input at
a processing unit to produce a new output.
(4) A learning rule that specifies how to adjust the
weights for a given input/output pair.
Self-Organizing Map (SOM): Two layer neural
network that maps the high-dimensional data onto low-
dimensional grid of neurons through unsupervised learn-
ing or competitive learning process. It allows the data
miner to view the clusters on the output maps.
TEAM LinG

Data Quality in Cooperative Information
Systems
Carlo Marchetti
Universit di Roma La Sapienza, Italy
Massimo Mecella
Universit di Roma La Sapienza, Italy
Monica Scannapieco
Universit di Roma La Sapienza, Italy
Antonino Virgillito
Universit di Roma La Sapienza, Italy
INTRODUCTION
A Cooperative Information System (CIS) is a large-
scale information system that interconnects various
systems of different and autonomous organizations,
geographically distributed and sharing common objec-
tives (De Michelis et al., 1997). Among the different
resources that are shared by organizations, data are
fundamental; in real world scenarios, organization A
may not request data from organization B, if it does not
trust Bs data (i.e., if A does not know that the quality of
the data that B can provide is high). As an example, in an
e-government scenario in which public administrations
cooperate in order to fulfill service requests from citi-
zens and enterprises (Batini & Mecella, 2001), admin-
istrations very often prefer asking citizens for data
rather than from other administrations that have stored
the same data, because the quality of such data is not
known. Therefore, lack of cooperation may occur due to
lack of quality certification.
Uncertified quality also can cause a deterioration of
the data quality inside single organizations. If organiza-
tions exchange data without knowing their actual qual-
ity, it may happen that data of low quality will spread all
over the CIS. On the other hand, CISs are characterized
by high data replication (i.e., different copies of the
same data are stored by different organizations). From
a data quality perspective, this is a great opportunity;
improvement actions can be carried out on the basis of
comparisons among different copies in order to select
the most appropriate one or to reconcile available cop-
ies, thus producing a new improved copy to be notified
to all interested organizations.
In this article, we describe possible solutions to data
quality problems in CISs that have been implemented
within the DaQuinCIS architecture. The description of the
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
297
,
architecture will allow us to understand the challenges
posed by data quality management in CISs. Moreover, the
presentation of the DaQuinCIS services will provide ex-
amples of techniques that can be used to face data quality
challenges and will show future research directions that
should be investigated.
BACKGROUND
Data quality traditionally has been investigated in the
context of single information systems. Only recently,
there is a growing attention toward data quality issues in
the context of multiple and heterogeneous information
systems (Berti-Equille, 2003; Bertolazzi & Scannapieco,
2001; Naumann et al., 1999). In cooperative scenarios,
the main data quality issues regard (Bertolazzi &
Scannapieco, 2001):
assessment of the quality of the data owned by
each organization; and
methods and techniques for exchanging and im-
proving quality information.
For the assessment issue, some of the results al-
ready achieved for traditional systems can be borrowed.
In the statistical area, a lot of work has been done since
the late 1960s. Record linkage techniques have been
proposed, most of them based on the Fellegi and Sunter
(1969) model. Also, edit imputation methods based on
the Fellegi and Holt (1976) model have been provided in
the same area. Instead, in the database area, record
matching techniques (Hernandez & Stolfo, 1998) and
data cleaning tools (Galhardas et al., 2000) have been
proposed as a contribution to data quality assessment. In
Winkler (2004), a survey of data quality assessment
TEAM LinG

techniques is provided, covering both statistical tech-
niques and data cleaning solutions.
When considering the issue of exchanging data and the
associated quality, a model to export both data and quality
data needs to be defined. Some conceptual models to
associate quality information to data have been proposed,
which include an extension of the entity-relationship model
(Wang et al., 1993) and a data warehouse conceptual model
with quality features described through the description
logic formalism (Jarke at al., 1995). Both models are for
a specific purpose: the former to introduce quality ele-
ments in relational database design; the latter to introduce
quality elements in the data warehouse design. In Mihaila
et al. (2000), the problem of the quality of Web-available
information has been faced in order to select data with high
quality coming from distinct sources; every source has to
evaluate some pre-defined data quality parameters and to
make their values available through the exposition of
metadata.
Furthermore, exchanging data in CISs poses impor-
tant problems that have been addressed by the data
integration literature. Data integration is the problem of
combining data residing at different sources and provid-
ing the user with a unified view of these data (Lenzerini,
2002). As described in Yan et al. (1999), when perform-
ing data integration, two different types of conflicts
may arise: semantic conflicts, due to heterogeneous
source models; and instance-level conflicts, due to what
happens when sources record inconsistent values on the
same objects. The data quality broker described in the
following is a system solving instance-level conflicts.
Other notable examples of data integration systems
within the same category are AURORA (Yan et al.,
1999) and the system described in Sattler et al. (2003).
AURORA supports conflict tolerant queries (i.e., it
provides a dynamic mechanism to resolve conflicts by
means of defined conflict resolution functions). The
system described in Sattler et al. (2003) describes how
to solve both semantic and instance-level conflicts. The
proposed solution is based on a multi-database query
language, called FraQL, which is an extension of SQL
with conflict resolution mechanisms. A system that also
takes into account metadata for instance-level conflict
resolution is described in Fan et al. (2001). Such a
system adopts the ideas of the context interchange
framework (Bressan et al., 1997); therefore, context-
dependent and independent conflicts are distinguished,
and, accordingly, to this very specific direction, con-
version rules are discovered on pairs of systems.
Finally, among the techniques explicitly proposed to
perform query answering in CISs, we cite Naumann et al.
(1999), in which an algorithm to perform query planning
based on the evaluation of data sources qualities, specific
queries qualities, and query results qualities is described.
298
Data Quality in Cooperative Information Systems
MAIN THRUST
In current government and business scenarios, organiza-
tions start cooperating in order to offer services to their
customers and partners. Organizations that cooperate
have business links (i.e., relationships, exchanged docu-
ments, resources, knowledge, etc.) connecting each
other. Specifically, organizations exploit business ser-
vices (e.g., they exchange data or require services to be
carried out) on the basis of business links, and, there-
fore, the network of organizations and business links
constitutes a cooperative business system. As an ex-
ample, a supply chain, in which some enterprises offer
basic products and some others assemble them in order
to deliver final products to customers, is a cooperative
business system. As another example, a set of public
administrations, which need to exchange information
about citizens and their health state in order to provide
social aids, is a cooperative business system derived
from the Italian e-government scenario (Batini &
Mecella, 2001).
A cooperative business system exists independently
of the presence of a software infrastructure supporting
electronic data exchange and service provisioning. In-
deed, cooperative information systems are software
systems supporting cooperative business systems; in
the remaining of this article, the following definition of
CIS is considered:
A cooperative information system is formed by a set of
organizations that cooperate through a communication
infrastructure N, which provides software services to
organizations as well as reliable connectivity. Each
organization is connected to N through a gateway G,
on which software services offered by the organization
to other organizations are deployed. A user is a
software or human entity residing within an
organization and using the cooperative system.
Several CISs are characterized by a high degree of
data replicated in different organizations; for example,
in an e-government scenario, the personal data of a
citizen are stored by almost all administrations. But in
such scenarios, the different organizations can provide
the same data with different quality levels; thus, any user
of data may appreciate to exploit the data with the
highest quality level, among the provided ones.
Therefore, only the highest quality data should be
returned to the user, limiting the dissemination of low
quality data. Moreover, the comparison of the gathered
data values might be used to enforce a general improve-
ment of data quality in all organizations.
In the context of the DaQuinCIS project1, we are
proposing an architecture for the management of data
TEAM LinG
Data Quality in Cooperative Information Systems
Figure 1. An architecture for enhancing data quality in cooperative information systems
Communication
Infrastructure
Quality Notification
Service (QNS)
Orgn
Cooperative
Gateway
Org2
Cooperative
Gateway
Data Quality
Broker (DQB)
quality in CISs; this architecture allows the diffusion of
data and related quality and exploits data replication to
improve the overall quality of cooperative data. The
interested reader can find a detailed description of the
design and implementation of the architecture in
Scannapieco, et al. (2004). Each organization offers
services to other organizations on its own cooperative
gateway and also to specific services to its internal back-
end systems. Therefore, cooperative gateways interface
both internally and externally through services. More-
over, the communication infrastructure offers some spe-
cific services. Services are all identical and peer (i.e.,
they are instances of the same software artifacts and act
both as servers and clients of the other peers, depending
on the specific activities to be carried out). The overall
architecture is depicted in Figure 1. Organizations ex-
port data and quality data according to a common model
referred to as Data and Data Quality (D2Q) model. It
includes the definitions of (i) constructs to represent
data, (ii) a common set of data quality properties, (iii)
constructs to represent them, and (iv) the association
between data and quality data.
In order to produce data and quality data according to
the D2Q model, each organization deploys on its coop-
erative gateway a quality factory service that is respon-
sible for evaluating the quality of its own data. The design
of the quality factory has been addressed in Cappiello, et
al. (2003).
The Data Quality Broker poses, on behalf of a re-
questing user, a data request over other cooperating
organizations, also specifying a set of quality require-
ments that the desired data have to satisfy; this is re-
ferred to as quality brokering function. Different cop-
ies of the same data received as responses to the request
are reconciled, and a best-quality value is selected and
proposed to organizations, which can choose to discard
their data and to adopt higher quality ones; this is re-
ferred to as quality improvement function. If the re-
Rating
,
Service
Quality
Factory (QF)
Org1 back-end systems
Cooperative
Gateway
internals of the
organization
quirements specified in the request cannot be satisfied,
then the broker initiates a negotiation with the user that
can optionally weaken the constraints on the desired
data. The data quality broker is, in essence, a data
integration system (Lenzerini, 2002) deployed as a
peer-to-peer system, which allows to pose a quality-
enhanced query over a global schema and to select data
satisfying such requirements.
The Quality Notification Service is a publish/sub-
scribe engine used as a quality message bus between
services and/or organizations. More specifically, it al-
lows quality-based subscriptions for users to be notified
on changes of the quality of data. For example, an orga-
nization may want to be notified if the quality of some
data it uses degrades below a certain threshold, or when
high quality data are available. Also the quality notifica-
tion service is deployed as a peer-to-peer system.
The Rating Service associates trust values to each
data source in the CIS. These values are used to deter-
mine the reliability of the quality evaluation performed
by organizations. The rating service is a centralized
service, to be provided by a third-party organization.
The interested reader can find further details on the
rating service design and implementation in De Santis,
et al. (2003).
FUTURE TRENDS
The complete development of a framework for data
quality management in CISs requires the solution of
further issues. An important aspect concerns the tech-
niques to be used for quality dimension measurement.
In both statistical and machine learning areas, some
techniques could be usefully exploited. The general
idea is to have quality values estimated with a certain
probability instead of a deterministic quality evalua-
299
TEAM LinG

tion. In this way, the task of assigning quality values to
each data value could be considerably simplified.
The data quality broker covers some aspects of qual-
ity-driven query processing in CISs. Nevertheless, there
is still the need to investigate instance reconciliation
techniques; whereas, quality values are not attached to
exported data, how is it possible to select between two
conflicting instances of same data? Current data inte-
gration systems simply do not provide any answer in
such cases, but looser semantics for query answering is
needed in order to make data integration systems actu-
ally work in real scenarios where errors and conflicts
are present.
Some further open issues concern trusting data
sources. Data ownership is an important one. From a
data quality perspective, assigning a responsibility on
data helps actually to engage improvement actions as
well as to trust sources providing data. In some cases,
laws help to assign responsibilities on data typologies,
but this is not always possible. Models and techniques
that allow trusting such sources with respect to provided
data are an open and important issue, especially when
data sources interact in open and dynamic environments
like peer-to-peer systems.
CONCLUSION
Managing data quality in CISs requires solving problems
from many research areas of computer science, such as
databases, software engineering, distributed comput-
ing, security, and information systems. This implies that
the proposal of integrated solutions is very challenging.
In this article, an architecture to support data quality
management in CISs has been described; such an archi-
tecture consists of modules that provide some solutions
to principal data quality problems in CISs; namely, qual-
ity-driven query answering, data quality access, data
quality maintenance, and trust management. The archi-
tecture is suitable in all contexts in which data stored by
different and distributed sources are overlapping and
affected by data errors. Among such contexts, the archi-
tecture has been validated in the Italian e-government
scenario, in which all public administrations store data
about citizens that need to be corrected and reconciled.
REFERENCES
Batini, C., & Mecella, M. (2001). Enabling Italian e-
government through a cooperative architecture. IEEE
Computer, 34(2), 40-45.
300
Data Quality in Cooperative Information Systems
Berti-Equille, L. (2003). Quality-extended query process-
ing for distributed processing. Proceedings of the ICDT03
International Workshop on Data Quality in Cooperative
Information Systems (DQCIS03), Siena, Italy.
Bertolazzi, P., & Scannapieco, M. (2001). Introducing data
quality in a cooperative context. Proceedings of the 6th
International Conference on Information Quality
(ICIQ01), Boston, Massachusetts.
Bressan, S. et al. (1997). The context interchange mediator
prototype. Proceedings of the ACM SIGMOD Interna-
tional Conference on Management of Data, Tucson,
Arizona.
Cappiello, C. et al. (2003). Data quality assurance in
cooperative information systems: A multi-dimension
quality certificate. Proceedings of the ICDT03 Inter-
national Workshop on Data Quality in Cooperative
Information Systems (DQCIS03), Siena, Italy.
De Michelis, G. et al. (1997). Cooperative information
systems: A manifesto. In M.P. Papazoglou, & G. Schlageter
(Eds.), Cooperative information systems: Trends & di-
rections. London: Academic Press.
De Santis, L., Scannapieco, M., & Catarci, T. (2003).
Trusting data quality in cooperative information sys-
tems. Proceedings of the 11th International Confer-
ence on Cooperative Information Systems
(CoopIS03), Catania, Italy.
Fan, W., Lu, H., Madnick, S., & Cheung, D. (2001).
Discovering and reconciling value conflicts for numeri-
cal data integration. Information Systems, 26(8), 635-656.
Fellegi, I., & Holt, D. (1976). A systematic approach to
automatic edit and imputation. Journal of the American
Statistical Association, 71, 17-35.
Fellegi, I., & Sunter, A. (1969). A theory for record linkage.
Journal of the American Statistical Association, 64, 1183-
1210.
Galhardas, H. et al. (2000). An extensible framework for
data cleaning. Proceedings of the 16th International
Conference on Data Engineering (ICDE 2000), San
Diego, California.
Hernandez, M., & Stolfo, S. (1998). Real-world data is
dirty: Data cleansing and the merge/purge problem. Jour-
nal of Data Mining and Knowledge Discovery, 1(2), 9-37.
Jarke, M., Lenzerini, M., Vassiliou, V., & Vassiliadis, P.
(Eds.). (1995). Fundamentals of data warehouses. Berlin,
Heidelberg, Germany: Springer Verlag.
TEAM LinG
Data Quality in Cooperative Information Systems
Lenzerini, M. (2002). Data integration: A theoretical per-
spective. Proceedings of the 21st ACM Symposium on
Principles of Database Systems (PODS 2002), Madison,
Wisconsin.
Mihaila, G., Raschid, L., & Vidal, M.(2000). Using quality
of data metadata for source selection and ranking. Pro-
ceedings of the Third International Workshop on the Web
and Databases (WebDB00), Dallas, Texas.
Naumann, F., Leser, U., & Freytag, J. (1999). Quality-
driven integration of heterogenous information sys-
tems. Proceedings of the 25th International Confer-
ence on Very Large Data Bases (VLDB99), Edinburgh,
Scotland.
Redman, T. (1996). Data quality for the information
age. Norwood, MA: Artech House.
Sattler, K., Conrad, S., & Saake, G. (2003). Interactive
example-driven integration and reconciliation for ac-
cessing database integration. Information Systems, 28(5),
393-413.
Scannapieco, M. et al. (2004). The DaQuinCIS architec-
ture: A platform for exchanging and improving data qual-
ity in cooperative information systems. Information Sys-
tems, 29(7), 551-582.
Wand, Y., & Wang, R. (1996). Anchoring data quality
dimensions in ontological foundations. Communica-
tions of the ACM, 39(11), 86-95.
Wang, R. (1998). A product perspective on total data
quality management. Communications of the ACM, 41(2),
58-65.
Wang, R., Kon, H., & Madnick, S. (1993). Data quality
requirements: Analysis and modeling. Proceedings of
the 9th International Conference on Data Engineer-
ing (ICDE 93), Vienna, Austria.
Winkler, W.E. (2004). Methods for evaluating and cre-
ating data quality. Information Systems, 29(7), 531-550.
Yan, L., & Ozsu, T. (1999). Conflict-tolerant queries in
AURORA. Proceedings of the Fourth International
Conference on Cooperative Information Systems
(CoopIS99), Edinburgh, Scotland.
KEY TERMS
,
Cooperative Information Systems: Set of geographi-
cally distributed information systems that cooperate on
the basis of shared objectives and goals.
Data Integration System: System that presents
data distributed over heterogeneous data sources ac-
cording to a unified view. A data integration system
allows processing of queries over such a unified view by
gathering results from the various data sources.
Data Quality: Data have a good quality if they are fit
for use. Data quality is measured in terms of many
dimensions or characteristics, including accuracy, com-
pleteness, consistency, and currency of electronic data.
Data Schema: Collection of data types and relation-
ships described according to a particular type language.
Data Schema Instance: Collection of data values
that are valid (i.e., conform) with respect to a data
schema.
Metadata: Data providing information (e.g., qual-
ity, provenance, etc.) about application data. Metadata
are associated with data according to specific data mod-
els.
Peer-to-Peer Systems: Distributed systems in
which each node can be both client and server with
respect to service and data provision.
Publish and Subscribe: Distributed communica-
tion paradigm according to which subscribers are noti-
fied about events that have been published by publishers
and that are of interest to them.
ENDNOTE
1
DaQuinCIS - Methodologies and Tools for Data
Quality inside Cooperative Information Systems
(http://www.dis.uniroma1.it/~dq) is a joint re-
search project carried out by DIS - Universit di
Roma La Sapienza, DISCo - Universit di Milano
Bicocca and DEI - Politecnico di Milano.
301
TEAM LinG
302
Data Quality in Data Warehouses
William E. Winkler
U.S. Bureau of the Census, USA
INTRODUCTION
Fayyad and Uthursamy (2002) have stated that the ma-
jority of the work (representing months or years) in
creating a data warehouse is in cleaning up duplicates
and resolving other anomalies. This article provides an
overview of two methods for improving quality. The
first is data cleaning for finding duplicates within files
or across files. The second is edit/imputation for main-
taining business rules and for filling in missing data. The
fastest data-cleaning methods are suitable for files with
hundreds of millions of records (Winkler, 1999b,
2003b). The fastest edit/imputation methods are suit-
able for files with millions of records (Winkler, 1999a,
2004b).
BACKGROUND
When data from several sources are successfully com-
bined in a data warehouse, many new analyses can be
done that might not be done on individual files. If
duplicates are present within a file or across a set of
files, then the duplicates might be identified. Data clean-
ing or record linkage uses name, address, and other
information, such as income ranges, type of industry,
and medical treatment category, to determine whether
two or more records should be associated with the same
entity. Related types of files might be combined. In the
health area, a file of medical treatments and related
information might be combined with a national death
index. Sets of files from medical centers and health
organizations might be combined over a period of years
to evaluate the health of individuals and discover new
effects of different types of treatments. Linking files is an
alternative to exceptionally expensive follow-up studies.
The uses of the data are affected by lack of quality
due to the duplication of records and missing or errone-
ous values of variables. Duplication can waste money
and yield error. If a hospital has a patient incorrectly
represented in two different accounts, then the hospital
might repeatedly bill the patient. Duplicate records may
inflate the numbers and amounts in overdue-billing cat-
egories. If the quantitative amounts associated with
some accounts are missing, then the totals may be
biased low. If values associated with variables such as
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
billing amounts are erroneous because they do not satisfy
edit or business rules, then totals may be biased low or
high. Imputation rules can supply replacement values for
erroneous or missing values that are consistent with the
edit rules and preserve joint probability distributions.
Files without error can be effectively data mined.
MAIN THRUST
This section provides an overview of data cleaning and
of statistical data editing and imputation. The cleanup
and homogenization of the files are preprocessing steps
prior to data mining.
Data Cleaning
Data cleaning is also referred to as record linkage or
object identification. Record linkage was introduced by
Newcombe, Kennedy, Axford, and James (1959) and
given a formal mathematical framework by Fellegi and
Sunter (1969). Notation is needed. Two files, A and B,
are matched. The idea is to classify pairs in a product
space, A x B, from two files A and B into M, the set of
true matches, and U, the set of true nonmatches. Fellegi
and Sunter considered ratios of conditional probabili-
ties of the form
R = P( | M) / P( | U) (1)
where is an arbitrary agreement pattern in a comparison
space . For instance, might consist of eight patterns
representing simple agreement or disagreement on the
largest name component, street name, and street number.
Alternatively, each might additionally account for
the relative frequency with which specific values of name
components such as Smith, Zabrinsky, AAA, and
Capitol occur. Ratio R, or any monotonely increasing
function of it, such as the natural log, is referred to as a
matching weight (or score).
The decision rule is given by the following state-
ments:
If R > T, then designate the pair as a match.
If T<R<T , then designate the pair as a possible
match and hold it for clerical review. (2)
TEAM LinG
Data Quality in Data Warehouses
If R < T , then designate the pair as a nonmatch.
The cutoff thresholds T and T are determined by a
priori error bounds on false matches and false
nonmatches. Rule 2 agrees with intuition. If con-
sists primarily of agreements, then would intu-
itively be more likely to occur among matches than
nonmatches, and Ratio 1 would be large. On the other
hand, if consists primarily of disagreements, then
Ratio 1 would be small. Rule 2 partitions the set
into three disjoint subregions. The region T<R<T is
referred to as the no-decision region or clerical-review
region. In some situations, resources are available to
review pairs clerically.
Linkages can be error prone in the absence of strong
or unique identifiers such as a verified social security
number that identifies an individual record or entity.
Weak identifiers such as name, address, and other
nonuniquely identifying information are used. The com-
bination of weak identifiers can determine whether a
pair of records represents the same entity. If errors or
differences exist in the representations of names and
addresses, then many duplicates can erroneously be
added to a warehouse. For instance, a the name of a
business may be John K Smith and Company in one
file and J. K. Smith, Inc. in another file. Without the
additional corroborating of information such as ad-
dresses, it is difficult to determine whether the two
names correspond to the same entity. With three ad-
dresses such as 123 E. Main Street, 123 East Main
St., and P.O. Box 456 and the two names, the linkage
can still be quite difficult. With suitable preprocessing
methods, it may be possible to represent the names in
forms in which the different components can be com-
pared. To use addresses of the forms 123 E. Main
Street and P.O. Box 456, it may be necessary to use
an auxiliary file or expensive follow up that indicates
that the addresses have at some time been associated
with the same entity.
If individual fields have a minor typographical error,
then string comparators that account for such errors can
allow effective comparisons (Winkler, 1995, 2004b;
Cohen, Ravikumar, & Fienberg, 2003). Individual fields
might be first name, last name, and street name, which
are delineated by standardization software. Rule-based
methods of standardization are available in commercial
software for addresses and in other software for names
(Winkler, 1995, 1999b). The probabilities in Equations
1 and 2 are referred to as matching parameters. If
training data consisting of matched and unmatched pairs
are available, then a supervised method requiring train-
ing data can be used for estimation of the matching
parameters. Optimal-matching parameters can sometimes
be estimated via unsupervised learning methods, such as
the EM algorithm. The parameters are known to vary
significantly across files (Winkler, 1999b). They can even ,
vary significantly across similar files representing an
urban area and an adjacent suburban area. If two files each
contain 1,000 or more records, than bringing together all
pairs from two files is impractical,due to the small number
of potential matches within the total set of pairs. Blocking
is the method of considering only pairs that agree exactly
(character by character) on subsets of fields. For instance,
a set of blocking criteria may be to consider only pairs that
agree on the U.S. Postal zip code and the first character of
the last name. Additional blocking passes may be needed
to obtain matching pairs that are missed by earlier block-
ing passes (Newcombe et al., 1959; Hernandez & Stolfo,
1995; Winkler, 2004a).
Statistical Data Editing and Imputation
Correcting inconsistent information and filling in miss-
ing information needs to be efficient and cost effective.
For single fields, edits are straightforward. A lookup
table may yield correct diagnostic or zip codes. For
multiple fields, an edit might require that an individual
younger than 15 years of age must have a marital status
of unmarried. If a record fails this edit, then a subse-
quent procedure would need to change either the age or
the marital status.
Editing has been done extensively in statistical agen-
cies since the 1950s. Early work was clerical. Later
computer programs applied if-then-else rules with logic
similar to the clerical review. The main disadvantage
was that edits that did not fail, for a record would
initially fail as the values in fields associated with edit
failures were changed. Fellegi and Holt (1976) provided
a theoretical model. In providing their model, they had
three goals:
1. The data in each record should be made to satisfy
all edits by changing the fewest possible variables
(fields).
2. Imputation rules should derive automatically from
edit rules.
3. When imputation is necessary, it should maintain
the joint distribution of variables.
Fellegi and Holt (1976; Theorem 1) proved that
implicit edits are needed for solving the problem of
Goal 1. Implicit edits are those that can be logically
derived from explicitly defined edits. Implicit edits
provide information about edits that do not fail initially
for a record but may fail as the values in fields that are
associated with failing edits are changed. The following
example illustrates some of the computational issues. An
edit can be considered as a set of points. Let edit E =
303
TEAM LinG

{married & age 15}. Let r be a data record. Then r E =>
r fails edit. This formulation is equivalent to If age 15,
then not married. If a record r fails a set of edits, then one
field in each of the failing edits must be changed. An
implicit edit E3 can be implied from two explicitly defined
edits E 1 and E2; i.e., E1 and E2 => E3.
E1 = {age 15, married, . }
E2 = { . , not married, spouse}
E3 = {age 15, . , spouse}
The edits restrict the fields age, marital status, and
relationship to head of household. Implicit edit E3 is
derived from E1 and E2. If E3 fails for a record r = {age 15,
not married, spouse}, then necessarily either E1 or E2 fails.
Assume that the implicit edit E3 is unobserved. If edit E2
fails for record r, then one possible correction is to change
the marital status field in record r to married in order to
obtain a new record r1. Record r1 does not fail for E2 but now
fails for E1. The additional information from edit E3 assures
that record r satisfies all the edits after changing one
additional field. For larger data situations with more edits
and more fields, the number of possibilities increases at a
very high exponential rate.
In data warehouse situations, the ease of implement-
ing the ideas of Fellegi and Holt (1996) by using gener-
alized edit software is dramatic. An analyst who has
knowledge of the edit situations might put together the
edit tables in a relatively short time. Kovar and Winkler
(1996) compared two edit systems on economic data.
Both were installed and run in less than one day. In many
business situations, only a few simple edit rules might be
needed. In their books on data quality, Redman (1996),
English (1999), and Loshin (2001) have described edit-
ing files to assure that business rules are satisfied. The
authors have not noted the difficulties of applying
hardcoded if-then-else rules and the relative ease of
applying Fellegi and Holts methods.
FUTURE TRENDS
There are several trends in data cleaning. First, some
research considers better search strategies to compen-
sate for typographical error. Winkler (2003b, 2004a)
applies efficient blocking strategies for bringing to-
gether pairs in a situation where one file and its indexes
are held in memory. This allows the matching of a mod-
erate size file of 100 million records (which will reside
in 4 GB of memory) against large administrative files of
upwards of 4 billion records. This type of record linkage
requires only one pass against both files, whereas con-
ventional record linkage requires many sorts, matching
passes of both files, and large amounts of disk space.
304
Data Quality in Data Warehouses
Chaudhuri, Gamjam, Ganti, and Motwani (2003) provide
a method of indexing that significantly improves over
brute-force methods, in which all pairs in two files are
compared. Second, some research deals with better string
comparators for comparing fields having typographical
errors. Cohen et al. (2003) have methods that improve
over the basic Jaro-Winkler methods (Winkler, 2004b).
Third, another research area investigates methods to
standardize and parse general names and address fields
into components that can be more easily compared.
Borkar, Desmukh, and Sarawagi (2001), Churches, Chris-
ten, Lu, and Zhu (2002), and Agichtein and Ganti (2004)
have Hidden Markov methods that work as well or better
than some of the rule-based methods. Although the
Hidden Markov methods require training data, Churches
et al. provide methods for quickly creating additional
training data for new applications. The additional train-
ing data supplement a core set of generic training data.
The training data consists of free-form records and the
corresponding records that have been processed into
components. Fourth, because training data are rarely
available and optimal matching parameters are needed,
some research investigates unsupervised learning meth-
ods that do not require training data. Ravikumar and
Cohen (2004) have unsupervised learning methods that
improve over the basic EM methods of Winkler (1995,
1999b, 2003b). Their methods are even competitive with
supervised learning methods. Fifth, other research con-
siders methods of (nearly) automatic error rate estima-
tion with little or no training data. Winkler (2002) consid-
ers methods that use unlabeled data and very small
subsets of labeled data (training data). Sixth, some
research considers methods for adjusting statistical and
other types of analyses for matching error. Lahiri and
Larsen (2004) have methods for improving the accuracy
of statistical analyses in the presence of matching error.
Their methods extend ideas introduced by Scheuren and
Winkler (1993, 1997).
There are three trends for edit/imputation research.
The first trend comprises faster ways of determining
the minimum number of variables containing values
that contradict the edit rules. DeWaal (2003a, 2003b,
2003c) applies Fourier-Motzkin and cardinality con-
strained Chernikova algorithms that allow direct bound-
ing of the number of computational paths. Riera-
Ledesma and Salazar-Gonzalez (2004) apply clever
heuristics in the setup of the problems that allow direct
integer programming methods to perform much faster.
The second trend is to apply machine learning methods.
Nordbotten (1995) applies neural nets to the basic
editing problem. Di Zio, Scanu, Coppola, Luzi, and
Ponti (2004) apply Bayesian networks to the imputation
problem. The advantage of these approaches is that they
do not require the detailed rule elicitation of other edit
TEAM LinG
Data Quality in Data Warehouses
approaches; they depend on representative training data.
The training data consists of unedited records and the
corresponding edited records after review by subject
matter specialists. The third trend comprises methods
that preserve both statistical distributions and edit con-
straints in the preprocessed data. Winkler (2003a) con-
nects the generalized imputation methods of Little and
Rubin (2002) with generalized edit methods of Winkler
(1999a, 2003b). The potential advantage is that the
statistical properties needed for data mining may be
preserved.
CONCLUSION
To data mine effectively, data need to be preprocessed
in a variety of steps that include removing duplicates,
performing statistical data editing and imputation, and
doing other cleanup and regularization of the data. If
moderate errors exist in the data, data mining may waste
computational and analytic resources with little gain in
knowledge.
REFERENCES
Agichtein, E., & Ganti, V. (2004). Mining reference
tables for automatic text segmentation. ACM Special
Interest Group on Knowledge Discovery and Data Min-
ing (pp. 20-29).
Borkar, V., Deshmukh, K., & Sarawagi, S. (2001). Auto-
matic segmentation of text into structured records.
ACM SIGMOD 2001 (pp. 175-186).
Chaudhuri, S., Gamjam, K., Ganti, V., & Motwani, R.
(2003). Robust and efficient match for on-line data
cleaning. Proceedings of the ACM SIGMOD Conferences,
2003 (pp. 313-324).
Churches, T., Christen, P., Lu, J., & Zhu, J. X. (2002).
Preparation of name and address data for record linkage
using hidden Markov models. BioMed Central Medical
Informatics and Decision Making, 2(9), Retrieved from
http://www.biomedcentral.com/1472-6947/2/9/
Cohen, W. W., Ravikumar, P., & Fienberg, S. E. (2003).
A comparison of string metrics for matching names and
addresses. Proceedings of the International Joint Con-
ference on Artificial Intelligence, Mexico.
De Waal, T. (2003a). Solving the error localization
problem by means of vertex generation. Survey Meth-
odology, 29(1), 71-79.
De Waal, T. (2003b). A fast and simple algorithm for
automatic editing of mixed data. Journal of Official Sta- ,
tistics, 19(4), 383-402.
De Waal, T. (2003c). Processing of erroneous and un-
safe data. Rotterdam: ERIM Research in Management.
Di Zio, M., Scanu, M., Coppola, L., Luzi, O., & Ponti, A.
(2004). Bayesian networks for imputation. Journal of
the Royal Statistical Society, A, 167(2), 309-322.
English, L. P. (1999). Improving data warehouse and
business information quality: Methods for reducing
costs and increasing profits. New York: Wiley.
Fayyad, U., & Uthurusamy, R. (2002). Evolving data mining
into solutions for insights. Communications of the Asso-
ciation of Computing Machinery, 45(8), 28-31.
Fellegi, I. P., & Holt, D. (1976). A systematic approach
to automatic edit and imputation. Journal of the Ameri-
can Statistical Association, 71, 17-35.
Fellegi, I. P., & Sunter, A. B. (1969). A theory of record
linkage. Journal of the American Statistical Associa-
tion, 64, 1183-1210.
Hernandez, M., & Stolfo, S. J. (1995). The merge/purge
problem for large databases. ACM SIGMOD 1995 (pp. 127-
138).
Kovar, J. G., & Winkler, W. E. (1996). Editing eco-
nomic data. Proceedings of the Section on Survey
Research Methods, American Statistical Association
(pp. 81-87). Retrieved from http://www.census.gov/srd/
www/byyear.html
Lahiri, P. A., & Larsen, M. D. (2004). Regression analy-
sis with linked data. Journal of the American Statisti-
cal Association.
Little, R. A., & Rubin, D. B., (2002). Statistical analy-
sis with missing data (2nd ed.). New York: Wiley.
Loshin, D. (2001). Enterprise knowledge manage-
ment: The data quality approach. San Diego: Morgan
Kaufman.
Newcombe, H. B., Kennedy, J. M., Axford, S. J., &
James, A. P. (1959). Automatic linkage of vital records.
Science, 130, 954-959.
Nordbotten, S. (1995). Editing statistical records by
neural networks. Journal of Official Statistics, 11, 393-
414.
Ravikumar, P., & Cohen, W. W. (2004). A hierarchical
graphical model for record linkage. Proceedings of the
Conference on Uncertainty in Artificial Intelligence,
USA. Retrieved from http://www.cs.cmu.edu/~wcohen
305
TEAM LinG

Redman, T. C. (1996). Data quality in the information age.
Boston, MA: Artech.
Riera-Ledesma, J., & Salazar-Gonzalez, J.-J. (2004). A
branch-and-cut algorithm for the error localization
problem in data cleaning (Tech. Rep.). Tenerife, Spain:
Universidad de la Laguna.
Scheuren, F., & Winkler, W. E. (1993). Regression
analysis of data files that are computer matched. Survey
Methodology, 19, 39-58.
Scheuren, F., & Winkler, W. E. (1997). Regression
analysis of data files that are computer matched, II.
Survey Methodology, 23, 157-165.
Winkler, W. E. (1995). Matching and record linkage. In
B. G. Cox (Eds.), Business survey methods (pp. 355-384).
New York: Wiley.
Winkler, W. E. (1999a). The state of statistical data
editing. In Statistical data editing (pp. 169-187). Rome:
ISTAT.
Winkler, W. E. (1999b). The state of record linkage and
current research problems. Proceedings of the Survey
Methods Section, Statistical Society of Canada (pp. 73-
80).
Winkler, W. E. (2002). Record linkage and Bayesian
networks. Proceedings of the Section on Survey Re-
search Methods, American Statistical Association,
Retrieved from http://www.census.gov/srd/www/
byyear.html
Winkler, W. E. (2003a). A contingency table model for
imputing data satisfying analytic constraints. Proceed-
ings of the Section on Survey Research Methods,
American Statistical Association, . Retrieved from http:/
/www.census.gov/srd/www/byyear.html
Winkler, W. E. (2003b). Data cleaning methods. Pro-
ceedings of the ACM Workshop on Data Cleaning,
Record Linkage and Object Identification, USA. Re-
trieved from http://csaa.byu.edu/kdd03cleaning.html
306
Data Quality in Data Warehouses
Winkler, W. E. (2004a). Approximate string comparator
search strategies for very large administrative lists. Pro-
ceedings of the Section on Survey Research Methods,
American Statistical Association.
Winkler, W. E. (2004b). Methods for evaluating and cre-
ating data quality. Information Systems, 29(7), 531-550.
KEY TERMS
Data Cleaning: The methodology of identifying
duplicates in a single file or across a set of files by using
a name, address, and other information.
Data Mining: The application of analytical methods
and tools to data for the purpose of identifying patterns
and relationships, such as classification, prediction,
estimation, or affinity grouping.
Edit Restraints: Logical restraints such as busi-
ness rules that assure that an employees listed salary in
a job category is not too high or too low or that certain
contradictory conditions, such as a male hysterectomy,
do not occur.
Imputation: The method of filling in missing data
that sometimes preserves statistical distributions and
satisfies edit restraints.
Preprocessed Data: In preparation for data mining,
data that have been through preprocessing such as data
cleaning or edit/imputation.
Rule Induction: The process of learning, from cases
or instances, if-then rule relationships consisting of an
antecedent (if-part, defining the preconditions or cov-
erage of the rule) and a consequent (then-part, stating a
classification, prediction, or other expression of a prop-
erty that holds for cases defined in the antecedent).
Training Data: A representative subset of records
for which the truth of classifications and relationships
is known and that can be used for rule induction in
machine learning models.
TEAM LinG

Data Reduction and Compression in
Database Systems
Alexander Thomasian
New Jersey Institute of Technology, USA
INTRODUCTION
Data compression is storing data such that it requires
less space than usual. Data compression has been effec-
tively used in storing data in a compressed form on
magnetic tapes, disks, and even main memory. In many
cases, updated data cannot be stored in place when it is
not compressible to the same or smaller size. Compression
also reduces the bandwidth requirements in transmitting
(program) code, data, text, images, speech, audio, and
video. The transmission may be from main memory to the
CPU and its caches, from tape and disk into main memory,
or over local, metropolitan, and wide area networks. When
data compression is used, transmission time improves or,
conversely, the required transmission bandwidth is re-
duced. Two excellent texts on this topic are Sayood (2002)
and Witten, Bell, and Moffat (1999).
Huffman encoding is a popular data compression
method. It substitutes the symbols of an alphabet with k
bits per symbol, so that frequent symbols are repre-
sented with fewer than k bits, and less common symbols
with more than k bits. A significant saving in space is
possible when the distribution is highly skewed, for
example, the Zipf distribution, because the average num-
ber of bits to represent symbols is smaller than k bits.
Arithmetic coding is a more sophisticated technique
that represents a string with an appropriate fraction.
Lempel-Ziv coding substitutes a character string by an
index to a dictionary or a previous occurrence and the
string length. Variations of these algorithms, separately
or in combination, are used in many applications.
Compression can be lossy or lossless. Lossy data
compression is utilized in cases where some data loss
can be tolerated, for example, a restored compressed
image may not be discernibly different from the origi-
nal. Lossless data compression, which restores com-
pressed data to its original value, is absolutely neces-
sary in some applications. Quantization is a lossy data
compression method, which represents data more
coarsely than the original signal.
General purpose data compression is applicable to
data warehouses and databases. For example, a variation
of the lossless Lempel-Ziv method has been applied to
DB2 records. This is accomplished by analyzing the
records in a relational table and building dictionaries,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
307
,
which are then used in compressing the data. Because
relational tables are organized as database pages, each
page on disk (and main memory) will hold compressed
data, so the number of records per page is doubled. Data
is uncompressed on demand, with or without hardware
assistance.
Data reduction operates at a higher level of abstrac-
tion than data compression, although data compression
methods can be used in conjunction with data reduction.
An example is quantizing the data in a matrix, which has
been dimensionally reduced via the SVD method, as I
describe in the following sections. This concludes the
discussion of data compression; the remainder of this
article deals with data reduction.
MAIN THRUST
Recent interest in data reduction resulted in the New
Jersey Data Reduction Report (Barbara, et al., 1997),
which classifies data reduction methods into parametric
and nonparametric. Histograms, clustering, and indexing
structures are examples of nonparametric methods.
Another classification is direct versus transform-based
methods. Singular value decomposition (SVD) and
discrete wavelet transforms are parametric transform-
based methods. In the following few sections, I briefly
introduce the aforementioned methods.
SVD
SVD is applicable to a two-dimensional MN matrix X,
where M is the number of objects, and there are N
features per object. For example, M may represent the
number of customers, and the columns represent the
amount they spend on any of the N products. M may be
in the millions, while N is the hundreds or even thou-
sands.
According to SVD we have the decomposition X =
USVt, where U is another MN matrix, S is a diagonal
NN matrix of singular values, sn, 1<n<N, and V holds
the eigenvectors of principal components. Alternatively,
the covariance matrix C, the product of the transpose of
X times X divided by M, can be decompoed as: C =VVt,
TEAM LinG

where is a diagonal matrix of eigenvalues n = sn2/ M. We
assume, without a loss in generality, that the eigenvalues
are in nonincreasing order, such that the transformation
of coordinates into the principal components will yield Y
= XV, whose columns are in decreasing order of their
energy or variance. There is a reduction in the rank of the
matrix when some eigenvalues are equal to zero. If we
retain the first p columns of Y, the Normalized Mean
Square Error (NMSE) is equal to the sum of the eigenval-
ues of discarded columns divided by the trace of the matrix
(sum of the eigenvalues or diagonal elements of C, which
remains invariant). A significant reduction in the number
of columns can be attained at a relatively small NMSE, as
shown in numerous studies (see Korn, Jagadish, &
Faloutsos, 1997).
Higher dimensional data, as in the case of data ware-
houses, for example, (product, customer, date) (dol-
lars) can be reduced to two dimensions by appropriate
transformations, for example, with products as rows and
(customer date) as columns.
The columns of dataset X may not be globally corre-
late for example, high-income customers buy expensive
items, and low-income customers buy economy items
so that the items bought by these two groups of custom-
ers are disjoint. Higher data compression (for a given
NMSE) can be attained by first clustering the data, using
an off-the-shelf clustering method, such as k-means (Dun-
ham, 2003), and then applying SVD to clusters (Castelli,
Thomasian, & Li, 2003). More sophisticated clustering
methods, which generate elliptical clusters, may yield
higher dimensionality reduction. An SVD-friendly clus-
tering method, which generates clusters amenable to
dimensionality reduction, is proposed in Chakrabarti and
Mehrotra (2000).
K-nearest-neighbor (k-NN) queries can be carried
out with respect to a dataset, which has been subjected
to SVD, by first transforming the query point to the
appropriate coordinates by using the principal compo-
nents. In the case of multiple clusters, we first need to
determine the cluster to which the query point belongs.
In the case of the k-means clustering method, the query
point belongs to the cluster with the closest centroid.
After determining the k nearest neighbors in the primary
cluster, I need to determine if other clusters are to be
searched. A cluster is searched if the hypersphere cen-
tered on the query point, with the k nearest neighbors
inside it, intersects with the hypersphere of that cluster.
This step is repeated until no more intersections exist.
Multidimensional scaling (MS) is another method
for dimensionality reduction (Kruskal & Wish, 1978).
Given the pair-wise distances or dissimilarities among
a set of objects, the goal of MS is to represent them in
k dimensions so that their distances are preserved. A
stress function, which is the sum of squares of the
308
Data Reduction and Compression in Database Systems
difference between the distances of points with k di-
mensions and the original distance, is used to represent
the goodness of the fit. The value of k should be selected
to be as small as possible, while stress is maintained at
an appropriately low level. A fast, approximate alterna-
tive is FASTMAP, whose goal is to find a k-dimensional
space that matches the distances of an NN matrix for N
points (Faloutsos & Lin, 1995).
WAVELETS
According to Fouriers theorem, a continuous function
can be expressed as the sum of sinusoidal functions. A
discrete signal with n points can be expressed by the n
coefficients of a Discrete Fourier Transform (DFT).
According to Parsevals theorem, the energy in the time
and frequency domain are equal (Faloutsos, 1996).
The DFT consists of the sum of sine and cosine
functions. I am interested in transforms, which can
capture a vector with as few coefficients as possible.
The Discrete Cosine Transform (DCT) achieves better
energy concentration than DFT and also solves the fre-
quency-leak problem that plagues DFT (Agrawal,
Faloutsos, & Swami, 1993).
The Discrete Wavelet Transform (DWT) is also
related to DFT but achieves better lossy data compres-
sion. The Haar transform is a simple wavelet transform
that operates on a time sequence and computes the sum
and difference of its halves, recursively. DWT can be
applied to signals with multiple dimensions, one dimen-
sion at a time (Press, Teukolsky, Vetterling, & Flannery,
1996). To illustrate how a single dimensional wavelet
transform works, consider an image with four pixels
having the following values: [9,7,3,5] (Stollnitz, Derose,
& Salesin, 1996). We obtain a lower resolution image by
substituting pairs of pixel values with their average:
[8,4]. Information is lost due to down sampling. The
original pixels can be recovered by storing detail coef-
ficients, given as 1=98 and 1=34, that is, [1,1].
Another averaging and detailing step yields [6] and [2].
The wavelet transform of the original image is then
[6,2,1,1]. In fact, for normalization purposes, the last
two coefficients have to be divided by the square root of
2. Wavelet compression is attained by not retaining all
the coefficients.
As far as data compression in data warehousing is
concerned, a k-d DWT can be applied to a k-d data cube
to obtain a compressed approximation by saving a frac-
tion of the strongest coefficients. An approximate com-
putation of multidimensional aggregates for sparse data
using wavelets is reported in Vitter and Wang (1999).
TEAM LinG
Data Reduction and Compression in Database Systems
REGRESSION, MULTIREGRESSION,
AND LOG-LINEAR MODELS
Linear regression in its simplest form, given two vectors
x and y, uses the least squares formula to obtain the
approximation Y = a+bX. In case y is a polynomial in x,
for example, Y = a + bX + cX2 + dX3, the transformation
Xi = xi can be used to obtain a linear model so that it can
be solved by using the least squares method. The case Y
= bXn can be handled by first taking logarithms of both
sides. In effect, a set of points can be represented by a
compact formula. Most cases have multiple columns of
data so that multiregression can be used.
Log-linear modeling arises in the context of discrete
multivariate analysis and seeks approximation to dis-
crete multidimensional probability distributions. For
example, a probability distribution over several variables
is factored into a product of the joint probabilities of the
subsets; this is where data compression is achieved.
HISTOGRAMS
Data compression in the form of histograms is a well-
known database technique. A histogram on a column or
attribute can assist the relational query optimizer in
estimating the selectivity of that attribute. For example,
if the column has a nonclustered index, the histogram can
determine that using the index would be more costly
because the selectivity of the attribute is low
(Ramakrishnan & Gehrke, 2003). In this case histograms
are categorized into equidepth, equiwidth, and com-
pressed categories, where in the last case separate buck-
ets are assigned to the most frequent attributes.
CLUSTERING
Clustering partitions records into multiple groups or
clusters, such that (a) records in one cluster are similar
to each other and (b) records in different clusters are
dissimilar to each other (Ramakrishnan & Gehrke, 2003).
For example, using the age and salaries of the employees
in a large company, a clustering algorithm may deter-
mine that there are three major groups: young employees
with low salaries, middle-aged employees with higher
salaries, and older employees with the highest salaries.
More generally, a cluster may be represented by the
coordinates of its centroid and possibly its radius, which
is the average distance from the centroid.
Data reduction is attained in clustering by represent-
ing points based on their membership in a cluster so that
effectively all points in the cluster are assigned the
coordinates of its centroid.
A large number of clustering methods have been
developed over the years. They can be roughly classi- ,
fied as statistical, database, and machine-learning meth-
ods (Barbara et al., 1997), although the last category is
rather limited. Statistical methods are classified as
hierarchical and partitioning methods. Hierarchical
methods have been classified into agglomerative or
bottom-up methods and divisive or top-down methods.
The number of clusters to be generated should be
specified for partitioning methods. The k-means and k-
medoids are two such methods, where the medoid is an
actual centrally located point belonging to the cluster,
and a centroid is not.
The k-means method applied to n points first ran-
domly selects k points as the initial centroids of the
clusters. Then these steps are followed: (a) Assign the
remaining n-k points to one of k clusters based on the
proximity of the point to the centroids, according to
Euclidean distance, (b) compute the new centroids of
the k clusters, whose coordinates are the means of
respective dimensions, (c) reassign all points to clus-
ters based on their proximity, and (d) quit if there is no
change from one iteration to the next; otherwise, go
back to step (b).
A comprehensive review of basic clustering meth-
ods from the viewpoint of database and data-mining
applications is given in Dunham (2003). There is em-
phasis on clustering methods for very large datasets,
which cannot be held in main memory. It is important to
keep the number of passes over the disk-resident data
for clustering to a minimum.
INDEX TREES
Index trees are used in facilitating access to large
datasets, for example, relational tables, which is usu-
ally done based on a single attribute, such as age, or two
attributes, age and salary, by two separate one-dimen-
sional index structures. B+ trees are the most popular
indexing structures for a single dimension, with each
node splitting the range of values assigned to it n + 1
ways, where n is the node fanout. The nodes of the B+
tree correspond to 4 or 8 KB database pages and have an
occupancy of at least 50%. B+ trees are self-organiz-
ing and remain balanced as a result of insertions and
deletions. The larger the value of n, the lower the tree.
Typical B+ trees have three to four levels; the top two
levels fit in main memory. To determine employees
who are 31 to 35 years old and earn $90,000 to $120,000
annually is then tantamount to a set intersection of
records qualified by the two B+ trees.
R-trees, developed in the mid-1980s, deal with
multiple dimensions simultaneously. Over the years,
309
TEAM LinG

numerous multidimensional indexing methods (MDIMs)
have been proposed (Gaede & Gunther, 1998), although
their usage in operational database systems remains rather
limited. MDIMs have been classified into data-partition-
ing and space-partitioning methods. In the former case
the partitioning is carried out based on insertions and
deletions of multidimensional points in the index. Ex-
amples are R-trees and their variations. Space-partition-
ing methods, such as quad-trees and k-d-b trees, recur-
sively partition the space globally when local overflows
occur. As a result, space-partitioning methods may not be
balanced. I will discuss only data-partitioning methods in
the remainder of this article.
MDIMs tend to be viable for a limited number of
dimensions, and as the number of dimensions increases,
the dimensionality curse sets in (Faloutsos, 1996). In
effect, the index loses its effectiveness, because a
rather large fraction of the pages constituting the index
are touched as part of query processing.
Indexes need some extensions to provide summary
data for data reduction. They can be considered as
hierarchical histograms, but there is no easy way to
extract this information from an index.
SAMPLING
Sampling achieves data compression by selecting an
appropriate subset of a large dataset to which a (rela-
tional) query is applied. Sampling can be categorized as
follows (Han & Kamber, 2001): (a) simple random
sample without replacement, (b) simple random sample
with replacement, (c) cluster sample, and (d) stratified
sample (examples follow).
Consider a large university, which maintains student
records in a relational table. Two columns of this table
are of interest: year of study (i.e., freshman, sopho-
more, junior, senior) and the grade point average (GPA).
The average GPA by year of study can be specified
succinctly as an SQL GROUP-By query. Assuming that
neither column is indexed, instead of scanning the table
to obtain the averages GPAs, the system may randomly
select records from the table to carry out this task.
Online query processing is possible in this context,
that is, the system starts displaying the average GPAs
based on samples it has obtained so far by year to the
user and an error bound. The running of the query can be
stopped as soon as the user is satisfied (Hellerstein,
Haas, & Wang, 1997). When the data is not ordered by
one of the attributes under consideration, for example,
it is ordered alphabetically by student last names; then
all the records in the page can be used in sampling to
reduce the number of disk accesses. This is an example
310
Data Reduction and Compression in Database Systems
of a cluster sample. Stratified sampling could be attained
if the records were indexed according to the year.
FUTURE TRENDS
With rapid increases in the volume of data being held for
data mining and data warehousing, lossy yet accurate
data compression methods are required. This is also
important from the viewpoint of collecting data from
remote sources with low bandwidth transmission capa-
bility. The new field of data streams (Golab & Ozsu,
2003) uses summarization methods, such as transmit-
ting averages rather than detailed values.
CONCLUSION
I have provided a summary of data reduction methods
applicable to data warehousing and databases in general.
I have also discussed data compression. Appropriate
references are given for further study.
ACKNOWLEDGMENT
Supported by NSF through Grant 0105485 in Computer
Systems Architecture.
REFERENCES
Agrawal, A., Faloutsos, C., & Swami, A. (1993). Effi-
cient similarity search in sequence databases. Proceed-
ings of the Foundations of Data Organization and
Algorithms Conference (pp. 69-84), USA.
Barbara, D., et al. (1997). The New Jersey data reduction
report. Data Engineering Bulletin, 20(4), 3-42.
Castelli, V., Thomasian, A., & Li, C. S. (2003). CSVD:
Clustering and singular value decomposition for ap-
proximate similarity search in high dimensional spaces.
IEEE Transactions on Knowledge and Data Engineer-
ing, 15(3), 671-685.
Chakrabarti, K., & Mehrotra, S. (2000). Local dimen-
sionality reduction: A new approach to indexing high-
dimensional spaces. Proceedings of the 26th Interna-
tional Conference on Very Large Data Bases (pp. 89-100),
Egypt.
Dunham, M. H. (2003). Data mining: Introductory and
advanced topics. Prentice-Hall.
TEAM LinG
Data Reduction and Compression in Database Systems
Faloutsos, C. (1996). Searching multimedia databases by
content. Kluwer Academic Publishers.
Faloutsos, C., & Lin, K. I. (1995). Fastmap: A fast algorithm
for indexing, data-mining and visualization of traditional
and multimedia datasets. Proceedings of the ACM SIGMOD
International Conference (pp. 163-174), USA.
Gaede, V., & Guenther, O. (1998). Multidimensional index-
ing methods. ACM Computing Surveys, 30(2), 170-231.
Golab, L., & Ozsu, M. T. (2003). Data stream management
issues A survey. ACM SIGMOD Record, 32(2), 5-14.
Han, J., & Kamber, M. (2001). Data mining: Concepts
and techniques. Morgan Kaufmann.
Hellerstein, J. M., Haas, P. J., & Wang, H. J. (1997).
Online aggregation. Proceedings of the ACM SIGMOD
International Conference (pp. 171-182), USA.
Korn, F., Jagadish, H., & Faloutsos, C. (1997). Effi-
ciently supporting ad hoc queries in large datasets of
time sequences. Proceedings of the ACM SIGMOD
International Conference (pp. 289-300), USA.
Kruskal, J. B., & Wish, M. (1978). Multidimensional
scaling. Beverly Hills, CA: Sage Publications.
Press, W. H., Teukolsky, S. A., Vetterling, W. T., &
Flannery, B. P. (1996). Numerical recipes in C: The art
of scientific computing. Cambridge University Press.
Ramakrishnan, K., & Gehrke, J. (2003). Database man-
agement systems (3rd ed.). McGraw-Hill.
Sayood, K. (2002). Introduction to data compression
(2nd ed.). Elsevier.
Stollnitz, E. J., Derose, T. D., & Salesin, D. H. (1996).
Wavelets for computer graphics: Theory and applica-
tions. Prentice-Hall.
Vetterli, M., & Kovacevic, J. (1995). Wavelets and subband
coding. Prentice-Hall.
Vitter, J. S., & Wang, M. (1999). An approximate com-
putation of multidimensional aggregates of sparse data
using wavelets. Proceedings of the ACM SIGMOD Inter-
national Conference (pp. 193-204), USA.
Witten, I. H., Bell, T., & Moffat, A. (1999). Managing
gigabytes: Compressing and indexing documents and ,
images (2nd ed.). Morgan Kaufmann.
KEY TERMS
Clustering: The process of grouping objects based
on their similarity and dissimilarity. Similar objects
should be in the same cluster, which is different from
the cluster for dissimilar objects.
Histogram: A data structure that maintains one or
more attributes or columns of a relational DBMS to
assist the query optimizer.
Index Tree: Partitions the space in a single or
multiple dimensions for efficient access to the subset
of the data that is of interest.
Karhunen-Loeve Transform (KLT): Utilizes prin-
cipal component analysis or singular value decomposi-
tion to minimize the distance error introduced for that
level of dimensionality reduction.
Principal Component Analysis (PCA): Computes
the eigenvectors for principal components and uses
them to transform a matrix X into a matrix Y, whose
columns are aligned with the principal components.
Dimensionality is reduced by discarding columns in Y
with the least variance or energy.
Sampling: A technique for selecting units from a
population so that by studying the sample, you may
fairly generalize your results back to the population.
Singular Value Decomposition (SVD): Attains
the same goal as PCA by decomposing the X matrix into
a U matrix, a diagonal matrix of eigenvalues, and a
matrix of eigenvectors the same as those obtained by
PCA.
Wavelet Transform: A method to transform data so
that it can be represented compactly.
311
TEAM LinG
312
Data Warehouse Back-End Tools
Alkis Simitsis
National Technical University of Athens, Greece
Dimitri Theodoratos
New Jersey Institute of Technology, USA
INTRODUCTION
The back-end tools of a data warehouse are pieces of
software responsible for the extraction of data from sev-
eral sources, their cleansing, customization, and insertion
into a data warehouse. They are known under the general
term extraction, transformation and loading (ETL) tools.
In all the phases of an ETL process (extraction and
exportation, transformation and cleaning, and loading),
individual issues arise and, along with the problems and
constraints that concern the overall ETL process, make its
lifecycle a very complex task.
BACKGROUND
A Data Warehouse (DW) is a collection of technologies
aimed at enabling the knowledge worker (executive, man-
ager, analyst, etc.) to make better and faster decisions.
Data warehouses typically are divided into the front-end
part concerning end users who access the data ware-
house with decision-support tools, and the back-stage
part, where the collection, integration, cleaning and trans-
formation of data takes place in order to populate the
warehouse. The architecture of a data warehouse exhibits
various layers of data in which data from one layer are
derived from data of the previous layer (Figure 1). The
processes that take part in the back stage of the data
warehouse are data intensive, complex, and costly
(Vassiliadis, 2000). Several reports mention that most of
these processes are constructed through an in-house
development procedure that can consume up to 70% of
the resources for a data warehouse project (Gartner, 2003).
In order to facilitate and manage the data warehouse
operational processes, commercial tools exist in the mar-
ket under the general title Extraction-Transformation-
Loading (ETL) tools. To give a general idea of the func-
tionality of these tools, we mention their most prominent
tasks, which include (a) the identification of relevant
information at the source side; (b) the extraction of this
information; (c) the customization and integration of the
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Figure 1. Abstract architecture of a data warehouse
information coming from multiple sources into a common
format; (d) the cleaning of the resulting data set on the
basis of database and business rules; and (e) the propa-
gation of the data to the data warehouse and/or data
marts. In the sequel, we will adopt the general acronym
ETL for all kinds of in-house or commercial tools and all
the aforementioned categories of tasks.
In Figure 2, we abstractly describe the general frame-
work for ETL processes. In the left side, we can observe
the original data providers (sources). Typically, data
providers are relational databases and files. The data from
these sources are extracted by extraction routines, which
provide either complete snapshots or differentials of the
data sources. Then, these data are propagated to the Data
Staging Area (DSA), where they are transformed and
cleaned before being loaded to the data warehouse. Inter-
mediate results, again in the form of (mostly) files or
relational tables, are part of the data-staging area. The
data warehouse is depicted in the right part of Figure 2 and
comprises the target data stores (i.e., fact tables for the
storage of information and dimension tables with the
description and the multidimensional, roll-up hierarchies
of the stored facts). The loading of the central warehouse
is performed from the loading activities depicted right
before the data warehouse data store.
TEAM LinG
Data Warehouse Back-End Tools
Figure 2. The environment of extract-transformation-
load processes
Extract Transform Load
& Clean
Sources DSA DW
State of the Art
In the past, there have been research efforts toward the
design and optimization of ETL tasks. We mention three
research prototypes: (a) the AJAX system (Galhardas et
al., 2000); (b) the Potters Wheel system (Raman &
Hellerstein, 2001); and (c) ARKTOS II (Arktos II, 2004). The
first two prototypes are based on algebras, which we find
mostly tailored for the case of homogenizing Web data;
the latter concerns the modeling and the optimization of
ETL processes in a customizable and extensible manner.
An extensive review of data quality problems and
related literature, along with quality management method-
ologies, can be found in Jarke, et al. (2000). Rundensteiner
(1999) offers a discussion of various aspects of data
transformations. Sarawagi (2000) offers a similar collec-
tion of papers in the field of data, including a survey
(Rahm & Do, 2000) that provides an extensive overview of
the field, along with research issues and a review of some
commercial tools and solutions on specific problems
(Monge, 2000; Borkar et al., 2000). In a related but different
context, we would like to mention the IBIS tool (Cal et al.,
2003). IBIS is an integration tool following the global-as-
view approach to answer queries in a mediated system.
Moreover, there is a variety of ETL tools in the market.
Simitsis (2003) lists the ETL tools available at the time that
this paper was written.
MAIN THRUST
In this section, we briefly review the problems and con-
straints that concern the overall ETL process, as well as
the individual issues that arise separately in each phase
of an ETL process (extraction and exportation, transfor-
mation and cleaning, and loading). Simitsis (2004) offers
a detailed study on the problems described in this paper
and presents a framework toward the modeling and the
optimization of ETL processes.
Scalzo (2003) mentions that 90% of the problems in
data warehouses arise from the nightly batch cycles that ,
load the data. At this stage, the administrators have to
deal with problems like (a) efficient data loading and (b)
concurrent job mixture and dependencies. Moreover, ETL
processes have global time constraints, including the
time they must be initiated and their completion deadlines.
In fact, in most cases, there is a tight time window in the
night that can be exploited for the refreshment of the data
warehouse, since the source system is off-line or not
heavily used during this period. Other general problems
include the scheduling of the overall process, the finding
of the right execution order for dependent jobs and job
sets on the existing hardware for the permitted time
schedule, and the maintenance of the information in the
data warehouse.
Phase I: Extraction and Transportation
During the ETL process, a first task that must be per-
formed is the extraction of the relevant information that
has to be propagated further to the warehouse
(Theodoratos et al., 2001). In order to minimize the overall
processing time, this involves only a fraction of the
source data that has changed since the previous execu-
tion of the ETL process, mainly concerning the newly
inserted and possibly updated records. Usually, change
detection is performed physically by the comparison of
two snapshots (one corresponding to the previous ex-
traction and the other to the current one). Efficient algo-
rithms exist for this task, like the snapshot differential
algorithms presented in Labio and Garcia-Molina (1996).
Another technique is log sniffing (i.e., the scanning of the
log file in order to reconstruct the changes performed
since the last scan). In rare cases, change detection can
be facilitated by the use of triggers. However, this solu-
tion is technically impossible for many of the sources that
are legacy systems or plain flat files. In numerous other
cases, where relational systems are used at the source
side, the usage of triggers also is prohibitive due to the
performance degradation that their usage incurs and to
the need to intervene in the structure of the database.
Moreover, another crucial issue concerns the transporta-
tion of data after the extraction, where tasks like ftp,
encryption-decryption, compression-decompression, and
so forth can possibly take place.
Phase II: Transformation and Cleaning
It is possible to determine typical tasks that take place
during the transformation and cleaning phase of an ETL
process. Rahm and Do (2000) further detail this phase in
the following tasks: (a) data analysis; (b) definition of
313
TEAM LinG

transformation workflow and mapping rules; (c) verifica-
tion; (d) transformation; and (e) backflow of cleaned data.
In terms of the transformation tasks, we distinguish two
main classes of problems (Lenzerini, 2002): (a) conflicts
and problems at the schema level (e.g., naming and struc-
tural conflicts) and (b) data-level transformations (i.e., at
the instance level).
The integration and transformation programs perform
a wide variety of functions, such as reformatting data,
recalculating data, modifying key structures of data, add-
ing an element of time to data warehouse data, identifying
default values of data, supplying logic to choose between
multiple sources of data, summarizing data, merging data
from multiple sources, and so forth.
In the sequel, we present four common ETL transforma-
tion cases as examples: (a) semantic normalization and
denormalization; (b) surrogate key assignment; (c) slowly
changing dimensions; and (d) string problems. The re-
search prototypes presented in the previous section and
several commercial tools already have done some piece of
progress in order to tackle problems like these four. Still,
their presentation in this paper aspires to make the reader
understand that the whole process should be discrimi-
nated from the way we resolved integration issues until now.
Semantic Normalization and
Denormalization
It is common for source data to be long denormalized
records, possibly involving more than 100 attributes. This
is due to the fact that bad database design or classical
COBOL tactics led to the gathering of all the necessary
information for an application in a single table/file. Fre-
quently, data warehouses also are highly denormalized in
order to answer certain queries more quickly. But some-
times, it is imperative to normalize somehow the input data.
Consider, for example, a table of the form
R(KEY,TAX,DISCOUNT,PRICE), which we would like to
transform to a table of the form R(KEY,CODE,AMOUNT).
For example, the input tuple t[key,30,60,70] is transformed
into the tuples t1[key,1,30]; t2[key,2,60]; t3[key,3,70]. The
transformation of the information organized in rows to
information organized in columns is called rotation or
denormalization, since frequently, the derived values (e.g.,
the total income, in our case) also are stored as columns,
functionally dependent on other attributes. Occasionally,
it is possible to apply the reverse transformation in order
to normalize denormalized data before being loaded to the
data warehouse.
Surrogate Keys
In a data warehouse project, we usually replace the keys
of the production systems with a uniform key, which we
314
Data Warehouse Back-End Tools
call a surrogate key (Kimball et al., 1998). The basic
reasons for this replacement are performance and seman-
tic homogeneity. Performance is affected by the fact that
textual attributes are not the best candidates for indexed
keys and need to be replaced by integer keys. More
importantly, semantic homogeneity causes reconcilia-
tion problems, since different production systems might
use different keys for the same object (synonyms) or the
same key for different objects (homonyms), resulting in
the need for a global replacement of these values in the
data warehouse. Observe row (20,green) in table Src_1 of
Figure 3. This row has a synonym conflict with row
(10,green) in table Src_2, since they represent the same
real-world entity with different IDs, and a homonym
conflict with row (20,yellow) in table Src_2 (over at-
tribute ID). The production key ID is replaced by a
surrogate key through a lookup table of the form
Lookup(SourceID,Source,SurrogateKey). The Source
column of this table is required, because there can be
synonyms in the different sources, which are mapped to
different objects in the data warehouse (e.g., value 10 in
tables Src_1 and Src_2). At the end of this process, the
data warehouse table DW has globally unique, recon-
ciled keys.
Slowly Changing Dimensions
Factual data are not the only data that change. The
dimension values change at the sources, too, and there
are several policies to apply for their propagation to the
data warehouse. Kimball, et al. (1998) present three poli-
cies for dealing with this problem: overwrite (Type 1);
create a new dimensional record (Type 2); push down the
changed value into an old attribute (Type 3).
For Type 1 processing, we only need to issue appro-
priate update commands in order to overwrite attribute
values in existing dimension table records. This policy
can be used whenever we do not want to track history in
dimension changes and if we want to correct erroneous
values. In this case, we use an old version of the dimen-
Figure 3. Surrogate key assignment
ID, COLOR
10, red
ID,
100,
COLOR
red
20, green 200, green
300, yellow
Src_1
ML
DW
ID, COLOR ML
10, green
20, yellow
Src_2 SourceID, Source, SurrogateKey
10, Src_1, 100
20, Src_1, 200
10, Src_2, 200
Lookup
20, Src_2, 300
TEAM LinG
Data Warehouse Back-End Tools
sion data Dold as they were received from the source and
their current version Dnew. We discriminate the new and
updated rows through the respective operators. The new
rows are assigned a new surrogate key through a function
application. The updated rows are assigned a surrogate
key, which is the same as the one that their previous
version had already being assigned. Then, we can join the
updated rows with their old versions from the target table,
which subsequently will be deleted, and project only the
attributes with the new values.
In the Type 2 policy, we copy the previous version of
the dimension record and create a new one with a new
surrogate key. If there is no previous version of the
dimension record, we create a new one from scratch;
otherwise, we keep them both. This policy can be used
whenever we want to track the history of dimension
changes.
Finally, Type 3 processing is also very simple, since,
again, we only have to issue update commands to existing
dimension records. For each attribute A of the dimension
table, which is checked for updates, we need to have an
extra attribute called old_A. Each time we spot a new value
for A, we write the current A value to the old_A field and
then write the new value to attribute A. In this way, we can
have both new and old values present at the same dimen-
sion record.
String Problems
A major challenge in ETL processes is the cleaning and
the homogenization of string data (e.g., data that stands
for addresses, acronyms, names, etc.). Usually, the ap-
proaches for the solution of this problem include the
application of regular expressions for the normalization of
string data to a set of reference values.
Phase III: Loading
The final loading of the data warehouse has its own
technical challenges. A major problem is the ability to
discriminate between new and existing data at loading
time. This problem arises when a set of records has to be
classified to (a) the new rows that need to be appended to
the warehouse and (b) rows that already exist in the data
warehouse, but their value has changed and must be
updated (e.g., with an UPDATE command). Modern ETL
tools already provide mechanisms for this problem, mostly
through language predicates. Also, simple SQL com-
mands are not sufficient, since the open-loop-fetch tech-
nique, where records are inserted one by one, is extremely
slow for the vast volume of data to be loaded in the
warehouse. An extra problem is the simultaneous usage
of the rollback segments and log files during the loading
process. The option to turn them off contains some risk
in the case of a loading failure. So far, the best technique ,
seems to be the usage of the batch loading tools offered
by most RDBMSs that avoid these problems. Other tech-
niques that facilitate the loading task involve the creation
of tables at the same time with the creation of the respec-
tive indexes, the minimization of inter-process wait states,
and the maximization of concurrent CPU usage.
FUTURE TRENDS
In terms of financial growth, in a recent study (Giga
Information Group, 2002), it is reported that the ETL
market reached a size of $667 million for year 2001; still, the
growth rate reached a rather low 11% (as compared to a
rate of 60% growth for year 2000). More recent studies
(Giga Information Group, 2002; Gartner, 2003) account the
ETL issue as a research challenge and pinpoint several
topics for future work:
Integration of ETL with XML adapters; EAI (Enter-
prise Application Integration) tools (e.g., MQ-Se-
ries); customized data quality tools; and the move
toward parallel processing of the ETL workflows.
Active ETL (Adzic & Fiore, 2003), meaning the need
to refresh the warehouse with as fresh of data as
possible (ideally, online).
Extension of the ETL mechanisms for non-tradi-
tional data, like XML/HTML, spatial, and biomedical.
CONCLUSION
ETL tools are pieces of software responsible for the
extraction of data from several sources, their cleansing,
customization, and insertion into a data warehouse. In all
the phases of an ETL process (extraction and exportation,
transformation and cleaning, and loading), individual
issues arise and, along with the problems and constraints
that concern the overall ETL process, make its lifecycle a
very troublesome task. The key factors underlying the
main problems of ETL workflows are (a) vastness of the
data volumes; (b) quality problems, since data are not
always clean and have to be cleansed; (c) performance,
since the whole process has to take place within a specific
time window; and (d) evolution of the sources and the
data warehouse, which can eventually lead even to daily
maintenance operations. Although state of the art in the
field of both research and commercial ETL tools includes
some signs of progress, much work remains to be done
before we can claim that this problem is resolved. In our
opinion, there are several issues that are technologically
315
TEAM LinG

open and that present interesting topics of research for
the future in the field of data integration in data ware-
house environments.
REFERENCES
Adzic, J., & Fiore, V., (2003). Data warehouse population
platform. Proceedings of 5th International Workshop on
the Design and Management of Data Warehouses
(DMDW), Berlin, Germany.
Arktos II. (2004). A framework for modeling and managing
ETL processes. Retrieved from http://www.dblab.ece.
ntua.gr/~asimi
Borkar, V., Deshmuk, K., & Sarawagi, S. (2000). Automati-
cally extracting structure from free text addresses. Bulletin
of the Technical Committee on Data Engineering, 23(4).
Cal, A. et al. (2003). IBIS: Semantic data integration at
work. Proceedings of the 15th CAiSE.
Galhardas, H., Florescu, D., Shasha, D., & Simon, E. (2000).
Ajax: An extensible data cleaning tool. Proceedings ACM
SIGMOD International Conference On the Management
of Data, Dallas, Texas.
Gartner. (2003). ETL magic quadrant update: Market
pressure increases. Retrieved from http://www.gartner.
com/reprints/informatica/112769.html
Giga Information Group. (2002). Market overview update:
ETL. Technical Report RPA-032002-00021.
Inmon, W.-H. (1996). Building the data warehouse. New
York: John Wiley & Sons, Inc.
Jarke, M., Lenzerini, M., Vassiliou, Y., & Vassiliadis, P.
(Eds.). (2000). Fundamentals of data warehouses.
Springer-Verlag.
Kimball, R., Reeves, L., Ross, M., & Thornthwaite, W.
(1998). The data warehouse lifecycle toolkit: Expert
methods for designing, developing, and deploying data
warehouses. New York: John Wiley & Sons.
Labio, W., & Garcia-Molina, H. (1996). Efficient snapshot
differential algorithms for data warehousing. Proceed-
ings of 22nd International Conference on Very Large
Data Bases (VLDB), Bombay, India.
Lenzerini, M. (2002). Data integration: A theoretical per-
spective. Proceedings of 21st Symposium on Principles
of Database Systems (PODS), Wisconsin.
Monge, A. (2000). Matching algorithms within a duplicate
detection system. Bulletin of the Technical Committee
on Data Engineering, 23(4).
316
Data Warehouse Back-End Tools
Rahm, E., & Do, H.H. (2000). Data cleaning: Problems and
current approaches. Bulletin of the Technical Committee
on Data Engineering, 23(4).
Raman, V., & Hellerstein, J. (2001). Potters wheel: An
interactive data cleaning system. Proceedings of 27th
International Conference on Very Large Data Bases
(VLDB), Rome, Italy.
Rundensteiner, E. (Ed.). (1999). Special issue on data
transformations. Bulletin of the Technical Committee on
Data Engineering, 22(1).
Sarawagi, S. (2000). Special issue on data cleaning. Bulle-
tin of the Technical Committee on Data Engineering, 23(4).
Scalzo, B. (2003). Oracle DBA guide to data warehousing
and star schemas. Upper Saddle River, NJ : Prentice Hall.
Simitsis, A. (2003). List of ETL tools. Retrieved from http:/
/www.dbnet.ece.ntua.gr/~asimi/ETLTools.htm
Simitsis, A. (2004). Modeling and managing extraction-
transformation-loading (ETL) processes in data ware-
house environments [doctoral thesis]. National Technical
University of Athens, Greece.
Theodoratos, D., Ligoudistianos, S., & Sellis, T. (2001).
View selection for designing the global data warehouse.
Data & Knowledge Engineering, 39(3), 219-240.
Vassiliadis, P. (2000). Gulliver in the land of data ware-
housing: Practical experiences and observations of a
researcher. Proceedings of 2nd International Workshop
on Design and Management of Data Warehouses
(DMDW), Sweden.
KEY TERMS
Data Mart: A logical subset of the complete data
warehouse. We often view the data mart as the restriction
of the data warehouse to a single business process or to
a group of related business processes targeted toward a
particular business group.
Data Staging Area (DSA): An auxiliary area of volatile
data employed for the purpose of data transformation,
reconciliation, and cleaning before the final loading of the
data warehouse.
Data Warehouse: A subject-oriented, integrated, time-
variant, non-volatile collection of data used to support
the strategic decision-making process for the enterprise.
It is the central point of data integration for business
intelligence and is the source of data for the data marts,
delivering a common view of enterprise data (Inmon, 1996).
TEAM LinG
Data Warehouse Back-End Tools
ETL: Extract, transform, and load (ETL) are data ware-
housing functions that involve extracting data from out-
side sources, transforming them to fit business needs,
and ultimately loading them into the data warehouse. ETL
is an important part of data warehousing, as it is the way
data actually gets loaded into the warehouse.
Online Analytical Processing (OLAP): The general
activity of querying and presenting text and number data
from data warehouses, as well as a specifically dimen-
sional style of querying and presenting that is exemplified
by a number of OLAP vendors.
Source System: An operational system of record
whose function is to capture the transactions of the ,
business. A source system is often called a legacy system
in a mainframe environment.
Target System: The physical machine on which the
data warehouse is organized and stored for direct query-
ing by end users, report writers, and other applications. A
target system is often called a presentation server.
317
TEAM LinG
318
Data Warehouse Performance
Beixin (Betsy) Lin
Montclair State University, USA
Yu Hong
BearingPoint Inc., USA
Zu-Hsu Lee
Montclair State University, USA
INTRODUCTION
A data warehouse is a large electronic repository of
information that is generated and updated in a structured
manner by an enterprise over time to aid business intelli-
gence and to support decision making. Data stored in a
data warehouse is non-volatile and time variant and is
organized by subjects in a manner to support decision
making (Inmon, Rudin, Buss, & Sousa, 1998). Data ware-
housing has been increasingly adopted by enterprises as
the backbone technology for business intelligence re-
porting and query performance has become the key to the
successful implementation of data warehouses. Accord-
ing to a survey of 358 businesses on reporting and end-
user query tools, conducted by Appfluent Technology,
data warehouse performance significantly affects the
Return on Investment (ROI) on Business Intelligence (BI)
systems and directly impacts the bottom line of the
systems (Appfluent Technology, 2002). Even though in
some circumstances it is very difficult to measure the
benefits of BI projects in terms of ROI or dollar figures,
management teams are still eager to have a single version
of the truth, better information for strategic and tactical
decision making, and more efficient business processes
by using BI solutions (Eckerson, 2003).
Dramatic increases in data volumes over time and the
mixed quality of data can adversely affect the performance
of a data warehouse. Some data may become outdated
over time and can be mixed with data that are still valid for
decision making. In addition, data are often collected to
meet potential requirements, but may never be used. Data
warehouses also contain external data (e.g. demographic,
psychographic, etc.) to support a variety of predictive
data mining activities. All these factors contribute to the
massive growth of data volume. As a result, even a simple
query may become burdensome to process and cause
overflowing system indices (Inmon, Rudin, Buss & Sousa,
2001). Thus, exploring the techniques of performance
tuning becomes an important subject in data warehouse
management.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
There are inherent differences between a traditional data-
base system and a data warehouse system, though to a
certain extent, all databases are similarly designed to
serve a basic administrative purpose, e.g., to deliver a
quick response to transactional data processes such as
entry, update, query and retrieval. For many conventional
databases, this objective has been achieved by online
transactional processing (OLTP) systems (e.g. Oracle
Corp, 2004; Winter & Auerbach, 2004). In contrast, data
warehouses deal with a huge volume of data that are more
historical in nature. Moreover, data warehouse designs
are strongly organized for decision making by subject
matter rather than by defined access or system privileges.
As a result, a dimension model is usually adopted in a data
warehouse to meet these needs, whereas an Entity-Rela-
tionship model is commonly used in an OLTP system. Due
to these differences, an OLTP query usually requires
much shorter processing time than a data warehouse
query (Raden, 2003). Performance enhancement tech-
niques are, therefore, especially critical in the arena of
data warehousing.
Despite the differences, these two types of database
systems share some common characteristics. Some tech-
niques used in a data warehouse to achieve a better
performance are similar to those used in OLTP, while some
are only developed in relation to data warehousing. For
example, as in an OLTP system, an index is also used in a
data warehouse system, though a data warehouse might
have different kinds of indexing mechanisms based on its
granularity. Partitioning is a technique which can be used
in data warehouse systems as well (Silberstein, Eacrett,
Mayer, & Lo, 2003).
On the other hand, some techniques are developed
specifically to improve the performance of data ware-
houses. For example, aggregates can be built to provide
a quick response time for summary information (e.g. Eacrett,
2003; Silberstein, 2003). Query parallelism can be imple-
mented to speed up the query when data are queried from
TEAM LinG
Data Warehouse Performance
several tables (Silberstein, et al., 2003). Caching and query
statistics are unique for data warehouses since the statis-
tics will help to build a smart cache for better performance.
Also, pre-calculated reports are useful to certain groups
of users who are only interested in seeing static reports
(Eacrett, 2003). Periodic data compression and archiving
helps to cleanse the data warehouse environment. Keep-
ing only the necessary data online will allow faster access
(e.g. Kimball, 1996).
MAIN THRUST
As discussed earlier, performance issues play a crucial
role in a data warehouse environment. This chapter de-
scribes ways to design, build, and manage data ware-
houses for optimum performance. The techniques of
tuning and refining the data warehouse discussed below
have been developed in recent years to reduce operating
and maintenance costs and to substantially improve the
performance of new and existing data warehouses.
Performance Optimization at the Data
Model Design Stage
Adopting a good data model design is a proactive way to
enhance future performance. In the data warehouse
design phase, the following factors should be taken into
consideration.
Granularity: Granularity is the main issue that
needs to be investigated carefully before the data
warehouse is built. For example, does the report
need the data at the level of store keeping units
(SKUs), or just at the brand level? These are size
questions that should be asked of business users
before designing the model. Since a data ware-
house is a decision support system, rather than a
transactional system, the level of detail required
is usually not as deep as the latter. For instance, a
data warehouse does not need data at the document
level such as sales orders, purchase orders, which
are usually needed in a transactional system. In
such a case, data should be summarized before
they are loaded into the system. Defining the data
that are needed no more and no less will
determine the performance in the future. In some
cases, the Operational Data Stores (ODS) will be
a good place to store the most detailed granular
level data and those data can be provided on the
jump query basis.
Cardinality: Cardinality means the number of pos-
sible entries of the table. By collecting business
requirements, the cardinality of the table can be
decided. Given a tables cardinality, an appropriate ,
indexing method can then be chosen.
Dimensional Models: Most data warehouse de-
signs use dimensional models, such as Star-Schema,
Snow-Flake, and Star-Flake. A star-schema is a
dimensional model with fully denormalized hierar-
chies, whereas a snowflake schema is a dimensional
model with fully normalized hierarchies. A star-flake
schema represents a combination of a star schema
and a snow-flake schema (e.g. Moody & Kortink,
2003). Data warehouse architects should consider
the pros and cons of each dimensional model before
making a choice.
Aggregates
Aggregates are the subsets of the fact table data (Eacrett,
2003). The data from the fact table are summarized into
aggregates and stored physically in a different table than
the fact table. Aggregates can significantly increase the
performance of the OLAP query since the query will read
fewer data from the aggregates than from the fact table.
Database read time is the major factor in query execution
time. Being able to reduce the database read time will help
the query performance a great deal since fewer data are
being read. However, the disadvantage of using aggre-
gates is its loading performance. The data loaded into the
fact table have to be rolled up to the aggregates, which
means any newly updated records will have to be updated
in the aggregates as well to make the data in the aggre-
gates consistent with those in the fact table. Keeping the
data as current as possible has presented a real challenge
to data warehousing (e.g. Bruckner & Tjoa, 2002). The
ratio of the database records transferred to the database
records read is a good indicator of whether or not to use
the aggregate technique. In practice, if the ratio is 1/10 or
less, building aggregates will definitely help performance
(Silberstein, 2003).
Database Partitioning
Logical partitioning means using year, planned/actual
data, and business regions as criteria to partition the
database into smaller data sets. After logical partitioning,
a database view is created to include all the partitioned
tables. In this case, no extra storage is needed and each
partitioned table will be smaller to accelerate the query
(Silberstein, Eacrett, Mayer & Lo, 2003). Take a multina-
tional company as an example. It is better to put the data
from different countries into different data targets, such
as cubes or data marts, than to put the data from all the
countries into one data target. By logical partitioning
(splitting the data into smaller cubes), query can read the
319
TEAM LinG

smaller cubes instead of large cubes and several parallel
processes can read the small cubes at the same time.
Another benefit of logical partitioning is that each parti-
tioned cube is less complex to load and easier to perform
the administration. Physical partitioning can also reach the
same goal as logical partitioning. Physical partitioning
means that the database table is cut into smaller chunks of
data. The partitioning is transparent to the user. The
partitioning will allow parallel processing of the query and
each parallel process will read a smaller set of data sepa-
rately.
Query Parallelism
In business intelligence reporting, the query is usually
very complex and it might require the OLAP engine to read
the data from different sources. In such case, the technique
of query parallelism will significantly improve the query
performance. Query parallelism is an approach to split the
query into smaller sub-queries and to allow parallel pro-
cessing of sub-queries. By using this approach, each sub-
process takes a shorter time and reduces the risk of system
hogging if a single long-running process is used. In con-
trast, sequential processing definitely requires a longer
time to process (Silberstein, Eacrett, Mayer & Lo, 2003).
Database Indexing
In a relational database, indexing is a well known technique
for reducing database read time. By the same token, in a
data warehouse dimensional model, the use of indices in
the fact table, dimension table, and master data table will
improve the database read time.
The Fact Table Index: By default the fact table
will have the primary index on all the dimension
keys. However, a secondary index can also be built
to fit a different query design (e.g. McDonald,
Wilmsmeier, Dixon, & Inmon, 2002). Unlike a
primary index, which includes all the dimension
keys, the secondary index can be built to include
only some dimension keys to improve the perfor-
mance. By having the right index, the query read
time can be dramatically reduced.
The Dimension Table Index: In a dimensional
model, the size of the dimension table is the decid-
ing factor affecting query performance. Thus, the
index of the dimension table is important to de-
crease the master data read time, and thus to im-
prove filtering and drill down. Depending on the
cardinality of the dimension table, different index
methods will be adopted to build the index. For a low
cardinality dimension table, Bit-Map index is usually
320
Data Warehouse Performance
adopted. In contrast, for a high cardinality dimen-
sion table, the B-Tree index should be used (e.g.
McDonald, Wilmsmeier, Dixon & Inmon, 2002).
The Master Data Table Index: Since the data
warehouse commonly uses dimensional models,
the query SQL plan always starts from reading the
master data table. Using indices on the master data
table will significantly enhance the query perfor-
mance.
Caching Technology
Caching technology will play a critical role in query
performance as the memory size and speed of CPUs
increase. After a query runs once, the result is stored in
the memory cache. Subsequently, similar query runs will
get the data directly from the memory rather than by
accessing the database again. In an enterprise server, a
certain block of memory is allocated for the caching use.
The query results and the navigation status can then be
stored in a highly compressed format in that memory
block. For example, a background user can be set up
during the off-peak time to mimic the way a real user runs
the query. By doing this, a cache is generated that can be
used by real business users.
Pre-Calculated Report
Pre-calculation is one of the techniques where the admin-
istrator can distribute the workload to off-peak hours and
have the result sets ready for faster access (Eacrett,
2003). There are several benefits of using the pre-calcu-
lated reports. The user will have faster response time
since calculation takes place on the fly. Also, the system
workload is balanced and shifted to off-peak hours.
Lastly, the reports can be available offline.
Use Statistics to Further Tune up the
System
In a real-world data warehouse system, the statistics data
of the OLAP are collected by the system. The statistics
provide such information as what the most used queries
are and how the data are selected. The statistics can help
further tune the system. For example, examining the
descriptive statistics of the queries will reveal the most
common used drill-down dimensions as well as the
combinations of the dimensions. Also, the OLAP statis-
tics will also indicate what the major time component is
out of the total query time. It could be database read time
or OLAP calculation time. Based on the data, one can
build aggregates or offline reports to increase the query
performance.
TEAM LinG
Data Warehouse Performance
Data Compression
Over time, the dimension table might contain some unnec-
essary entries or redundant data. For example, when some
data has been deleted from the fact table, the correspond-
ing dimension keys will not be used any more. These keys
need to be deleted from the dimension table in order to
have a faster query time since the query execution plan
always start from the dimension table. A smaller dimen-
sion table will certainly help the performance.
The data in the fact table need to be compressed as
well. From time to time, some entries in the fact table might
contain just all zeros and need to be removed. Also,
entries with the same dimension key value should be
compressed to reduce the size of the fact table.
Kimball (1996) pointed out that data compression
might not be a high priority for an OLTP system since the
compression will slow down the transaction process.
However, compression can improve data warehouse per-
formance significantly. Therefore, in a dimensional model,
the data in the dimension table and fact table need to be
compressed periodically.
Data Archiving
In the real data warehouse environment, data are being
loaded daily or weekly. The volume of the production data
will become huge over time. More production data will
definitely lengthen the query run time. Sometimes, there
are too much unnecessary data sitting in the system.
These data may be out-dated and not important to the
analysis. In such cases, the data need to be periodically
archived to offline storage so that they can be removed
from the production system (e.g. Uhle, 2003). For example,
three years of production data are usually sufficient for a
decision support system. Archived data can still be pulled
back to the system if such need arises
FUTURE TRENDS
Although data warehouse applications have been evolv-
ing to a relatively mature stage, several challenges remain
in developing a high performing data warehouse environ-
ment. The major challenges are:
How to optimize the access to volumes of data based
on system workload and user expectations (e.g.
Inmon, Rudin, Buss & Sousa, 1998)? Corporate
information systems rely more and more on data
warehouses that provide one entry point for access
to all information for all users. Given this, and
because of varying utilization demands, the system
should be tuned to reduce the system workload
during peak user periods.
How to design a better OLAP statistics recording
system to measure the performance of the data ,
warehouse? As mentioned earlier, query OLAP sta-
tistics data is important to identify the trend and the
utilization of the queries by different users. Based
on the statistics results, several techniques could
be used to tune up the performance, such as aggre-
gates, indices and caching.
How to maintain data integrity and keep the good
data warehouse performance at the same time? In-
formation quality and ownership within the data
warehouse play an important role in the delivery of
accurate processed data results (Ma, Chou, & Yen,
2000). Data validation is thus a key step towards
improving data reliability and quality. Accelerating
the validation process presents a challenge to the
performance of the data warehouse.
CONCLUSION
Information technologies have been evolving quickly in
recent decades. Engineers have made great strides in
developing and improving databases and data warehous-
ing in many aspects, such as hardware; operating sys-
tems; user interface and data extraction tools; data mining
tools, security, and database structure and management
systems.
Many success stories in developing data warehous-
ing applications have been published (Beitler & Leary,
1997; Grim & Thorton, 1997). An undertaking of this scale
creating an effective data warehouse, however, is costly
and not risk-free. Weak sponsorship and management
support, insufficient funding, inadequate user involve-
ment, and organizational politics have been found to be
common factors contributing to failure (Watson, Gerard,
Gonzalez, Haywood & Frenton, 1999). Therefore, data
warehouse architects need to first clearly identify the
information useful for the specific business process and
goals of the organization. The next step is to identify
important factors that impact performance of data ware-
housing implementation in the organization. Finally all the
business objectives and factors will be incorporated into
the design of the data warehouse system and data models,
in order to achieve high-performing data warehousing
and business intelligence.
REFERENCES
Appfluent Technology. (2002, December 2). A study on
reporting and business intelligence application usage.
Retrieved from http://searchsap.techtarget.com/
whitepaperPage/0,293857,sid21_gci866993,00.html
321
TEAM LinG

Beitler, S.S., & Leary, R. (1997). Sears EPIC transforma-
tion: Converting from mainframe legacy systems to On-
Line Analytical Processing (OLAP). Journal of Data
Warehousing, 2, 5-16.
Bruckner, R.M., & Tjoa, A.M. (2002). Capturing delays
and valid times in data warehousestowards timely con-
sistent analyses. Journal of Intelligent Information Sys-
tems, 19(2), 169-190.
Eacrett, M. (2003). Hitchhikers guide to SAP business
information warehouse performance tuning. SAP White
Paper.
Eckerson, W. (2003). BI StatShots. Journal of Data Ware-
housing, 8(4), 64.
Grim, R., & Thorton, P.A. (1997). A customer for life: the
warehouseMCI approach. Journal of Data Warehousing,
2, 73-79.
Inmon, W.H., Imhoff, C., & Sousa, R. (2001). Corporate
information factory. New York: John Wiley & Sons, Inc.
Inmon, W.H., Rudin, K., Buss, C.K., & Sousa, R. (1998).
Data Warehouse Performance. New York: John Wiley &
Sons, Inc.
Kimball, R. (1996). The data warehouse toolkit. New York:
John Wiley & Sons, Inc.
Ma, C., Chou, D.C., & Yen, D.C. (2000). Data warehous-
ing, technology assessment and management. Industrial
Management + Data Systems, 100, 125
McDonald, K., Wilmsmeier, A., Dixon, D.C., & Inmon,
W.H. (2002, August). Mastering the SAP business infor-
mation warehouse. Hoboken, NJ: John Wiley & Sons,
Inc.
Moody, D., & Kortink, M.A.R. (2003). From ER models to
dimensional models, part II: Advanced design issues.
Journal of Data Warehousing, 8, 20-29.
Oracle Corp. (2004). Largest transaction processing db on
Unix runs oracle database. Online Product News, 23(2).
Peterson, S. (1994). Stars: A pattern language for query
optimized schema. Sequent Computer Systems, Inc. White
Paper.
Raden, N. (2003). Real time: Get real, part II. Intelligent
Enterprise, 6(11), 16.
Silberstein, R. (2003). Know how network: SAP BW perfor-
mance monitoring with BW statistics. SAP White Paper.
Silberstein, R., Eacrett, M., Mayer, O., & Lo, A. (2003). SAP
BW performance tuning. SAP White Paper.
322
Data Warehouse Performance
Uhle, R. (2003). Data aging with mySAP business intelli-
gence. SAP White Paper.
Watson, H., Gerard, J., Gonzalez, L.E., Haywood, M.E., &
Fenton, D. (1999). Data warehousing failures: Case stud-
ies and findings. Journal of Data Warehousing, 4, 44-55.
Winter, R., & Auerbach, K. (2004). Contents under pres-
sure: scalability challenges for large databases. Intelli-
gent Enterprise, 7(7), 18-25.
KEY TERMS
Cache: A region of a computers memory which stores
recently or frequently accessed data so that the time of
repeated access to the same data can decrease.
Granularity: The level of detail or complexity at which
an information resource is described.
Indexing: In data storage and retrieval, the creation
and use of a list that inventories and cross-references
data. In database operations, a method to find data more
efficiently by indexing on primary key fields of the data-
base tables.
ODS (Operational Data Stores): A system with
capability of continuous background update that keeps
up with individual transactional changes in operational
systems versus a data warehouse that applies a large load
of updates on an intermittent basis.
OLAP (Online Analytical Processing): A category
of software tools for collecting, presenting, delivering,
processing and managing multidimensional data (i.e.,
data that has been aggregated into various categories or
dimensions) in order to provide analytical insights for
business management.
OLTP (Online Transaction Processing): A stan-
dard, normalized database structure designed for trans-
actions in which inserts, updates, and deletes must be
fast.
Service Management: The strategic discipline for
identifying, establishing, and maintaining IT services to
support the organizations business goal at an appropri-
ate cost.
SQL (Structured Query Language): A standard
interactive programming language used to communicate
with relational databases in order to retrieve, update, and
manage data.
TEAM LinG

Data Warehousing and Mining in Supply Chains
Richard Mathieu
Saint Louis University, USA
Reuven R. Levary
Saint Louis University, USA
INTRODUCTION
Every finished product has gone through a series of
transformations. The process begins when manufactur-
ers purchase the raw materials that will be transformed
into the components of the product. The parts are then
supplied to a manufacturer, who assembles them into the
finished product and ships the completed item to the
consumer. The transformation process includes numer-
ous activities (Levary, 2000). Among them are
Designing the product
Designing the manufacturing process
Determining which component parts should be pro-
duced in house and which should be purchased
from suppliers
Forecasting customer demand
Contracting with external suppliers for raw materials
or component parts
Purchasing raw materials or component parts from
suppliers
Establishing distribution channels for raw materials
and component parts from suppliers to manufac-
turer
Establishing of distribution channels to the suppli-
ers of raw materials and component parts
Establishing distribution channels from the manu-
facturer to the wholesalers and from wholesalers to
the final customers
Manufacturing the component parts
Transporting the component parts to the manufac-
turer of the final product
Manufacturing and assembling the final product
Transporting the final product to the wholesalers,
retailers, and final customer
Each individual activity generates various data items
that must be stored, analyzed, protected, and transmitted
to various units along a supply chain.
A supply chain can be defined as a series of activities
that are involved in the transformation of raw materials
into a final product, which a customer then purchases
(Levary, 2000). The flow of materials, component parts,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
323
,
and products is moving downstream (i.e., from the initial
supply sources to the end customers). The flow of infor-
mation regarding the demand for the product and orders
to suppliers is moving upstream, while the flow of infor-
mation regarding product availability, shipment sched-
ules, and invoices is moving downstream. For each orga-
nization in the supply chain, its customer is the subse-
quent organization in the supply chain, and its subcon-
tractor is the prior organization in the chain.
BACKGROUND
Supply chain data can be characterized as either transac-
tional or analytical (Shapiro, 2001). All new data that are
acquired, processed, and compiled into reports that are
transmitted to various organizations along a supply chain
are deemed transactional data (Davis & Spekman, 2004).
Increasingly, transactional supply chain data is processed
and stored in enterprise resource planning systems, and
complementary data warehouses are developed to sup-
port decision-making processes (Chen R., Chen, C., &
Chang, 2003; Zeng, Chiang, & Yen, 2003). Organizations
such as Home Depot, Lowes, and Volkswagen have
developed data warehouses and integrated data-mining
methods that complement their supply chain management
operations (Dignan, 2003a; Dignan, 2003b; Hofmann,
2004). Data that are used in descriptive and optimization
models are considered analytical data (Shapiro, 2001).
Descriptive models include various forecasting models,
which are used to forecast demands along supply chains,
and managerial accounting models, which are used to
manage activities and costs. Optimization models are
used to plan resources, capacities, inventories, and prod-
uct flows along supply chains.
Data collected from consumers are the core data that
affect all other data items along supply chains. Informa-
tion collected from the consumers at the point of sale
include data items regarding the sold product (e.g., type,
quantity, sale price, and time of sale) as well as information
about the consumer (e.g., consumer address and method
of payment). These data items are analyzed often. Data-
mining techniques are employed to determine the types of
TEAM LinG

items that are appealing to consumers. The items are
classified according to consumers socioeconomic back-
grounds and interests, the sale price that consumers are
willing to pay, and the location of the point of sale.
Data regarding the return of sold products are used to
identify potential problems with the products and their
uses. These data include information about product qual-
ity, consumer disappointment with the product, and legal
consequences. Data-mining techniques can be used to
identify patterns in returns so that retailers can better
determine which type of product to order in the future and
from which supplier it should be purchased. Retailers are
also interested in collecting data regarding competitors
sales so that they can better promote their own product
and establish a competitive advantage.
Data related to political and economic conditions in
supplier countries are of interest to retailers. Data-mining
techniques can be used to identify political and economic
patterns in countries. Information can help retailers choose
suppliers who are situated in countries where the flow of
products and funds is expected to be stable for a reason-
ably long period of time.
Manufacturers collect data regarding a) particular
products and their manufacturing process, b) suppliers,
and c) the business environment. Data regarding the
product and the manufacturing process include the char-
acteristics of products and their component parts ob-
tained from CAD/CAM systems, the quality of products
and their components, and trends in theresearch and
development (R & D) of relevant technologies. Data-
mining techniques can be applied to identify patterns in
the defects of products, their components, or the manu-
facturing process. Data regarding suppliers include avail-
ability of raw materials, labor costs, labor skills, techno-
logical capability, manufacturing capacity, and lead time
of suppliers. Data related to qualified teleimmigrants (e.g.,
engineers and computer software developers) is valuable
to many manufacturers. Data-mining techniques can be
used to identify those teleimmigrants having unique knowl-
edge and experience. Data regarding the business envi-
ronment of manufacturers include information about com-
petitors, potential legal consequences regarding a prod-
uct or service, and both political and economic conditions
in countries where the manufacturer has either facilities or
business partners. Data-mining techniques can be used
to identify possible liability concerning a product or
service as well as trends in political and economic condi-
tions in countries where the manufacturer has business
interests.
Retailers, manufacturers, and suppliers are all inter-
ested in data regarding transportation companies. These
data include transportation capacity, prices, lead time,
and reliability for each mode of transportation.
324
Data Warehousing and Mining in Supply Chains
MAIN THRUST
Data Aggregation in Supply Chains
Large amounts of data are being accumulated and stored
by companies belonging to supply chains. Data aggrega-
tion can improve the effectiveness of using the data for
operational, tactical, and strategic planning models. The
concept of data aggregation in manufacturing firms is
called group technology (GT). Nonmanufacturing firms
are also aggregating data regarding products, suppliers,
customers, and markets.
Group Technology
Group technology is a concept of grouping parts, re-
sources, or data according to similar characteristics. By
grouping parts according to similarities in geometry,
design features, manufacturing features, materials used,
and/or tooling requirements, manufacturing efficiency
can be enhanced, and productivity increased. Manufac-
turing efficiency is enhanced by
Performing similar activities at the same work center
so that setup time can be reduced
Avoiding duplication of effort both in the design
and manufacture of parts
Avoiding duplication of tools
Automating information storage and retrieval
(Levary, 1993)
Effective implementation of the GT concept necessi-
tates the use of a classification and coding system. Such
a system codes the various attributes that identify simi-
larities among parts. Each part is assigned a number or
alphanumeric code that uniquely identifies the parts
attributes or characteristics. A parts code must include
both design and manufacturing attributes.
A classification and coding system must provide an
effective way of grouping parts into part families. All parts
in a given part family are similar in some aspect of design
or manufacture. A part may belong to more than one
family.
A part code is typically composed of a large number
of characters that allow for identification of all part at-
tributes. The larger the number of attributes included in a
part code, the more difficult the establishment of standard
procedures for classifying and coding. Although numer-
ous methods of classification and coding have been
developed, none has emerged as the standard method.
Because different manufacturers have different require-
ments regarding the type and composition of parts codes,
TEAM LinG
Data Warehousing and Mining in Supply Chains
customized methods of classification and coding are gen-
erally required. Some of the better known classification
and coding methods are listed by Groover (1987).
After a code is established for each part, the parts are
grouped according to similarities and are assigned to part
families. Each part family is designed to enhance manufac-
turing efficiency in a particular way. The information re-
garding each part is arranged according to part families in
a GT database. The GT database is designed in such a way
that users can efficiently retrieve desired information by
using the appropriate code.
Consider part families that are based on similarities of
design features. A GT database enables design engineers
to search for existing part designs that have characteris-
tics similar to those of a new part that is to be designed. The
search begins when the design engineer describes the
main characteristics of the needed part with the help of a
partial code. The computer then searches the GT database
for all the items with the same code. The results of the
search are listed on the computer screen, and the designer
can then select or modify an existing part design after
reviewing its specifications. Selected designs can easily
be retrieved. When design modifications are needed, the
file of the selected part is transferred to a CAD system.
Such a system enables the design engineer to effectively
modify the parts characteristics in a short period of time.
In this way, efforts are not duplicated when designing
parts.
The creation of a GT database helps reduce redun-
dancy in the purchasing of parts as well. The database
enables manufacturers to identify similar parts produced
by different companies. It also helps manufacturers to
identify components that can serve more than a single
function. In such ways, GT enables manufacturers to
reduce both the number of parts and the number of suppli-
ers. Manufacturers that can purchase large quantities of a
few items rather than small quantities of many items are
able to take advantage of quantity discounts.
Aggregation of Data Regarding Retailing
Products
Retailers may carry thousands of products in their stores.
To effectively manage the logistics of so many products,
product aggregation is highly desirable. Products are
aggregated into families that have some similar character-
istics. Examples of product families include the following:
Products belonging to the same supplier
Products requiring special handling, transportation,
or storage
Products intended to be used or consumed by a
specific group of customers
Products intended to be used or consumed in a
specific season of the year ,
Volume and speed of product movement
The methods of the transportation of the products
from the suppliers to the retailers
The geographical location of suppliers
Method of transaction handling with suppliers; for
example, EDI, Internet, off-line
As in the case of GT, retailing products may belong
to more than one family.
Aggregation of Data Regarding
Customers of Finished Products
To effectively market finished products to customers, it
is helpful to aggregate customers with similar character-
istics into families. Examples of customers of finished
product families include
Customers residing in a specific geographical re-
gion
Customers belonging to a specific socioeconomic
group
Customers belonging to a specific age group
Customers having certain levels of education
Customers having similar product preferences
Customers of the same gender
Customers with the same household size
Similar to both GT and retailing products, customers
of finished products may belong to more than one family.
FUTURE TRENDS
Supply Chain Decision Databases
The enterprise database systems that support supply
chain management are repositories for large amounts of
transaction-based data. These systems are said to be
data rich but information poor. The tremendous amount
of data that are collected and stored in large, distributed
database systems has far exceeded the human ability for
comprehension without analytic tools. Shapiro estimates
that 80% of the data in a transactional database that
supports supply chain management is irrelevant to deci-
sion making and that data aggregations and other analy-
ses are needed to transform the other 20% into useful
information (2001). Data warehousing and online analyti-
cal processing (OLAP) technologies combined with tools
for data mining and knowledge discovery have allowed
the creation of systems to support organizational deci-
sion making.
325
TEAM LinG

The supply chain management (SCM) data warehouse
must maintain a significant amount of data for decision
making. Historical and current data are required from
supply chain partners and from various functional areas
within the firm in order to support decision making in
regard to planning, sourcing, production, and product
delivery. Supply chains are dynamic in nature. In a supply
chain environment, it may be desirable to learn from an
archived history of temporal data that often contains
some information that is less than optimal. In particular,
SCM environments are typically characterized by variable
changes in product demand, supply levels, product at-
tributes, machine characteristics, and production plans.
As these characteristics change over time, so does the
data in the data warehouses that support SCM decision
making. We should note that Kimball and Ross (2002) use
a supply value chain and a demand supply chain as the
framework for developing the data model for all business
data warehouses.
The data warehouses provide the foundation for de-
cision support systems (DSS) for supply chain manage-
ment. Analytical tools (simulation, optimization, & data
mining) and presentation tools (geographic information
systems and graphical user interface displays) are coupled
with the input data provided by the data warehouse
(Marakas, 2003). Simchi-Levi, D., Kaminsky, and Simchi-
Levi, E. (2000) describe three DSS examples: logistics
network design, supply chain planning and vehicle rout-
ing, and scheduling. Each DSS requires different data
elements, has specific goals and constraints, and utilizes
special graphical user interface (GUI) tools.
The Role of Radio Frequency
Identification (RFID) in Supply Chains
Data Warehousing
The emerging RFID technology will generate large amounts
of data that need to be warehoused and mined. Radio
frequency identification (RFID) is a wireless technology
that identifies objects without having either contact or
sight of them. RFID tags can be read despite environmen-
tally difficult conditions such as fog, ice, snow, paint, and
widely fluctuating temperatures. Optically read technolo-
gies, such as bar codes, cannot be used in such environ-
ments. RFID can also identify objects that are moving.
Passive RFID tags have no external power source.
Rather, they have operating power generated from a
reader device. The passive RFID tags are very small and
inexpensive. Further, they have a virtually unlimited op-
erational life. The characteristics of these passive RFID
tags make them ideal for tracking materials through sup-
ply chains. Wal-Mart has required manufacturers, suppli-
326
Data Warehousing and Mining in Supply Chains
ers, distributors, and carriers to incorporate RFID tags
into both products and operations. Other large retailers
are following Wal-Marts lead in requesting RFID tags to
be installed in goods along their supply chain. The tags
follow products from the point of manufacture to the store
shelf. RFID technology will significantly increase the
effectiveness of tracking materials along supply chains
and will also substantially reduce the loss that retailers
accrue from thefts. Nonetheless, civil liberty organiza-
tions are trying to stop RFID tagging of consumer goods,
because this technology has the potential of affecting
consumer privacy. RFID tags can be hidden inside objects
without customer knowledge. So RFID tagging would
make it possible for individuals to read the tags without
the consumers even having knowledge of the tags exist-
ence.
Sun Microsystems has designed RFID technology to
reduce or eliminate drug counterfeiting in pharmaceutical
supply chains (Jaques, 2004). This technology will make
the copying of drugs extremely difficult and unprofitable.
Delta Air Lines has successfully used RFID tags to track
pieces of luggage from check-in to planes (Brewin, 2003).
The luggage-tracking success rate of RFID was much
better than that provided by bar code scanners.
Active RFID tags, unlike passive tags, have an inter-
nal battery. The tags have the ability to be rewritten and/
or modified. The read/write capability of active RFID tags
is useful in interactive applications such as tracking work
in process or maintenance processes. Active RFID tags
are larger and more expensive than passive RFID tags.
Both the passive and active tags have a large, diverse
spectrum of applications and have become the standard
technologies for automated identification, data collec-
tion, and tracking. A vast amount of data will be recorded
by RFID tags. The storage and analysis of this data will
pose new challenges to the design, management, and
maintenance of databases as well as to the development
of data-mining techniques.
CONCLUSION
A large amount of data is likely to be gathered from the
many activities along supply chains. This data must be
warehoused and mined to identify patterns that can lead
to better management and control of supply chains. The
more RFID tags installed along supply chains, the easier
data collection becomes. As the tags become more popu-
lar, the data collected by them will grow significantly. The
increased popularity of the tags will bring with it new
possibilities for data analysis as well as new warehousing
and mining challenges.
TEAM LinG
Data Warehousing and Mining in Supply Chains
REFERENCES
Brewin, B. (2003, December). Delta says radio frequency
ID devices pass first bag. Computer World, 7. Retrieved
March 29, 2005 from www.computerworld.com/
mobiletopics/mobile/technology/story/0,10801,88446,
00/
Chen, R., Chen, C., & Chang, C. (2003). A Web-based ERP
data mining system for decision making. International
Journal of Computer Applications in Technology, 17(3),
156-158.
Davis, E. W., & Spekman, R. E. (2004). Extended enter-
prise. Upper Saddle River, NJ: Prentice Hall.
Dignan, L. (2003a, June 16). Data depot. Baseline Maga-
zine.
Dignan, L. (2003b, June 16). Lowes big plan. Baseline
Magazine.
Groover, M. P. (1987). Automation, production systems,
and computer-integrated manufacturing. Englewood
Cliffs, NJ: Prentice-Hall.
Hofmann, M. (2004, March). Best practices: VW revs its
B2B engine. Optimize Magazine, 22-30.
Jaques, R. (2004, February 20). Sun pushes RFID drug
technology. E-business/Technology/News. Retrieved
March 29, 2005, from www.vnunet.com/News/1152921
Kimball, R., & Ross, M. (2002). The data warehouse
toolkit (2nd ed.). New York: Wiley.
Levary, R. R. (1993). Group technology for enhancing the
efficiency of engineering activities. European Journal of
Engineering Education, 18(3), 277-283.
Levary, R. R. (2000, MayJune). Better supply chains
through information technology. Industrial Management,
42, 24-30.
Marakas, G. M. (2003). Modern data warehousing, min-
ing and visualization. Upper Saddle River, NJ: Prentice-
Hall.
Shapiro, J. F. (2001). Modeling the supply chain. Pacific
Grove, CA: Duxbury. ,
Simchi-Levi, D., Kaminsky, P., & Simchi-Levi, E. (2000).
Designing and managing the supply chain. Boston, MA:
McGraw-Hill.
Zeng, Y., Chiang, R., & Yen, D. (2003). Enterprise integra-
tion with advanced information technologies: ERP and
data warehousing. Information Management and Com-
puter Security, 11(3), 115-122.
KEY TERMS
Analytical Data: All data that are obtained from opti-
mization, forecasting, and decision support models.
Computer Aided Design (CAD): An interactive com-
puter graphics system used for engineering design.
Computer Aided Manufacturing (CAM): The use of
computers to improve both the effectiveness and effi-
ciency of manufacturing activities.
Decision Support System (DSS): Computer-based
systems designed to assist in managing activities and
information in organizations.
Graphical User Interface (GUI): A software interface
that relies on icons, bars, buttons, boxes, and other
images to initiate computer-based tasks for users.
Group Technology (GP): The concept of grouping
parts, resources, or data according to similar characteris-
tics.
Online Analytical Processing (OLAP): A form of
data presentation in which data are summarized, aggre-
gated, deaggregated, and viewed in the frame of a table or
cube.
Supply Chain: A series of activities that are involved
in the transformation of raw materials into a final product
that is purchased by a customer.
Transactional Data: All data that are acquired, pro-
cessed, and compiled into reports, which are transmitted
to various organizations along a supply chain.
327
TEAM LinG
328
Data Warehousing Search Engine
Hadrian Peter
University of the West Indies, Barbados
Charles Greenidge
University of the West Indies, Barbados
INTRODUCTION
Modern database systems have incorporated the use of
DSS (Decision Support Systems) to augment their deci-
sion-making business function and to allow detailed
analysis of off-line data by higher-level business man-
agers (Agosta, 2000; Kimball, 1996).
The data warehouse is an environment that is readily
tuned to maximize the efficiency of performing decision
support functions. However the advent of commercial uses
of the Internet on a large scale has opened new possibilities
for data capture and integration into the warehouse.
The data necessary for decision support can be di-
vided roughly into two categories: internal data and exter-
nal data. In this article, we focus on the need to augment
external data capture from Internet sources and provide a
tri-partite, high-level model termed the Data Warehouse
Search Engine (DWSE) model, to perform the same.
We acknowledge efforts that also are being made to
retrieve internal information from Web sources by use of
the Web warehouse, which stores the Web users mouse
and keyboard activities online, the so called clickstream
data. A number of Web warehouse initiatives have been
proposed, including WHOWEDA (Madria et al., 1999).
To be clear, data warehouses have focused on large
volumes of long-term historical data (a number of weeks,
months, or years old), but the presence of the Internet
with its data, which is short-lived and volatile, and im-
proved automation and integration activities make the
shorter time scales for the refresh cycle more attractive.
To attain the maximum benefits of the DWSE, signifi-
cant technical contributions that surpass existing imple-
mentations must be encouraged from the wider database,
information-retrieval, and search-engine technology com-
munities. Our model recognizes the fact that a cross-
disciplinary approach to research is the only way to guar-
antee further advancement in the area of decision support.
BACKGROUND
Data warehousing methodologies are concerned with
the collection, organization, and analysis of data taken
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
from several heterogeneous sources, all aimed at aug-
menting end-user business function (Berson & Smith,
1997; Inmon, 2003; Wixom & Watson, 2001).
Central to the use of heterogeneous data sources is
the need to extract, clean, and load data from a variety of
operational sources. Operational data not only need to
be cleaned by removing bad records or invalid fields, but
also typically must be put through a merge/purge pro-
cess that removes redundancies and records that are
inconsistent and lack integrity (Celko, 1995).
External data is key to business function and deci-
sion making, and includes sources of information such
as newspapers, magazines, trade publications, personal
contacts, and news releases. In the case where external
data is being used in addition to data taken from dispar-
ate operational sources, this external data may require a
similar cleaning/merge/purge process to be applied to
guarantee consistency (Higgins, 2003).
The World Wide Web represents a large and growing
source of external data but is notorious for the presence
of bad, unauthorized, or otherwise irregular data (Brake,
1997; Pfaffenberger, 1996). Thus, the need for cleaning
and integrity checking activities increases when the
Web is being used to gather external data (Sander-
Beuermann & Schomburg, 1998). The prevalence of
viruses and worms on the Internet, and the problems with
unsolicited e-mail (spam) show that communications
coming across the Internet must be filtered. The clean-
ing process may include pre-programmed rules to adjust
values, the use of sophisticated AI techniques, or simply
mechanisms to spot anomalies that then can be fixed
manually.
MAIN THRUST
Informed decision making in a data warehouse relies on
suitable levels of granularity for internal data, where
users can drill down from low to higher levels of
aggregation. External data in the form of multimedia
files, informal contacts, or news sources are often
marginalized (Inmon, 2002; Kimball, Barquin &
Edelstein, 1997) due to their unstructured nature.
TEAM LinG
Data Warehousing Search Engine
Figure 1. Merits of internal/external data
Internal Data
Trusted Sources
Known Structure
+ =
Known Frequency
Consistent
True decision making embraces as many pertinent
sources of information as possible so that a holistic
perspective of facts, trends, and individual pieces of
data can be obtained. Increasingly, the growth of com-
merce and business on the Internet has meant that in
addition to traditional modes of disseminating informa-
tion, the Internet has become a forum for ready posting
of information of all kinds (Hall, 1999).
The prevalence of so much information on the Internet
means that it is potentially a superior source of external
data for the data warehouse. Since such data typically
originate from a variety of sources (sites), it has to
undergo a merging and transformation process before it
can be used in the data warehouse. In the case of internal
data, which forms the core of the data warehouse, previ-
ous manual methods of applying ad-hoc tools and tech-
niques to the data cleaning process are being replaced by
more automated forms such as ETL (Extract, Trans-
form, Load). Although current ETL standardized pack-
ages are expensive, they offer productivity gains in the
long run (Earls, 2003; Songini, 2004).
The existence of the so-called invisible Web and
ongoing efforts to gain access to these untapped sources
suggest that the future of external data retrieval will
enjoy the same interest as that shown in internal data
(Inmon, 2002; Sherman & Price, 2003; Smith, 2001).
The need for reliable and consistent external data pro-
vides the motivation for an intermediate layer between
raw data gathered from the Internet and external data
storage areas lying within the domain of the data ware-
house (Agosta, 2000).
Until there is a maturing of widely available tools
with the ability to access the invisible Web, there will be
a continued reliance on information retrieval techniques,
as contrasted with data retrieval techniques, to gather
external data (van Rijsbergen, 1979). The need for three
environments to be present to process external data
from Web sources into the warehouse suggests a three-
tier solution to this problem. Accordingly, we propose
a tri-partite model called the Data Warehouse Search
,
Decision-Making
External Data Data
Unproven Sources Informative
Unknown Structure Challenging
Varying Frequency Wide in Scope
Unpredictable Balanced
Engine Model (DWSE), which has an intermediate data
extraction/cleaning layer functionally called the Meta-
Data Engine, sandwiched between the data warehouse
and search engine environments.
The DWSE Model
The data warehouse and search engine environments
serve two distinct and important roles at the current
time, but there is scope to utilize the strengths of both
in conjunction for maximum usefulness (Barquin &
Edelstein, 1997; Sonnenreich & Macinta, 1998). Our
proposed DWSE model seeks to allow cooperative links
between data warehouse and search engine with an aim
of satisfying external data requirements. The model
consists of (1) data warehouse (DW), (2) meta-data
engine (MDE) and (3) search engine (SE). The MDE is
the component that provides a bridge over which infor-
mation must pass from one environment to the other.
The MDE enhances queries coming from the warehouse
and also captures, merges, and formats information
returned by the search engine (Devlin, 1998).
The new model, through the MDE, seeks to augment
the operations of both by allowing external data to be
collected for the business analyst, while improving the
search engine searches through query modifications of
queries emerging from the data warehouse.
The generalized process is as follows. A query origi-
nates in the warehouse environment and is modified by
the MDE so that it is specific and free of nonsense
words. A word that has a high occurrence in a text but
conveys little specific information about the subject of
the text is deemed to be a nonsense word. Typically,
these words include pronouns, conjunctions, and proper
names. This term is synonymous with noise word, as
found in information retrieval texts (Belew, 2000). The
modified query is transmitted to the search engine that
performs its operations and retrieves its results docu-
ments. The documents returned are analyzed by the
MDE, and information is prepared for return to the
329
TEAM LinG

Figure 2. The DWSE model
Data Warehouse Meta-Data Engine Search Engine
Enterprise Wide
S o u r ces Extract/
Transform SE
Meta
Data
Storage
SE Files
DW
Extraction/
Cleaning DBMS
DSS Tool
End User
warehouse. Finally, the information relating to the answer
to the query is returned to the warehouse environment.
The entire process may take days or weeks, as both
warehouse and search engine operate independently ac-
cording to their own schedules.
The design of both search engine and data warehouse
are skill-intensive and specialized tasks. Prudent judg-
ment dictates that nothing should be done to add to the
burden of building and maintaining these complex sys-
tems. History shows that many information technology
(IT) projects fail when expediency overtakes deliberate
design. In Figure 2, we see the cooperation among com-
ponents of the DWSE.
The DWSE model considers only the external data
requirements of the data warehouse. The SE component
performs actual searches but no analysis of documents.
The indexing of the retrieved documents and other analy-
sis is done by the MDE.
Meta-Data Engine Model
To cope with the radical differences between the SE and
DW designs, we propose a meta-data engine to coordi-
nate all activities. Typical commercial SEs are com-
posed of a crawler (spider) and an indexer (mite). The
indexer is used to codify results into a database for easy
querying. Our approach reduces the complexity of the
search engine by moving the indexing tasks to the meta-
data engine. The meta-data engine seeks to form a bridge
between the diverse SE and DW environments. The pur-
pose of the meta-data engine is to facilitate an automatic
information retrieval mode. Figure 3 illustrates the con-
cept of the bridge between DW and SE environments, and
Figure 4 outlines the main features of the MDE.
The following are some of the main functions of the
meta-data engine:
Capture and transform data arising from SE (e.g., handle
HTML, XML, .pdf, and other document formats).
330
Data Warehousing Search Engine
Figure 3. Bridging the architectures: The meta-data
engine
Web
I Meta-Data
n Engine
t
e
r
n
e
t
Data Hybrid Search
Warehouse Engine
Internal Data Internet
Index words retrieved during SE crawls.
Manage the scheduling and control of tasks aris-
ing from the operation of both DW and SE.
Provide a neutral platform so that SE and DW
operational cycles and architectures cannot in-
terfere with each other.
Track meta-data that may be used to enhance the
quality of queries and results in the future (e.g.,
monitor the use of domain-specific words/phrases
such as jargon for a particular domain).
Of major interest in Figure 4 is the automatic infor-
mation retrieval (AIR) component. AIR seeks to high-
light the existence of documents that are related to a
given query (Belew, 2000) using a process that is differ-
ent from data retrieval. Data retrieval (van Rijsbergen,
1979), as used in relation to database management sys-
tems (RDBMSs) requires an exact match of terms, uses
an artificial query language (e.g., SQL), and uses deduc-
tion. Information retrieval, on the other hand (as is
common for Internet searches), uses partial matching
and a natural query language, and inductively produces
relevant results (Spertus & Stein, 1999). This idea of
relevance distinguishes the IR search from the normal
data retrieval where a matching of terms is done.
ANALYSIS OF MODEL
Strengths
The major strengths of this model are:
Independence
This model carries logical independence. There is a
deliberate attempt to ensure that the best practices of
each individual model is adhered to by insistence that
TEAM LinG
Data Warehousing Search Engine
Figure 4. Characteristics of the meta-data engine
Meta-Data Engine
Operate in AIR
Mode
Process Queries
Emerging From
Data Warehouse
Provide Neutral
Platform
Process Meta-Data
there be logical independence in the design. This is very
important, since many IT projects fail when good design
is compromised.
This model carries physical independence. There is
no need to overburden the capacities of existing ware-
house projects by insistence on the simultaneous stor-
age of vast quantities of external data. The model allows
for the transfer of summarized results via network con-
nections, while the physical machines may reside con-
tinents apart.
This model carries administrative independence.
There is the implicit expectation that the three separate
subsystems comprising the model can and will be under
the control of three separate administrators. This is
important because warehouse administration is a spe-
cialist area, as is search engine maintenance. By allow-
ing specialists to focus on their respective disciplines,
the chances for effective administration are enhanced.
Value-Added Approach
This model complements the concept of maximizing the
utility of acquired data by adding value to this data even
long after its day-to-day usefulness has expired. The
ability to add pertinent external data to volumes of aging
internal data extends the horizons for the usefulness of
the internal data indefinitely.
Adaptability
The new model provides for scalability, as the logical/
physical independence is not restricted to any one sys-
tem or architecture. Databases in any sector of the
model can be provided for independently without nega-
tively impacting the other parts of the system. The new
model also allows for flexibility by allowing develop-
ment to take place in three languages. For example,
when there is a query language for the warehouse, perl ,
for the meta-data engine, and java for the search engine,
the strengths of each can be utilized effectively.
The new model allows for security and integrity to
be maintained. The warehouse need not expose itself to
dangerous integrity questions or to increasingly mali-
cious threats on the Internet. By isolating the sector that
interfaces with the Internet from the sector that carries
vital internal data, the prospects for good security are
improved.
Relieves Information Overload
By seeking to remove, repackage, and reprocess exter-
nal data, we are indirectly removing time-consuming
filtering activities by employees and busy executives.
There is a growing need to protect users from being
overwhelmed by increasing volumes of data. In the
future, we will rely on smart, accurate, automated sys-
tems to guide us through the realms of extraneous data
to the cores of knowledge.
Potential Weaknesses
Relies on fast-changing search engine technolo-
gies. Improvements or modifications to the way
information formatted and accessed on the Internet
will impact on the search and retrieval process.
Introduces the need for more system overhead and
administrators. A warehouse administrator, search
engine administrator, and information retrieval
expert (meta-data engine) will have to be con-
tracted. The systems themselves will have to be
configured so that cooperation among them re-
mains easy.
Storage of large volumes of irrelevant informa-
tion is a problem. The cost of maintaining and
indexing unmanageable quantities of Internet data
may discount the benefits derived from analysis of
the external data. Figure 5 highlights possible
drawbacks to using the proposed model.
FUTURE TRENDS
The future is notoriously unpredictable, yet the follow-
ing trends can be seen readily for the next decade.
1. Maturing of the data warehouse tools and tech-
niques (Agosta, 2000; Greenfield, 1996; Inmon,
2002; Schwartz, 2003).
2. Increases in the volumes of data being stored
online and off-line.
331
TEAM LinG

Figure 5. Hidden dangers in DWSE model
B D
A meta-data engine) as an integral part. Current data ware-
Large volumes of redundant/unusable data may be stored
End-user analysis may become skewed by wrong use of external data
Validity and accuracy of external data not always known, so there
are risks involved in using the external data.
Figure 8.2 Hidden dangers in DWSE model
3. Central importance of the Internet and its related
technologies (Goodman, 2000).
4. Increasing sophistication and focus of Web sites.
5. Increased need to automate analysis and decision
support on the growing volumes of data; both
internal and external to the organization.
6. Adoption of specialized search engines (Raab,
1999).
7. Adoption of specialized sites (vortals) catering to
a particular category of user (Rupley, 2000).
8. Ability to mine invisible Web, increasing poten-
tial benefits from a data warehouse search engine
model (Sherman & Price, 2003; Smith, 2001;
Soudah, Sullivan, 2000).
The DWSE model is poised to gain prominence as
practitioners realize the benefits of three distinct and
architecturally independent layers. We are confident
that with vital and open architectures and models of
clarity, data warehousing efforts can be successful in
the long term.
CONCLUSION
More research is needed to investigate how to enable
the data warehouse to take advantage of the growing
external data stores on the Internet. Comparison of
results obtained using the three-tiered approach of the
DWSE model, as compared with the traditional external
data approaches, must be carried out. Unclear also is
how the traditional Online Transaction Processing
(OLTP) systems will funnel data toward this model. If
operational systems start to retain significant quantities
of external data, it may mitigate against this model as
more and more external data will be available in the
warehouse through normal integration processes. On
the other hand, the costs of dealing with such external
data in the warehouse may be so prohibitive that an
332
Data Warehousing Search Engine
alternate path into the meta-data engine of the new model
may be required.
In the case of the second outcome, a new model may
be proposed with this process (OLTP external data to
house models already allow for the establishment of
data marts and querying by external tools. If these new
processes are considered desirable components of the
data warehousing process, then a new data warehouse
model could be proposed.
Relevance remains a thorny issue. The problem of
providing relevant data without anyone to manually verify
its relevance is challenging. Semantic Web developments
promise to address questions of relevance on the Internet.
REFERENCES
Agosta, L. (2000). The essential guide to data ware-
housing. New Jersey: Prentice-Hall.
Barquin, R., & Edelstein, H. (Eds.). (1997). Planning
and designing the data warehouse. New Jersey:
Prentice-Hall.
Belew, R.K. (2000). Finding out about: A cognitive
perspective on search engine technology and the
WWW. New York: Cambridge University Press.
Berson, A., & Smith, S.J. (1997). Data warehousing,
data mining and olap. New York: McGraw-Hill.
Brake, D. (1997). Lost in cyberspace [Electronic ver-
sion]. New Scientist.
Celko, J. (1995). Dont warehouse dirty data.
Datamation, 42-53.
Devlin, B. (1998). Meta-data: The warehouse atlas. DB2
Magazine, 3(1), 8-9.
Earls, A.R. (2003). ETL: Preparation is the best bet.
Computerworld, 37(34), 25-27.
Goodman, A. (2000). Searching for a better way [Elec-
tronic version].
Greenfield, L. (1996). Dont let data warehousing
gotchas getcha. Datamation, 76-77.
Hall, C. (1999). Enterprise information portals: Hot air
or hot technology [Electronic version]. Business Intel-
ligence Advisor, 111(11).
Higgins, K.J. (2003). Warehouse data earns its keep.
Network Computing, 14(8), 111-115.
Inmon, W.H. (2002). Building the data warehouse.
New York: John Wiley & Sons.
TEAM LinG
Data Warehousing Search Engine
Inmon, W.H. (2003). The story so far. Computerworld,
37(15), 26-27.
Kimball, R. (1996). Dangerous preconceptions [Electronic
version].
Kimball, R. (1997). A dimensional modeling manifesto
[Electronic version]. DBMS Magazine.
Madria, S.K. et al. (1999). Research issues in Web data
mining. Proceedings of Data Warehousing and Knowl-
edge Discovery, First International Conference.
Pfaffenberger, B. (1996). Web search strategies. New
York: MIS Press.
Raab, D.M. (1999). Enterprise information portals [Elec-
tronic version]. Relationship Marketing Report.
Rupley, S. (2000). From portals to vortals. PC Magazine.
Sander-Beuermann, W., & Schomburg, M. (1998).
Internet information retrievalThe further development
of meta-search engine technology. Proceedings of the
Internet Summit, Internet Society, Geneva, Switzerland.
Schwartz, E. (2003). Data warehouses get active.
InfoWorld, 25(48), 12-13.
Sherman, C., & Price, G. (2003). The invisible Web:
Uncovering sources search engines cant see. Library
Trends, 52(2), 282-299.
Smith, C.B. (2001). Getting to know the invisible Web.
Library Journal, 126(11), 16-19.
Songini, M.L. (2004). ETL. Computerworld, 38(5), 23-24.
Sonnenreich, W., & Macinta, T. (1998). Web
developer.com guide to search engines. New York:
Wiley Computer Publishing.
Soudah, T. (2000). Search, and you shall find [Electronic
version].
Spertus, E., & Stein, L.A. (1999). Squeal: A structured
query language for the Web [Electronic version].
Sullivan, D. (2000). Invisible Web gets deeper [Elec-
tronic version]. The Search Engine Report.
Van Rijsbergen, C.J. (1979). Information retrieval [Elec-
tronic version]. In Finding out about. [CD-ROM]. Rich-
ard Belew.
Wixom, B.H., & Watson, H.J. (2001). An empirical inves-
tigation of the factors affecting data warehousing suc- ,
cess. MIS Quarterly, 25(1), 17-39.
KEY TERMS
Data Retrieval: Denotes the standardized database
methods of matching a set of records, given a particular
query (e.g., use of the SQL SELECT command on a
database.
Decision Support System (DSS): An interactive
arrangement of computerized tools tailored to retrieve
and display data regarding business problems and queries.
ETL (Extract/Transform/Load): This term speci-
fies a category of software that efficiently handles three
essential components of the warehousing process. First,
data must be extracted (removed from originating sys-
tem), then transformed (reformatted and cleaned), and
third, loaded (copied/appended) into the data warehouse
database system.
External Data: A broad term indicating data that is
external to a particular company. Includes electronic
and non-electronic formats.
Information Retrieval: Denotes the attempt to
match a set of related documents to a given query using
semantic considerations (e.g., library catalogue sys-
tems often employ information retrieval techniques).
Internal Data: Previously cleaned warehouse data
that originated from the daily information processing
systems of a company.
Invisible Web: Denotes those significant portions
of the Internet where data is stored which are inacces-
sible to the major search engines. The invisible Web
represents an often ignored/neglected source of poten-
tial online information.
Metadata: Data about data; in the data warehouse, it
describes the contents of the data warehouse.
333
TEAM LinG
334

Data Warehousing Solutions for Reporting

Problems
Juha Kontio
Turku Polytechnic, Finland

INTRODUCTION tems, such as order-entry systems. The second largest

Reporting is one of the basic processes in all organiza-
tions. It provides information for planning and decision
making and, on the other hand, information for analyzing
the correctness of the decisions made at the beginning
of the process. Reporting is based on the data that the
operational information systems contain. Reports can
be produced directly from these operational databases,
but an operational database is not organized in a way that
naturally supports analysis. An alternative way is to
organize the data in such a way that supports analysis
easily. Typically, this method leads to the introduction
of a data warehouse.
In the summer of 2002, a multiple case study re-
search was launched in six Finnish organizations (see
Table 1). The researchers studied the databases of these
organizations and identified the trends in database ex-
ploitation. One of the main ideas was to study the
diffusion of database innovations. In practice this meant
that the researchers described the present database ar-
chitecture and identified the future plans and present
problems. The data for this research was mainly col-
lected with semistructured interviews, and altogether,
54 interviews were arranged.
The research processed data of 44 different infor-
mation systems. Most (40%) of the analyzed informa-
tion systems were online transaction processing sys-
category (30%) comprised information systems relat-
ing to decision support and reporting. Only one pilot
data warehouse was among these systems, but on the
other hand, customized reporting systems were used,
for example, in SOK, SSP, and OPTI. Reporting was
commonly recognized as an area where interviewees
were not satisfied and were hoping for improvements.
This article focuses on describing the reporting
problems that the organizations are facing and explains
how they can exploit a data warehouse to overcome
these problems.
BACKGROUND
The term data warehouse was first introduced as a
subject-oriented, integrated, nonvolatile, and time-vari-
ant collection of data in support of managements deci-
sions (Inmon, 1992). A simpler definition says that a
data warehouse is a store of enterprise data designed to
facilitate management decision making (Kroenke,
2004). A data warehouse differs from traditional data-
bases in many ways. Its structure is different than that of
traditional databases, and different functionalities are
required (Elmasri & Navathe, 2000). The aim is to
integrate all corporate information into one repository,
where the information is easily accessed, queried, ana-

Table 1. The case organizations

Organization/ Line of Business Private/ Turnover Employees

Abbreviation Used Public 2002 2002
in This Research (Millions )
SOK Co-operative society (main Private 2998 4645
Corporation/SOK businesses: food and groceries,
hardware)
Salon Seudun Telecommunication Private 28 121
Puhelin, Ltd./SSP
Statistics National statistics Public 52 1074
Finland/STAT
State Provincial Regional administrative Public 350
Office of Western authority
Finland/WEST
TS-Group, Ltd.,/TS Printing services and Private 69.7 2052
communications (Consolidated
corporation)
Optiroc, Ltd./OPTI Building materials Private 149 388

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
Data Warehousing Solutions for Reporting Problems
lyzed, and used as a basis for the reports (Begg & Connolly,
2002). A data warehouse provides decision support to
organizations with the help of analytical databases and
Online Analytical Processing (OLAP) tools (Gorla, 2003).
A data warehouse (see Figure 1) receives data from the
operational databases on a regular basis, and new data is
added to the existing data. The warehouse contains both
detailed aggregated data and summarized data to speed
up the queries. It is typically organized in smaller units
called data marts, which support the specific analysis
needs of a department or business unit (Bonifati, Cattaneo,
Ceri, Fuggetta, & Paraboschi, 2001).
In the case organizations, the idea of the data ware-
house has been discussed, but so far no data warehouses
exist, although in one case, a data warehouse pilot is in
use. The rationale for these discussions is that at the
moment, the reporting and the analyzing possibilities
are not serving the organizations very well. Actually, the
interviewees identified many problems in reporting.
In the SOK Corporation, the interviewees complained
that information is distributed in numerous information
systems; thus, building a comprehensive view of the
information is difficult. Another problem is in financial
reporting. A financial report taken from different infor-
mation systems gives different results, though they
should be equal. A reason for this inequality is that the
data is not harmonized and processed similarly. In the
restaurant business of SOK Corporation, an essential
piece of information is the sales figures of the products.
It should be able to analyze which, where, and how many
products have been bought. In the whole SOK Corpora-
tion, analyzing different customers and their behavior in
detail is, at the moment, impossible. The interviewees
also mentioned that a common database containing all
products of the co-operative society might help in re-
porting, but defining a common classification of the
Figure 1. Data warehousing
Operational Valuable information for processes
databases
Data
Warehouse
Summarized
Cleaning
Reformatting
data
Analytical
Detailed tools
data
Data
Data
Other
Data Filter Data
Mart
Mart
Inputs Mart
products will be a demanding task. Centralization of the
data is also one topic that has been discussed in SOK ,
Corporation, which has been justified with improve-
ments in reporting.
In Salon Seudun Puhelin, Ltd., the interviewees men-
tioned that the major information system is somehow
used inconsistently. Therefore, the data is not consis-
tent and influences the reporting. This company has
developed its own reporting application with Microsoft
Access, but the program is not capable of managing files
over 1 GB, which reduces the possibilities of using the
system. According to the interviewees, this limit pre-
vents, for example, the follow-up of daily sales. Another
problem concerning the reporting system is that users
are incapable of defining their own reports when their
needs change. Analyzing customer data is also difficult,
because collecting all customer data together is a very
burdensome task. Therefore, Salon Seudun Puhelin, Ltd.,
has also discussed a data warehouse solution for three
reasons: a) to get rid of the size limits, b) to provide a
system for users where they can easily define new
reports, and c) to gain more versatile analysis possibili-
ties.
The State Provincial Office of Western Finland is a
joint regional administrative authority of seven minis-
tries. One of their yearly responsibilities is to evaluate
the basic service in their region. In practice, this respon-
sibility means that they gather and analyze a large amount
of data. The first problem is that they have not used a
special data management tool. The lack of an adequate
tool for data management makes it difficult to do any
time-series analysis, which many of the interviewees
hoped for. Another problem is that the results should be
easily distributed in forms of different reports, but at
the moment, this is not the case.
In TS-Group, Ltd., a data warehouse pilot has been
implemented. The pilot enables versatile reporting and,
as the interviewees mentioned, this opportunity should
not be lost. However, some reporting problems still
exist. For example, the distribution and the format of the
reports should be solved. In the department of financial
management, the reporting system does not support the
latest operating systems and, therefore, only some com-
puters are capable of using the system.
In Optiroc, Ltd., reports are generated directly from
the operational databases that are not designed for re-
porting purposes. One problem attendant on this design
is that the reports run slowly. In principle, users should
also be able to create their reports by themselves, but in
reality, only a few of them are able to. Maybe this is one
reason that the interviewees presented very critical
comments on reporting. The interviewees mentioned as
well that the implementation of a data warehouse system
is strongly supported and is seen as a solution for prob-
335
TEAM LinG

lems in reporting. A data warehouse was also justified
because, with it, the company could serve customers
better and could produce customized reports. At the
moment, customer reporting is analyzed to develop re-
porting and to define necessary tools.
MAIN THRUST
The case organizations have plans to start exploiting a
data warehouse, but before the introduction of a data
warehouse, plenty of design must be accomplished in all
these cases. Designing a data warehouse requires quite
different techniques than the design of an operational
database (Golfarelli & Rizzi, 1998). Modeling a data
model for a data warehouse is seen as one of the most
critical phases in the development process (Bonifati et
al., 2001). This data modeling has specific features that
distinguish it from normal data modeling (Busborg,
Christiansen, & Tryfona, 1999). At the beginning, the
content of the operational information systems, the in-
terconnections between them, and the equivalent entities
should be understood (Blackwood, 2000). In practice,
this entails studying the data models of the operational
databases and developing an integrated schema to en-
hance the data interoperability (Bonifati et al., 2001).
The data modeling of a data warehouse is called dimen-
sionality modeling (DM) (Golfarelli & Rizzi, 1998;
Begg & Connolly, 2002). Dimensional models were
developed to support analytical tasks (Loukas & Spen-
cer, 1999). Dimensionality modeling concentrates on
facts and the properties of the facts and dimensions
connected to facts (Busborg et al., 1999). Facts are
numeric, and quantitative data of the business and dimen-
sions describe different dimensions of the business
(Bonifati et al., 2001). Fact tables contain all the busi-
ness events to be analyzed, and dimension tables define
how to analyze fact information (Loukas & Spencer,
1999). The result of the dimensionality modeling is
typically presented in a star model or in a snowflake
model (Begg & Connolly, 2002). Multidimensional
schema (MDS) is a more generic term that collectively
refers to both schemas (Martyn, 2004). When a star
model is used, the fact tables are normalized, but dimen-
sion tables are not. When dimension tables are normal-
ized too, the star model turns into a snowflake model
(Bonifati et al., 2001).
Ideally, an information system such as a data ware-
house should be correct, fast, and friendly (Martyn,
2004). Correctness is especially important in data ware-
houses to ensure that decisions are based on accurate
information. Actually, an estimated 30% to 50% of
information in a typical database is either missing or
incorrect (Blackwood, 2000). This idea emphasizes the
336
Data Warehousing Solutions for Reporting Problems
need to pay attention to the quality of the source data in
the operational databases (Finnegan & Sammon, 2000).
One problem with MDS is that when the business envi-
ronment changes, the evolution of multidimensional
schemas is not as manageable as with normalized schemas
(Martyn, 2004). It is also said that any architecture not
based on third normalized form can cause the failure of
a data warehouse project (Gardner, 1998). On the other
hand, a dimensional model provides a better solution for
a decision support application than a pure normalized
relational model does (Loukas & Spencer, 1999). All of
the above is actually related to efficiency, because a large
amount of data is processed during analysis. Typically,
this is a question about needed joins in the database
level. Usually, a star schema is the most efficient design
for a data warehouse, because the denormalized tables
require fewer joins (Martyn, 2004). However, recent de-
velopments in storage technology, access methods (such
as bitmap indexes), and query optimization indicate that
the performance with the third normalized form should be
tested before moving to multidimensional schemas
(Martyn, 2004). From this 3NF schema, a natural step
toward MDS is to use denormalization, which will
support both efficiency and flexibility issues (Finnegan
& Sammon, 2000). Still, it is possible to define neces-
sary SQL views on top of the 3NF schema without
denormalization (Martyn, 2004). Finally, during the
design, the issues of physical and logical design should
be separated; physical design is about performance, and
logical design is about understandability (Kimball,
2001).
The OLAP tools, which are based on MDS views,
access data warehouses for complex data analysis and
decision support activities (Kambayashi, Kumar,
Mohania, & Samtani, 2004). These tools typically in-
clude assessing the effectiveness of a marketing cam-
paign, forecasting product sales, and planning capacity.
The architecture of the underlying database of the data
warehouse categorizes the different analysis tools
(Begg & Connolly, 2002). Depending on the schema
type, the terms Relational OLAP (ROLAP), Multidi-
mensional OLAP (MOLAP), and Hybrid OLAP (HOLAP)
are used (Kroenke, 2004). ROLAP is a preferable
choice when a) the information needs change frequently,
b) the information should be as current as possible, and
c) the users are sophisticated computer users (Gorla,
2003). The main differences between ROLAP and
MOLAP are in the currency of data and in the data
storage processing capacity. MOLAP populates its own
structure of the original data when it is loaded from the
operational databases (Dodds, Hasan, Hyland, &
Veeraraghavan, 2000). In MOLAP, the data is stored in
a special-purpose MDS (Begg & Connolly, 2002). ROLAP,
on the other hand, analyzes the original data, and the
TEAM LinG
Data Warehousing Solutions for Reporting Problems
users can drill down to the unit data level (Dodds et al.,
2000). ROLAP uses a meta-data layer to avoid the creation
of an MDS. It typically utilizes the SQL extensions, such
as CUBE and ROLLUP in Oracle DBMS (Begg & Connolly,
2002).
FUTURE TRENDS
In the future, the SOK Corporation needs to build a
comprehensive view of the information. They can
achieved this goal first by building an enterprise data
model and then by modifying the existing information
systems. From the reporting point of view, the opera-
tional data should be further modeled into a data ware-
house solution. After these steps, the problems relating
to reporting should be solved.
In order to improve the consistency of the data at
Salon Seudun Puhelin, Ltd., most concerns need to be
placed on the correctness of the data and the ways users
work with the information systems. Until then, exploit-
ing a data warehouse is not rational. For this company, it
is reasonable to evaluate the true requirements of a data
warehouse for reporting, because other solutions that
are easier than starting a data warehouse project might
be on hand.
In the State Provincial Office of Western Finland
(WEST), the most important issue is acquiring a suitable
tool for managing the collected data. After that step, the
emphasis can shift to reporting, analyzing, and in time-
series. Basically, the environment of WEST is ideal for
exploiting a data warehouse. The processes and the
functions are heavily dependent on the analysis and the
developments in time series.
In TS-Group, Ltd., a data warehouse could solve the
problems in reporting as well. Introducing a data ware-
house would define the standard formats of the reports,
which are currently causing a problem. At the same time,
the distribution problems of the reports would be solved.
In Optiroc, Ltd., a data warehouse can solve the
slowness of running reports and the difficulties in analy-
sis. One reason for the slowness is that the reports run
directly from the operational databases; moving the
origin of the data to a data warehouse might offer
improvements. Another problem deals with the analysis
and the possibilities to define necessary reports on the
fly. Introducing necessary OLAP tools and training the
users sufficiently will solve this problem.
In Statistics Finland, the interviewees mentioned
that reporting is not a problem. However, a data ware-
house might offer extra value in data analysis. At the
moment, though, data warehousing is not the most acute
topic in information technology development in Statis-
tics Finland.
The presented cases reflect the overall situation in
different organizations quite well and might predict ,
what will happen in the future. The cases show that
organizations have clear problems in analyzing the op-
erational data. They have developed their own applica-
tions to ease the reporting but are still living with
inadequate solutions. Data warehousing has been identi-
fied as a possible solution for the reporting problems,
and the future will show whether data warehouses dif-
fuse in the organizations.
CONCLUSION
The case organizations have recognized the possibilies
of a data warehouse to solve problems in reporting.
Their initiatives are based on business requirements,
which is a good starting point because to be successful,
a data warehouse should be a business-driven initiative
in partnership with the information technology depart-
ment (Gardner, 1998; Finnegan & Sammon, 2000).
However, the first step should be the analysis of the real
needs of a data warehouse. At the same time, the present
problems in reporting should be solved if possible.
To really support reporting, the operational data
should be organized like that in a data warehouse. In
practice, this means studying and analyzing the existing
operational databases. As a result, a data model describ-
ing the necessary elements of the data warehouse should
be achieved. A suggestion is to first produce an enter-
prise level data model to describe all the data and their
interconnections. This enterprise level data model will
be the basis for the data warehouse design. As the theory
suggested at the beginning of this article, a normalized
data model with SQL views should be produced and
tested. This model can further be denormalized into a
snowflake or a star model when performance require-
ments are not met with the normalized schema.
Producing a data model for the data warehouse is
only one part of the process. In addition, special atten-
tion should be taken in designing data extraction from
operational databases to the data warehouse. The enter-
prise level data model helps these databases understand
the data and thus eases the development of data extrac-
tion solutions. When the data warehouse is in use,
automated tools are necessary to speed up the loading of
the operational data into the data warehouse (Finnegan
& Sammon, 2000). Before loading data into the data
warehouse, the organizations should also analyze the
correctness and the quality of the data in the operational
databases.
Finally, as this article shows, a data warehouse is a
relevant alternative for solving reporting problems that
337
TEAM LinG

the case organizations are currently facing. After the
implementation of a data warehouse, organizations must
ensure that the possible users of the system are edu-
cated in order to fully take advantage of the new possi-
bilities that the data warehouse offers. Of course, orga-
nizations should remember that there is no quick jump
to data warehouse exploitation.
REFERENCES
Begg, C., & Connolly, T. (2002). Database systems: A
practical guide to design, implementation, and man-
agement. Addison-Wesley.
Blackwood, P. (2000). Eleven steps to success in data
warehousing. Business Journal, 14(44), 26-27.
Bonifati, A., Cattaneo, F., Ceri, S., Fuggetta, A., &
Paraboschi, S. (2001). Designing data marts for data
warehouses. ACM Transactions on Software Engineer-
ing and Methodology, 10(4), 452-483.
Busborg, F., Christiansen, J. G. B., & Tryfona, N. (1999).
StarER: A conceptual model for data warehouse design.
Proceedings of the ACM International Workshop on
Data Warehousing and OLAP, USA.
Dodds, D., Hasan, H., Hyland, P., & Veeraraghavan, R.
(2000). Approaches to the development of multidimen-
sional databases: Lessons from four case studies. 31(3),
10-23. Retrieved from the ACM SIGMIS database.
Elmasri, R., & Navathe, S. B. (2000). Fundamentals of
database systems. Reading, MA: Addison-Wesley.
Finnegan, P., & Sammon, D. (2000). The ten command-
ments of data warehousing. 31(4), 82-91. Retrieved from
the ACM SIGMIS database.
Gardner, S. R. (1998). Building the data warehouse.
Communications of the ACM, 41(9), 52-60.
Golfarelli, M., & Rizzi, S. (1998). A methodological
framework for data warehouse design. Proceedings of
the ACM International Workshop on Data Warehous-
ing and OLAP, USA.
Gorla, N. (2003). Features to consider in a data ware-
housing system. Communications of the ACM, 46(11),
111-115.
338
Data Warehousing Solutions for Reporting Problems
Inmon, W. H. (1992). Building the data warehouse. New
York: Wiley.
Kambayashi, Y., Kumar, V., Mohania, M., & Samtani, S.
(2004). Recent advances and research problems in data
warehousing. Lecture Notes in Computer Science, 1552,
81-92.
Kimball, R. (2001). A trio of interesting snowflakes.
Intelligent Enterprise, 4, 30-32.
Kroenke, D. M. (2004). Database processing: Funda-
mentals, design and implementation. Upper Saddle
River, NJ: Pearson Prentice Hall.
Loukas, T., & Spencer, T. (1999, October). From star to
snowflake to ERD: Comparing data warehouse design
approaches. Enterprise Systems.
Martyn, T. (2004). Reconsidering multi-dimensional
schemas. ACM SIGMOD Record, 33(1), 83-88.
KEY TERMS
Data Extraction: A process in which data is trans-
ferred from operational databases to a data warehouse.
Dimensionality Modeling: A logical design tech-
nique that aims to present data in a standard, intuitive
form that allows for high-performance access.
Normalization/Denormalization: Normalization
is a technique for producing a set of relations with
desirable properties, given the data requirements of an
enterprise. Denormalization is a step backward in the
normalization process for example, to improve per-
formance.
OLAP: The dynamic synthesis, analysis, and con-
solidation of large volumes of multidimensional data.
Snowflake Model: A variant of the star schema, in
which dimension tables do not contain denormalized
data.
Star Model: A logical structure that has a fact table
containing factual data in the center, surrounded by
dimension tables containing reference data.
TEAM LinG
 339

Database Queries, Data Mining, and OLAP ,

Lutz Hamel
University of Rhode Island, USA

INTRODUCTION definition is routinely accomplished via the data defini-

Modern, commercially available relational database
systems now routinely include a cadre of data retrieval
and analysis tools. Here we shed some light on the
interrelationships between the most common tools and
components included in todays database systems: query
language engines, data mining components, and online
analytical processing (OLAP) tools. We do so by pair-
wise juxtaposition, which will underscore their differ-
ences and highlight their complementary value.
tion language (DDL) facilities of SQL by specifying
either a star or snowflake schema (Kimball, 1996).
MAIN THRUST
The following sections contain the pair wise compari-
sons between the tools and components considered in
this chapter.
Database Queries vs. Data Mining

BACKGROUND Virtually all modern, commercial database systems are

Todays commercially available relational database sys-
tems now routinely include tools such as SQL database
query engines, data mining components, and OLAP
(Craig, Vivona, & Bercovitch, 1999; Oracle, 2001;
Scalzo, 2003; Seidman, 2001). These tools allow devel-
opers to construct high powered business intelligence
(BI) applications which are not only able to retrieve
records efficiently but also support sophisticated analy-
ses such as customer classification and market segmen-
tation. However, with powerful tools so tightly inte-
grated with the database technology, understanding the
differences between these tools and their comparative
advantages and disadvantages becomes critical for ef-
fective application development. From the practitioners
point of view questions like the following often arise:
Is running database queries against large tables
considered data mining?
Can data mining and OLAP be considered synony-
mous?
Is OLAP simply a way to speed up certain SQL
queries?
based on the relational model formalized by Codd in the
1960s and 1970s (Codd, 1970) and the SQL language
(Date, 2000) which allows the user to efficiently and
effectively manipulate a database. In this model a data-
base table is a representation of a mathematical relation,
that is, a set of items that share certain characteristics or
attributes. Here, each table column represents an at-
tribute of the relation and each record in the table
represents a member of this relation. In relational data-
bases the tables are usually named after the kind of
relation they represent. Figure 1 is an example of a table
that represents the set or relation of all the customers of
a particular store. In this case the store tracks the total
amount of money spent by its customers.
Relational databases do not only allow for the cre-
ation of tables but also for the manipulation of the tables
and the data within them. The most fundamental opera-
tion on a database is the query. This operation enables
the user to retrieve data from database tables by assert-
ing that the retrieved data needs to fulfill certain crite-
ria. As an example, consider the fact that the storeowner
might be interested in finding out which customers
spent more than $100 at the store. The following query

The issue is being complicated even further by the

Figure 1. A relational database table representing
fact that data analysis tools are often implemented in
customers of a store
terms of data retrieval functionality. Consider the data
mining models in the Microsoft SQL server which are
Id Name ZIP Sex Age Income Children Car Total
implemented through extensions to the SQL database Spent
query language (e.g., predict join) (Seidman, 2001) or 5 Peter 05566 M 35 $40,000 2 Mini $250.00
Van
the proposed SQL extensions to enable decision tree
classifiers (Sattler & Dunemann, 2001). OLAP cube 22 Maureen 04477 F 26 $55,000 0 Coupe $50.00

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG

returns all the customers from the above customer table
that spent more than $100:
SELECT * FROM CUSTOMER_TABLE WHERE
TOTAL_SPENT > $100;
This query returns a list of all instances in the table
where the value of the attribute Total Spent is larger
than $100. As this example highlights, queries act as
filters that allow the user to select instances from a
table based on certain attribute values. It does not matter
how large or small the database table is, a query will
simply return all the instances from a table that satisfy
the attribute value constraints given in the query. This
straightforward approach to retrieving data from a data-
base has also a drawback. Assume for a moment that our
example store is a large store with tens of thousands of
customers (perhaps an online store). Firing the above
query against the customer table in the database will
most likely produce a result set containing a very large
number of customers and not much can be learned from
this query except for the fact that a large number of
customers spent more than $100 at the store. Our innate
analytical capabilities are quickly overwhelmed by large
volumes of data.
This is where differences between querying a data-
base and mining a database surface. In contrast to a
query, which simply returns the data that fulfills certain
constraints, data mining constructs models of the data in
question. The models can be viewed as high level sum-
maries of the underlying data and are in most cases more
useful than the raw data, since in a business sense they
usually represent understandable and actionable items
(Berry & Linoff, 2004). Depending on the questions of
interest, data mining models can take on very different
forms. They include decision trees and decision rules
for classification tasks, association rules for market
basket analysis, as well as clustering for market seg-
mentation among many other possible models. Good
overviews of current data mining techniques and models
can be found in Berry & Linoff (2004), Han & Kamber
(2001), Hand, Mannila, & Smyth (2001), and Hastie,
Tibshirani, & Friedman (2001).
To continue our store example, in contrast to a
query, a data mining algorithm that constructs decision
rules might return the following set of rules for custom-
ers that spent more than $100 from the store database:
IF AGE > 35 AND CAR = MINIVAN THEN TOTAL SPENT
> $100
OR
IF SEX = M AND ZIP = 05566 THEN TOTAL SPENT > $100
340
Database Queries, Data Mining, and OLAP
These rules are understandable because they summa-
rize hundreds, possibly thousands, of records in the
customer database and it would be difficult to glean this
information off the query result. The rules are also action-
able. Consider that the first rule tells the storeowner that
adults over the age of 35 that own a minivan are likely to
spend more than $100. Having access to this information
allows the storeowner to adjust the inventory to cater to
this segment of the population, assuming that this repre-
sents a desirable cross-section of the customer base.
Similar with the second rule, male customers that reside in
a certain ZIP code are likely to spend more than $100.
Looking at census information for this particular ZIP
code, the storeowner could again adjust the store inven-
tory to also cater to this population segment, presumably
increasing the attractiveness of the store and thereby
increasing sales.
As we have shown, the fundamental difference be-
tween database queries and data mining is the fact that in
contrast to queries data mining does not return raw data
that satisfies certain constraints, but returns models of
the data in question. These models are attractive be-
cause in general they represent understandable and ac-
tionable items. Since no such modeling ever occurs in
database queries we do not consider running queries
against database tables as data mining, it does not matter
how large the tables are.
Database Queries vs. OLAP
In a typical relational database queries are posed against
a set of normalized database tables in order to retrieve
instances that fulfill certain constraints on their at-
tribute values (Date, 2000). The normalized tables are
usually associated with each other via primary/foreign
keys. For example, a normalized database of our store
with multiple store locations or sales units might look
something like the database given in Figure 2. Here, PK
and FK indicate primary and foreign keys, respectively.
From a user perspective it might be interesting to ask
some of the following questions:
How much did sales unit A earn in January?
How much did sales unit B earn in February?
What was their combined sales amount for the
first quarter?
Even though it is possible to extract this information
with standard SQL queries from our database, the nor-
malized nature of the database makes the formulation of
the appropriate SQL queries very difficult. Further-
more, the query process is likely to be slow due to the
fact that it must perform complex joins and multiple
TEAM LinG
Database Queries, Data Mining, and OLAP
Figure 2. Normalized database schema for a store
(Source: Craig et al., 1999, Figure 3.2)
scans of entire database tables in order to compute the
desired aggregates.
By rearranging the database tables in a slightly differ-
ent manner and using a process called pre-aggregation or
computing cubes the above questions can be answered
with much less computational power enabling a real time
analysis of aggregate attribute values OLAP (Craig et
al., 1999; Kimball, 1996; Scalzo, 2003). In order to
enable OLAP, the database tables are usually arranged
into a star schema where the innermost table is called the
fact table and the outer tables are called dimension tables.
Figure 3 shows a star schema representation of our store
organized along the main dimensions of the store busi-
ness: customers, sales units, products, and time.
The dimension tables give rise to the dimensions in
the pre-aggregated data cubes. The fact table relates the
Figure 3. Star schema for a store database (Source:
Craig et al., 1999, Figure 3.3)
dimensions to each other and specifies the measures that
are to be aggregated. Here the measures are dollar_total, ,
sales_tax, and shipping_charge. Figure 4 shows a
three-dimensional data cube pre-aggregated from the
star schema in Figure 3 (in this cube we ignored the
customer dimension, since it is difficult to illustrate four-
dimensional cubes). In the cube building process the
measures are aggregated along the smallest unit in each
dimension, giving rise to small pre-aggregated segments
in a cube.
Data cubes can be seen as a compact representation
of pre-computed query results1. Essentially, each seg-
ment in a data cube represents a pre-computed query
result to a particular query within a given star schema.
The efficiency of cube querying allows the user to
interactively move from one segment in the cube to
another enabling the inspection of query results in real
time. Cube querying also allows the user to group and
ungroup segments, as well as project segments onto
given dimensions. This corresponds to such OLAP
operations as roll-ups, drill-downs, and slice-and-dice,
respectively (Gray, Bosworth, Layman, & Pirahesh,
1997). These specialized operations in turn provide
answers to the kind of questions mentioned above.
As we have seen, OLAP is enabled by organizing a
relational database in a way that allows for the pre-
aggregation of certain query results. The resulting data
cubes hold the pre-aggregated results giving the user
the ability to analyze these aggregated results in real
time using specialized OLAP operations. In a larger
context we can view OLAP as a methodology for the
organization of databases along the dimensions of a
business making the database more comprehensible to
the end user.
Data Mining vs. OLAP
Is OLAP data mining? As we have seen, OLAP is en-
abled by a change to the data definition of a relational
Figure 4. A three-dimensional data cube
341
TEAM LinG

database in such a way that it allows for the pre-compu-
tation of certain query results. OLAP itself is a way to look
at these pre-aggregated query results in real time. How-
ever, OLAP itself is still simply a way to evaluate queries,
which is different from building models of the data as in
data mining. Therefore, from a technical point of view we
cannot consider OLAP to be data mining. Where data
mining tools model data and return actionable rules,
OLAP allows users to compare and contrast measures
along business dimensions in real time.
It is interesting to note that recently a tight integra-
tion of data mining and OLAP has occurred. For ex-
ample, Microsoft SQL Server 2000 not only allows
OLAP tools to access the data cubes but also enables its
data mining tools to mine data cubes (Seidman, 2001).
FUTURE TRENDS
Perhaps the most important trend in the area of data
mining and relational databases is the liberation of data
mining tools from the single table requirement. This
new breed of data mining algorithms is able to take
advantage of the full relational structure of a relational
database obviating the need of constructing a single
table that contains all the information to be used in the
data mining task (Deroski & Lavra , 2001). This allows for
data mining tasks to be represented naturally in terms of
the actual database structures, for example Yin, Han,
Yang, & Yu (2004), and also allows for a natural and tight
integration of data mining tools with relational databases.
CONCLUSION
Modern, commercially available relational database
systems now routinely include a cadre of data retrieval
and analysis tools. Here, we briefly described and con-
trasted the most often-bundled tools: SQL database
query engines, data mining components, and OLAP tools.
Contrary to many assertions in the literature and busi-
ness press, performing queries on large tables or ma-
nipulating query data via OLAP tools is not considered
data mining due to the fact that no data modeling occurs
in these tools. On the other hand, these three tools
complement each other and allow developers to pick the
tool that is right for their application: queries allow ad
hoc access to virtually any instance in a database; data
mining tools can generate high-level, actionable sum-
maries of data residing in database tables; and OLAP
allows for real-time access to pre-aggregated measures
along important business dimensions. In this light it does
342
Database Queries, Data Mining, and OLAP
not seem surprising that all three tools are now routinely
bundled.
REFERENCES
Berry, M.J.A., & Linoff, G.S. (2004). Data mining tech-
niques: For marketing, sales, and customer relationship
management (2nd ed.). New York: John Wiley & Sons.
Codd, E.F. (1970). A relational model of data for large
shared data banks. Communications of the ACM, 13(6),
377-387.
Craig, R.S., Vivona, J.A., & Bercovitch, D. (1999). Microsoft
data warehousing. New York: John Wiley & Sons.
Date, C.J. (2000). An introduction to database systems
(7th ed.). Reading, MA: Addison-Wesley.
Deroski, S., & Lavra , N. (2001). Relational data mining.
Berlin, New York: Springer.
Gray, J., Bosworth, A., Layman, A., & Pirahesh, H. (1997).
Data cube: A relational aggregation operator generalizing
group-by, cross-tab, and sub-totals. Data Mining and
Knowledge Discovery, 1(1), 29-53.
Han, J., & Kamber, M. (2001). Data mining: Concepts and
techniques. San Francisco: Morgan Kaufmann Publishers.
Hand, D.J., Mannila, H., & Smyth, P. (2001). Principles
of data mining. Cambridge, MA: MIT Press.
Hastie, T., Tibshirani, R., & Friedman, J.H. (2001). The
elements of statistical learning: Data mining, infer-
ence, and prediction. New York: Springer.
Kimball, R. (1996). The data warehouse toolkit: Prac-
tical techniques for building dimensional data ware-
houses. New York: John Wiley & Sons.
Oracle. (2001). Oracle 9i Data Mining. Oracle White Paper.
Pendse, N. (2001). Multidimensional data structures.
Retrieved from http://www.olapreport.com/
MDStructures.htm
Sattler, K., & Dunemann, O. (2001, November 5-10). SQL
database primitives for decision tree classifiers. Paper pre-
sented at the 10th International Conference on Information
and Knowledge Management, Atlanta, Georgia.
Scalzo, B. (2003). Oracle DBA guide to data warehousing
and star schemas. Upper Saddle River, NJ: Prentice Hall
PTR.
TEAM LinG
Database Queries, Data Mining, and OLAP
Seidman, C. (2001). Data mining with Microsoft SQL
Server 2000 technical reference. Microsoft Press.
Yin, X., Han, J., Yang, J., & Yu, P.S. (2004). CrossMine: Efficient
classification across multiple database relations. Paper pre-
sented at the 20th International Conference on Data Engi-
neering (ICDE 2004), Boston, MA, USA.
KEY TERMS
Business Intelligence: Business intelligence (BI) is a
broad category of technologies that allows for gathering,
storing, accessing and analyzing data to help business
users make better decisions. (Source: http://
www.oranz.co.uk/glossary_text.htm)
Data Cubes: Also known as OLAP cubes. Data stored
in a format that allows users to perform fast multi-
dimensional analysis across different points of view.
The data is often sourced from a data warehouse and
relates to a particular business function. (Source: http:/
/www.oranz.co.uk/glossary_text.htm)
Normalized Database: A database design that ar-
ranges data in such a way that it is held at its lowest level
avoiding redundant attributes, keys, and relationships.
(Source: http://www.oranz.co.uk/glossary_text.htm)
OLAP (Online Analytical Processing): A cat-
egory of applications and technologies for collecting, ,
managing, processing and presenting multidimensional
data for analysis and management purposes. (Source:
http://www.olapreport.com/glossary.htm)
Query: This term generally refers to databases. A
query is used to retrieve database records that match
certain criteria. (Source: http://usa.visa.com/business/
merchants/online_trans_glossary.html)
SQL (Structured Query Language): SQL is a stan-
dardized programming language for defining, retriev-
ing, and inserting data objects in relational databases.
Star Schema: A database design that is based on a
central detail fact table linked to surrounding dimension
tables. Star schemas allow access to data using business
terms and perspectives. (Source: http://www.ds.uillinois.
edu/glossary.asp)
ENDNOTE
1
Another interpretation of data cubes is as an effec-
tive representation of multidimensional data along
the main business dimensions (Pendse, 2001).
343
TEAM LinG
344
Database Sampling for Data Mining
Patricia E.N. Lutu
University of Pretoria, South Africa
INTRODUCTION
In data mining, sampling may be used as a technique for
reducing the amount of data presented to a data mining
algorithm. Other strategies for data reduction include
dimension reduction, data compression, and
discretisation. For sampling, the aim is to draw, from a
database, a random sample, which has the same character-
istics as the original database. This chapter looks at the
sampling methods that are traditionally available from the
area of statistics, how these methods have been adapted
to database sampling in general and database sampling
for data mining in particular.
BACKGROUND
Given the rate at which database/data warehouse sizes are
growing, attempts at creating faster/more efficient algo-
rithms that can process massive data sets may eventually
become futile exercises. Modern database and data ware-
house sizes are in the region of 10s or 100s of terabytes,
and sizes continue to grow. A query issued on such a
database/data warehouse could easily return several mil-
lions of records.
While the costs of data storage continue to decrease,
the analysis of data continues to be hard. This is the case
for even traditionally simple problems requiring aggrega-
tion, for example, the computation of a mean value for
some database attribute. In the case of data mining, the
computation of very sophisticated functions, on very
large numbers of database records, can take several hours,
or even days. For inductive algorithms, the problem of
lengthy computations is compounded by the fact that
many iterations are needed in order to measure the train-
ing accuracy as well as the generalization accuracy.
There is plenty of evidence to suggest that, for induc-
tive data mining, the learning curve flattens after only a
small percentage of the available data from a large data set
has been processed (Catlett, 1991; Kohavi, 1996; Provost
et al., 1999). The problem of overfitting (Dietterich, 1995)
also dictates that the mining of massive data sets should
be avoided. The sampling of databases has been studied
by researchers for some time. For data mining, sampling
should be used as a data reduction technique, allowing a
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
data set to be represented by a much smaller random
sample that can be processed much more efficiently.
MAIN THRUST
There are a number of key issues to be considered before
obtaining a suitable random sample for a data mining task.
It is essential to understand the strengths and weaknesses
of each sampling method. It is also essential to under-
stand which sampling methods are more suitable to the
type of data to be processed and the data mining algorithm
to be employed. For research purposes, we need to look
at a variety of sampling methods used by statisticians, and
attempt to adapt them to sampling for data mining.
Some Basics of Statistical Sampling
Theory
In statistics, the theory of sampling, also known as statis-
tical estimation or the representative method, deals with
the study of suitable methods of selecting a representa-
tive sample of a population, in order to study or estimate
values of specific characteristics of the population
(Neyman, 1934). Since the characteristics being studied
can only be estimated from the sample, confidence inter-
vals are calculated to give the range of values within
which the actual value will fall, with a given probability.
There are a number of sampling methods discussed in
the literature, for example the book by Rao (Rao, 2000).
Some methods appear to be more suited to database
sampling than others. Simple random sampling (SRS),
stratified random sampling, and cluster sampling are three
such methods. Simple random sampling involves select-
ing at random elements of the population, P, to be studied.
The method of selection may be either with replacement
(SRSWR) or without replacement (SRSWOR). For very
large populations, however, SRSWR and SRSWOR are
equivalent. For simple random sampling, the probabilities
of inclusion of the elements may or may not be uniform.
If the probabilities are not uniform then a weighted ran-
dom sample is obtained.
The second method of sampling is stratified random
sampling. Here, before the samples are drawn, the popu-
lation P is divided into several strata, p1, p2,.. p k, and the
TEAM LinG
Database Sampling for Data Mining
sample S is composed of k partial samples s1, s 2,.., sk, each
drawn randomly, with replacement or not, from one of the
strata. Rao (2000) discusses several methods of allocating
the number of sampled elements for each stratum. Bryant
et al. (1960) argue that, if the sample is allocated to the
strata in proportion to the number of elements in the
strata, it is virtually certain that the stratified sample
estimate will have a smaller variance than a simple random
sample of the same size. The stratification of a sample may
be done according to one criterion. Most commonly
though, there are several alternative criteria that may be
used for stratification. When this is the case, the different
criteria may all be employed to achieve multi-way stratifi-
cation. Neyman (1934) argues that there are situations
when it is very difficult to use an individual unit as the unit
of sampling. For such situations, the sampling unit should
be a group of elements, and each stratum should be
composed of several groups. In comparison with strati-
fied random sampling, where samples are selected from
each stratum, in cluster sampling a sample of clusters is
selected and observations/measurements are made on the
clusters. Cluster sampling and stratification may be com-
bined (Rao, 2000).
Database Sampling Methods
Database sampling has been practiced for many years for
purposes of estimating aggregate query results, database
auditing, query optimization, and, obtaining samples for
further statistical processing (Olken, 1993). Static sam-
pling (Olken, 1993) and adaptive (dynamic) sampling
(Haas & Swami, 1992) are two alternatives for obtaining
samples for data mining tasks. In recent years, many
studies have been conducted in applying sampling to
inductive and non-inductive data mining (John & Lan-
gley, 1996; Provost et al., 1999; Toivonen, 1996).
Simple Random Sampling
Simple random sampling is by far, the simplest method of
sampling a database. Simple random sampling may be
implemented using sequential random sampling or reser-
voir sampling. For sequential random sampling, the prob-
lem is to draw a random sample of size n without replace-
ment, from a file containing N records. The simplest
sequential random sampling method is due to Fan et al.
(1962) and Jones (1962). An independent uniform random
variate [from the uniform interval (0,1)] is generated for
each record in the file to determine whether the record
should be included in the sample. If m records have
already been chosen from among the first t records in
the file, the (t+1)st record is chosen with probability
(RQsize/RMsize), where RQsize = (n-m) is the number of
records that still need to be chosen for the sample, and
RMsize = (N-t) is the number of records in the file still to ,
be processed. This sampling method is commonly referred
to as method S (Vitter, 1987).
The reservoir sampling method (Fan et al., 1962;
Jones, 1962; Vitter, 1985, 1987) is a sequential sampling
method over a finite population of database records, with
an unknown population size. Olken (1993) discuss its use
in sampling of database query outputs on the fly. This
technique produces a sample of size S, by initially placing
the first S records of the database/file/query in the reser-
voir. For each subsequent kth database record, that record
is accepted with probability S/k. If accepted, it replaces a
randomly selected record in the reservoir.
Acceptance/Rejection sampling (A/R sampling) can
be used to obtain weighted samples (Olken, 1993). For a
weighted random sample, the probabilities of inclusion of
the elements of the population are not uniform. For data-
base sampling, the inclusion probability of a data record
is proportional to some weight calculated from the records
attributes. Suppose that one database record rj is to be
drawn from a file of n records with the probability of
inclusion being proportional to the weight wj. This may be
done by generating a uniformly distributed random inte-
ger 1 j n and then accepting the sampled record rj with
probability j = w j / w max, where wmax is the maximum
possible value for wj. The acceptance test is performed by
generating another uniform random variate uj, 0 uj 1,
and accepting rj iff uj < j. If r j is rejected, the process is
repeated until some rj is accepted.
Stratified Sampling
Density biased sampling (Palmer & Faloutsos, 2000) is a
method that combines clustering and stratified sampling.
In density biased sampling, the aim is to sample so that
within each cluster points are selected uniformly, the
sample is density preserving, and the sample is biased by
cluster size. Density preserving in this context means that
the expected sum of weights of the sampled points for
each cluster is proportional to the clusters size. Since it
would be infeasible to determine the clusters apriori,
groups are used instead to represent all the regions in n-
dimensional space. Sampling is then done to be density
preserving for each group. The groups are formed by
placing a d-dimensional grid over the data. In the d-
dimensional grid, the d dimensions of each cell are labeled
either with a bin value for numeric attributes, or by a
discrete value for categorical attributes. The d-dimen-
sional grid defines the strata for multi-way stratified
sampling. A one-pass algorithm is used to perform the
weighted sampling, based on the reservoir algorithm.
345
TEAM LinG

Adaptive Sampling
Lipton et al. (1990) use adaptive sampling, also known as
sequential sampling, for database sampling. In sequential
sampling, a decision is made after each sampled element
whether to continue sampling. Olken (1993) has observed
that sequential sampling algorithms outperform conven-
tional single-stage algorithms in terms of the number of
sample elements required, since they can adjust the sample
size to the population parameters. Haas & Swami (1992)
have proved that sequential sampling uses the minimum
sample size for the required accuracy.
John & Langley (1996) have proposed a method they
call dynamic sampling, which combines database sam-
pling with the estimation of classifier accuracy. The method
is most efficiently applied to classification algorithms that
are incremental, for example nave Bayes and artificial
neural-network algorithms such as backpropagation. They
define the concept of probably close enough (PCE),
which they use for determining when a sample size pro-
vides an accuracy that is probably good enough. Good
enough in this context means that there is a small prob-
ability that the mining algorithm could do better by using
the entire database. The smallest sample size n is chosen
from a database of size N, so that: Pr (acc(N) acc(n) >
) <= , where: acc(n) is the accuracy after processing a
sample of size n, and is a parameter that describes what
close enough means. The method works by gradually
increasing the sample size n until the PCE condition is
satisfied.
Provost, Jensen, & Oates (1999) use progressive sam-
pling, another form of adaptive sampling, and analyse its
efficiency relative to induction with all available examples.
The purpose of progressive sampling is to establish nmin,
the size of the smallest sufficient sample. They address the
issue of convergence, where convergence means that a
learning algorithm has reached its plateau of accuracy. In
order to detect convergence, they define the notion of a
sampling schedule S as S = {no, n1, .., nk} where ni is an
integer that specifies the size of the sample, and S is a
sequence of sample sizes to be provided to an inductive
algorithm. They show that schedules in which n i increases
geometrically as: {no, a.n o, a2.no,., ak.no}, are asymptoti-
cally optimal. As one can see, progressive sampling is
similar to the adaptive sampling method of John & Langley
(1996), except that a non-linear increment for the sample
size is used.
THE SAMPLING PROCESS
Several decisions need to be made when sampling a
database. One needs to decide on a sampling method, a
346
Database Sampling for Data Mining
suitable sample size, and the level accuracy that can be
tolerated. These issues are discussed below.
Deciding on a Sampling Method
The data to be sampled may be balanced, imbalanced,
clustered or unclustered. These characteristics will af-
fect the quality of the sample obtained. While simple
random sampling is very easy to implement, it may pro-
duce non-representative samples for data that is
imbalanced or clustered. On the other hand, stratified
sampling, with a good choice of strata cells, can be used
to produce representative samples, regardless of the
characteristics of the data. Implementation of one-way
stratification should be straight forward, however, for
multi-way stratification, there are many considerations
to be made. For example, in density-biased sampling, a d-
dimensional grid is used. Suppose each dimension has n
possible values (or bins). The multi-way stratification
will result in n d strata cells. For large d, this is a very large
number of cells. When it is not easy to estimate the
sample size in advance, adaptive (or dynamic) sampling
may be employed, if the data mining algorithm is incre-
mental.
Determining the Representative
Sample Size
For static sampling, the question must be asked: What
is the size of a representative sample? A sample is
considered statistically valid if it is sufficiently similar to
the database from where it is drawn (John & Langley,
1996). Univariate sampling may be used to test that each
field in the sample comes from the same distribution as
the parent database. For categorical fields, the chi-squared
test can be used to test the hypothesis that the sample
and the database come from the same distribution. For
continuous-valued fields, a large-sample test can be
used to test the hypothesis that the sample and the
database have the same mean. It must however be pointed
out that obtaining fixed size representative samples from
a database is not a trivial task, and consultation with a
statistician is recommended.
For inductive algorithms, the results from the theory
of probably approximately correct (PAC) learning have
been suggested in the literature (Valiant, 1984; Haussler,
1990). These have however been largely criticized for
overestimating the sample size (e.g., Haussler, 1990). For
incremental inductive algorithms, dynamic sampling (John
& Langley, 1996; Provost et al., 1999) may be employed
to determine when a sufficient sample has been pro-
cessed. For association rule mining, the methods de-
scribed by Toivonen (1996) may be used to determine the
sample size.
TEAM LinG
Database Sampling for Data Mining
Determining the Accuracy and
Confidence of Results Obtained from
Samples
For the task of inductive data mining, suppose we have
estimated the classification error for a classifier con-
structed from sample S, to be errors (h), as the proportion
of the test examples that are misclassified. Statistical
theory, based on the central limit theorem, enables us to
conclude that, with approximately N% probability, the
true error lies in the interval:
Errors (h) Z N (Errors (h)(1 Errors (h)) / n )
As an example, for a 95% probability, the value of ZN
is 1.96. Note that, ErrorS (h) is the mean error and (ErrorS
(h) (1- ErrorS (h))/n) is the variance of the error. Mitchell
(1997) gives a detailed discussion of the estimation of
classifier accuracy and confidence intervals. For non
inductive data mining algorithms, there are known ways
of estimating the error and confidence in the results
obtained from samples. For association rule mining, for
example, Toivonen (1996) states that the lower bound for
the size of the sample, given an error bound and a
maximum probability , is given by:
1 2
| s | ln
2 2
The value is the error in estimating the frequencies
of frequent item sets for some give set of attributes.
FUTURE TRENDS
There two main thrusts in research on establishing the
sufficient sample size for a data mining task. The theoreti-
cal approach, most notably the PAC framework and the
Vapnik-Chernonenkis (VC) dimension have produced re-
sults that are largely criticized by data mining researchers
and practitioners. However, research will continue in this
area, and may eventually yield practical guidelines. On the
empirical front, researchers will continue to conduct simu-
lations that will lead to generalizations on how to estab-
lish sufficient sample sizes.
CONCLUSIONS
Modern databases and data warehouses are massive, and
will continue to grow in size. It is essential for researchers
to investigate data reduction techniques that can greatly
reduce the amount of data presented to a data mining
algorithm. More than fifty years of research has produced ,
a variety of sampling algorithms for database tables and
query result sets. The selection of a sampling method
should depend on the nature of the data as well as the
algorithm to be employed. Estimation of sample sizes for
static sampling is a tricky issue. More research in this area
is needed in order to provide practical guidelines. Strati-
fied sampling would appear to be a versatile approach to
sampling any type of data. However, more research is
needed to address especially the issue of how to define
the strata for sampling.
REFERENCES
Bryant, E.C., Hartley, H.O., & Jessen, R.J. (1960). Design
and estimation in two-way stratification. Journal of the
American Statistical Association, 55, 105-124.
Catlett, J. (1991). Megainduction: A test flight. In Pro-
ceedings of the Eighth Workshop on Machine Learning
(pp. 596-599). San Mateo, CA: Morgan Kaufman.
Dietterich, T. (1995). Overfitting and undercomputing in
machine learning. ACM Computing Surveys, 27(3), 326-
327.
Fan, C., Muller, M., & Rezucha, I. (1962). Development of
sampling plans by using sequential (item by item) selec-
tion techniques and digital computers. Journal of the
American Statistical Association, 57, 387-402.
Haas, P.J., & Swami, A.N. (1992). Sequential sampling
procedures for query size estimation. IBM Technical
Report RJ 8558. IBM Alamaden.
Haussler, D. (1990). Probably approximately correct learn-
ing. National Conference on Artificial Intelligence.
John, G.H., & Langley, P. (1996). Static versus dynamic
sampling for data mining. In Proceedings of the Second
International Conference on Knowledge Discovery in
Databases and Data Mining. AAAI/MIT Press.
Jones, T. (1962). A note on sampling from tape files.
Communications of the ACM, 5, 343-343.
Kohavi, R. (1996). Scaling up the accuracy on nave-Bayes
classifiers: A decision tree hybrid. In Proceedings of the
Second International Conference on Knowledge Dis-
covery and Data Mining.
Lipton, R., Naughton, J., & Schneider, D. (1990). Practical
selectivity estimation through adaptive sampling. In ACM
SIGMOD International Conference on the Management
of Data (pp. 1-11).
347
TEAM LinG

Mitchell, T.M. (1997). Machine learning. McGraw-Hill.
Neyman, J. (1934). On the two different aspects of the
representative method: The method of stratified sampling
and the method of purposive selection. Journal of the
Royal Statistical Society, 97, 558-625.
Olken, F. (1993). Random sampling from databases. PhD
thesis. University of California at Berkeley.
Palmer C.R., & Faloutsos, C. (2000). Density biased sam-
pling: An improved method for data mining and cluster-
ing. In Proceedings of the ACM SIGMOD Conference (pp.
82-92).
Provost, F., Jensen, D., & Oates, T. (1999). Efficient
progressive sampling. In Proceedings of the Fifth ACM
DIGKDD International Conference on Knowledge Dis-
covery and Data Mining (pp. 23-32), San Diego, CA.
Rao, P.S.R.S. (2000). Sampling methodologies with ap-
plications. FL: Chapman & Hall/CRC.
Toivonen, H. (1996). Sampling large databases for asso-
ciation rules. In Proceedings of the Twenty-Second
Conference on Very Large Databases (VLDDB96),
Mumbai India.
Valiant, L.G. (1984). A theory of the learnable. Communi-
cations of the ACM, 27 (11), 1134-1142.
Vitter, J.S. (1985). Random sampling with a reservoir. ACM
Transactions on Mathematical Software, 11, 37-57.
Vitter, J.S. (1987). An efficient method for sequential
random sampling. ACM Transactions on Mathematical
Software, 13 (1), 58-67.
KEY TERMS
Cluster Sampling: In cluster sampling, a sample of
clusters is selected and observations/measurements are
made on the clusters.
Density-Biased Sampling: A database sampling
method that combines clustering and stratified sampling.
348
Database Sampling for Data Mining
Dynamic Sampling (Adaptive Sampling): A method
of sampling where sampling and processing of data pro-
ceed in tandem. After processing each incremental part of
the sample, a decision is made whether to continue sam-
pling or not.
Reservoir Sampling: A database sampling method that
implements uniform random sampling on a database table of
unknown size, or a query result set of unknown size.
Sequential Random Sampling: A database sam-
pling method that implements uniform random sam-
pling on a database table whose size is known.
Simple Random Sampling: Simple random sampling
involves selecting at random elements of the population
to be studied. The sample S is obtained by selecting at
random single elements of the population P.
Simple Random Sampling with Replacement
(SRSWR): A method of simple random sampling where an
element stands a chance of being selected more than once.
Simple Random Sampling without Replacement
(SRSWOR): A method of simple random sampling where
each element stands a chance of being selected only
once.
Static Sampling: A method of sampling where the
whole sample is obtained before processing begins. The
user must specify the sample size.
Stratified Sampling: For this method, before the
samples are drawn, the population P is divided into
several strata, p1, p2,.. pk, and the sample S is composed
of k partial samples s1, s 2,.., sk, each drawn randomly, with
replacement or not, from one of the strata.
Uniform Random Sampling: A method of simple ran-
dom sampling where the probabilities of inclusion for
each element are equal.
Weighted Random Sampling: A method of simple
random sampling where the probabilities of inclusion for
each element are not equal.
TEAM LinG

DEA Evaluation of Performance of E-Business
Initiatives
Yao Chen
University of Massachusetts Lowell, USA
Luvai Motiwalla
University of Massachusetts Lowell, USA
M. Riaz Khan
University of Massachusetts Lowell, USA
INTRODUCTION
The Internet has experienced a phenomenal growth in
attracting people and commerce activities over the last
decadefrom a few thousand people in 1993 to 150+
million in 1999, and about one billion by 2004 (Bingi et al.,
2000). This growth has attracted a variety of organizations
initially to provide marketing information about their
products and services and, customer support, and later to
conduct business transactions with customers or busi-
ness partners on the Web. These electronic business (EB)
initiatives could be the implementation of intranet and/or
extranet applications like B2C, B2B, Web-CRM, Web-
marketing, and others, to take advantage of the Web-
based economic model, which offers opportunities for
internal efficiencies and external growth.
It has been recognized that the Internet economic
model is more efficient at the transaction cost level and
elimination of the middleman in the distribution channel,
and also can have a big impact on the market efficiency.
Web-enabling business processes are particularly attrac-
tive in the new economy, where product life cycles are
short and efficient, while the market for products and
services is global. Similarly, management of these compa-
nies expects a much better financial performance than
their counterparts in the industry, which had not adopted
these EB initiatives (Hoffman et. al., 1995; Wigand &
Benjamin, 1995).
EB allows organizations to expand their business
reach. One of the key benefits of the Web is access to and
from global markets. The Web eliminates several geo-
graphical barriers for a corporation that wants to conduct
global commerce. While traditional commerce relied on
value-added networks (VANs) or private networks, which
were expensive and provided limited connectivity (Pyle,
1996), the Web makes electronic commerce cheaper with
extensive global connectivity. Corporations have been
able to produce goods anywhere and deliver electroni-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
349
,
cally or physically via couriers. This enables organiza-
tions the flexibility to expand into different product lines
and markets quickly, with low investments. Secondly,
24x7 availability, better communication with customers,
and sharing of the organizational knowledge base allows
organizations to provide better customer service. This
can translate to better customer retention rates as well as
repeat orders. Finally, the rich interactive media and
database technology of the Web allows for unconstrained
awareness, visibility, and opportunity for an organization
to promote its products and services. This enhances
organizations abilities to attract new customers, thereby
increasing their overall markets and profitability. Despite
the recent dot-com failures, EB has made tremendous
inroads in traditional corporations. Forrester Research in
its survey found 90% of the firms plan to conduct some e-
commerce, business-to-consumer (B2C), or business-to-
business (B2B), and predicts EB transactions to rise to
about $6.9 trillion by 2004. As a result, the management
has started to believe in the Internet because of its ability
to attract and retain more customers, reduce sales and
distribution overheads, and global access to markets with
an expectation of an increase in sales revenues, higher
profits, and better returns for the stockholders (Choi &
Winston, 2000; Motiwalla & Khan, 2002; Steinfield &
Whitten, 1999; White, 1999).
BACKGROUND
It is important that we use a comprehensive performance
evaluation tool to examine whether these EB initiatives
have a positive impact on the financial performance.
Managers are often interested in evaluating how effi-
ciently EB initiatives are with respect to multiple inputs
and outputs. Single-measure gap analysis is often used as
a fundamental method in performance evaluation and best
practice identification. It is extremely difficult to show
TEAM LinG
 DEA Evaluation of Performance of E-Business Initiatives
benchmarks where multiple measurements exist. It is rare
that one single measure can suffice for the purpose of
performance assessment. In our empirical study, there are
multiple measures that characterize the performance of
retail companies. This requires that the research tool used
here have the flexibility to deal with changing production
technology in the context of multiple performance mea-
sures. Data envelopment analysis (DEA) is originally
developed to measure the relative efficiency of peer
decision-making units (DMUs) in a multiple input-output
setting. DEA has been proven to be an excellent method-
ology for performance evaluation and benchmarking (Zhu,
2003).
Based on Cooper, Seiford, and Zhu (2004), the specific
reasons for using DEA are given as follows. First, DEA is
a data-oriented approach for evaluating the performance
of a set of peer DMUs, which convert multiple inputs into
multiple outputs. In our case, the DMUs can be, for
example, corporations that have launched EB activities.
For each corporation, each year can be regarded as a
DMU. Second, DEA is a methodology directed to fron-
tiers rather than central tendencies. Instead of trying to fit
a regression plane through the center of the data, as is
done in statistical regression, for example, one floats a
piecewise linear surface to rest on top of the observations.
Because of this approach, DEA proves particularly adept
at uncovering relationships that remain hidden in other
methodologies. Third, DEA does not require explicitly
formulated assumptions of functional form as in linear and
nonlinear regression models. This flexibility allows us to
identify the multi-dimensional efficient frontier without
the need for explicitly expressing the technology change
and organizational knowledge.
In order to discriminate the performance among the
efficient DMUs, a super-efficiency DEA model in which a
DMU under evaluation is excluded from the reference set
is developed. However, the super-efficiency model has
been restricted to the case of constant returns to scale
(CRS), because the non-CRS super-efficiency DEA model
can be infeasible (Seiford & Zhu, 1998; 1999; Zhu, 1996).
It is difficult to precisely define infeasibility. As a
result, one cannot rank the performance of a set of DMUs.
In fact, an input-oriented super-efficiency DEA model
measures the input super-efficiency when outputs are
fixed at their current levels. Likewise, an output-oriented
super-efficiency DEA model measures the output super-
efficiency when inputs are fixed at their current levels.
From the different uses of the super-efficiency concept,
we see that super-efficiency can be interpreted as the
degree of efficiency stability or input saving/output sur-
plus achieved by an efficient DMU. If super-efficiency is
used as an efficiency stability measure, then infeasibility
means that an efficient DMUs efficiency classification is
350
stable to any input changes, if an input-oriented super-
efficiency DEA model is used (or any output changes, if
an output-oriented super-efficiency DEA model is used).
Therefore, we can use + to represent the super-efficiency
score (i.e., infeasibility means the highest super-efficiency).
Chen (2004) shows that (i) if an efficient DMU does not
possess any input super-efficiency (input saving), it must
possess output super-efficiency (output surplus), and (ii)
if an efficient DMU does not possess any output super-
efficiency, it must possess input super-efficiency. We
thus can use both input-oriented and output-oriented
super-efficiency DEA models to fully characterize the
super-efficiency.
Based on the above derivations, Chen, et al. (2004) are
able to rank the performance of a set of publicly held
corporations in retail industry over the period 1997-2000.
Specifically, the objective of this study is to determine
whether the financial data support the beneficial claims
made in the popular literature that EB has boosted the
bottom-line.
MAIN TRUST
To present our DEA methodology, we assume that there
are n DMUs to be evaluated. Each DMU consumes vary-
ing amounts of m different inputs to produce s different
outputs. Specifically, DMUj consumes amount xij of input
i and produces amount yrj of output r. We assume that xij
> 0 and y rj > 0 and further assume that each DMU has at
least one positive input and one positive output value.
The input-output oriented super efficiency models
whose frontier exhibits VRS can be expressed as Seiford
and Zhu (1999) in Box 1, where xio and yro are respec-
tively the ith input and rth output for a DMU o under
evaluation.
Let o represent the score for characterizing the super-
efficiency in terms of input saving, we have
VRS super* if the input - oriented super - efficiency model is feasible
o = 1 o if the input - oriented super - efficiency model is infeasible
Note that o > 1. If o >1, a specific efficient DMU o has
input super-efficiency. If o = 1, DMU o does not have
input super-efficiency. Similarly, let o represent the
score for characterizing the output super-efficiency, we
have
oVRS super* if the output - oriented super - efficiency model is feasible
o =
1 if the output - oriented super - efficiency model is infeasible
TEAM LinG
DEA Evaluation of Performance of E-Business Initiatives

Box 1.

min oVRS-super
,
max oVRS super
n n
s.t. x
j =1
j ij oVRS super xio i = 1,2,..., m; s.t. x j ij xio i = 1,2,..., m;
j =1
j o j o
n n
y
j=1
j rj y ro r = 1,2,..., s; j y rj oVRS super y ro r = 1,2,..., s;
j=1
j o j o
n n

j=1
j =1 j =1
j=1
j o j o

oVRS-super 0 oVRS super 0

j 0 jo j 0 ( j o)

Note that o < 1. If o < 1, a specific efficient DMU o
has output super-efficiency. If o = 1, DMU o does not
have output super-efficiency.
The DEA inputs and outputs can be developed from
the financial ratios. For example, the inputs can include (i)
number of employees, (ii) inventory cost, (iii) total current
assets, and (iv) cost of sales; and the outputs can include
(i) revenue and (ii) net income. In our study, 75% of the EB
companies are efficient, and 57% of the non-EB companies
are efficient. Thus, in general, the EB companies have
performed better. In terms of the super-efficiency DEA
model, the EB companies demonstrate a better perfor-
mance than the companies that have not yet adopted the
EB initiatives (Chen et al., 2004).
FUTURE TRENDS
It has been recognized that the link between IT investment
and firm performance is indirect. Future research should
focus on multi-stage performance of IT impacts on firm
performance. For example, Chen and Zhu (2004) developed
a preliminary DEA-based model to (i) characterize the
indirect impact of IT on firm performance, (ii) identify the
efficient frontier of two value-added stages related to IT
investment and profit generation, and (iii) highlight firms
that can be further analyzed for best practice benchmarking.
CONCLUSION
The analysis of the retail industry data indicates that there
is some evidence that the EB initiatives have had some
degree of favorable impact on the financial performance of
companies that have moved in this direction. Results
indicate that the EB companies performed better in some
measures than the non-EB companies included in the
sample. Further, by contrasting the performance of EB
companies against the non-EB companies, the findings
confirm that the EB companies have benefited from the
innovation and the strategic integration of the EB technologies.
REFERENCES
Bingi, P., Mir, A., & Khamalah, J. (2000). The challenges
facing global e-commerce. Information Systems Man-
agement, 6(2), 26-34.
Charnes, A, Cooper, W.W., & Rhodes, E. (1978). Measur-
ing the efficiency of decision making units. European
Journal of Operational Research, 2, 429-444.
Chen, Y. (2004). Ranking efficient units in DEA. OMEGA,
32, 213-219.
Chen, Y., Motiwalla, L., & Khan, M.R. (2004). Using
super-efficiency DEA to evaluate financial performance
of e-business initiative in the retail industry. Interna-
tional Journal of Information Technology and Decision
Making, 3(2), 337-351.
Chen, Y., & Zhu, J. (2004). Measuring information
technologys indirect impact on firm performance. Infor-
mation Technology & Management Journal, 5(1), 9-22.
Choi, S., & Winston, A. 2000. Benefits and requirements
for interoperability in electronic marketplace. Technol-
ogy in Society, 22, 33-44.
Cooper, W.W., Seiford, L.M., & Zhu, J. (2004). Handbook
on data envelopment analysis. Boston: Kluwer Aca-
demic Publishers.

351

TEAM LinG
 DEA Evaluation of Performance of E-Business Initiatives
Hoffman, D., Novak, T., & Chatterjee, P. (1995). Commer-
cial scenarios for the Web: Opportunities and challenges.
Journal of Computer-Mediated Communications, 1(3).
Motiwalla, L., & Khan, M.R. (2002). Financial impact of e-
business initiatives in the retail industry. Journal of
Electronic Commerce in Organization, 1(1), 55-73.
Pyle, R. (1996). Commerce and the Internet. Communica-
tions of the ACM, 39(6), 23.
Seiford, L.M., & Zhu, J. (1998). Sensitivity analysis of
DEA models for simultaneous changes in all the data.
Journal of the Operational Research Society, 49, 1060-
1071.
Seiford, L.M., & Zhu, J. (1999). Infeasibility of super-
efficiency data envelopment analysis models. INFOR, 37,
174-187.
Steinfield, C., & Whitten, P. (1999). Community level
socio-economic impacts of electronic commerce. Journal
Of Computer-Mediated Communications, 5(2).
White, G. (1999, December 3). How GM, Ford think Web
can make a splash on the factory floor. Wall Street Jour-
nal, 1, 1.
Wigand, R., & Benjamin, R. (1995). Electronic commerce:
effects on electronic markets. Journal of Computer Medi-
ated Communication, 1(3).
Zhu, J. (1996). Robustness of the efficient DMUs in data
envelopment analysis. European Journal of Operational
Research, 90(3), 451-460.
Zhu, J. (2003). Quantitative models for performance evalu-
ation and benchmarking: Data envelopment analysis
with spreadsheets and DEA excel solver. Boston: Kluwer
Academic Publishers.
352
KEY TERMS
Data Envelopment Analysis (DEA): A data-oriented
mathematical programming approach that allows multiple
performance measures in a single model.
Decision Making Unit (DMU): The subject under
evaluation.
Efficient: Full efficiency is attained by any DMU if and
only if none of its inputs or outputs can be improved
without worsening some of its other inputs or outputs.
Electronic Business (EB): Business transactions in-
volving exchange of goods and services with customers
and/or business partners over the Internet.
Inputs/Outputs: Refer to the performance measures
used in DEA evaluation. Inputs usually refer to the re-
sources used, and outputs refer to the outcomes achieved
by an organization or DMU.
Returns to Scale (RTS): RTS are considered to be
increasing if a proportional increase in all the inputs
results in a more than proportional increase in the single
output. In DEA, the concept of returns to scale is extended
to multiple inputs and multiple outputs situations.
Super-Efficiency: The input savings or output sur-
pluses achieved by an efficient DMU.
TEAM LinG

Decision Tree Induction
Roberta Siciliano
University of Naples Federico II, Italy
Claudio Conversano
University of Cassino, Italy
INTRODUCTION
Decision Tree Induction (DTI) is an important step of
the segmentation methodology. It can be viewed as a
tool for the analysis of large datasets characterized by
high dimensionality and nonstandard structure. Seg-
mentation follows a nonparametric approach, since no
hypotheses are made on the variable distribution. The
resulting model has the structure of a tree graph. It is
considered a supervised method, since a response crite-
rion variable is explained by a set of predictors.
In particular, segmentation consists of partitioning
the objects (also called cases, individuals, observations,
etc.) into a number of subgroups (on the basis of suitable
partitioning of the modalities of the explanatory vari-
ables, the so-called predictors) in a recursive way, so
that a tree-structure is produced. Typically, partitioning
is in two subgroups yielding to binary trees, although
ternary trees as well as r-way trees also can be built up.
Two main targets can be achieved with tree-struc-
turesclassification and regression treeson the ba-
sis of the type of response variable, which can be cat-
egorical or numerical.
Tree-based methods are characterized by two main
tasks: exploratory and decision. The first is to describe
with the tree structure the dependence between the
response and the predictors. The decision task is prop-
erly of DTI, aiming to define a decision rule for unseen
objects for estimating unknown response class/values
as well as validating the accuracy of the final results.
For example, trees often are considered in credit-
scoring problems in order to describe and classify good
and bad clients of a bank on the basis of socioeconomic
indicators (e.g., age, working conditions, family status,
etc.) and financial conditions (e.g., income, savings,
payment methods, etc.). Conditional interactions de-
scribing the client profile can be detected looking at the
paths along the tree, when going from the top to the
terminal nodes. Each internal node of the tree is as-
signed a partition (or a split for binary tree) of the
predictor space, and each terminal node is assigned a
label class/value of the response. As a result, each tree
path, characterized by a sequence of predictor interac-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
353
,
tions, can be viewed as a production rule yielding to a
specific label class/value. The set of production rules
constitutes the predictive learning of the response class/
value of new objects, where only measurements of the
predictors are known. As an example, a new client of a
bank is classified as a good client or a bad one by
dropping it down the tree according to the set of splits
(binary questions) of a tree path, until a terminal node
labeled by a specific response-class is reached.
BACKGROUND
The appealing aspect for the segmentation user is that
the final tree provides a comprehensive description of
the phenomenon in different contexts of application,
such as marketing, credit scoring, finance, medical di-
agnosis, and so forth.
Segmentation can be considered as an exploratory
tool but also as a confirmatory nonparametric model.
Exploration can be obtained by performing a recursive
partitioning of the objects until a stopping rule defines
the final structure to interpret. Confirmation is a differ-
ent problem, requiring definition of decision rules,
usually obtained by performing a pruning procedure
soon after a partitioning one. Important questions arise
when using segmentation for predictive learning goals
(Hastie et al., 2001; Zhang, 1999). The tree structure
that fits the data and can be used for unseen objects
cannot be the simple result of any partitioning algo-
rithm. Two aspects should be jointly considered: the
tree size (i.e., the number of terminal nodes) and the
accuracy of the final decision rule evaluated by an error
measure. In fact, a weak point of decision trees is the
sensitivity of the classification/prediction rules mea-
sured by the size of the tree and its accuracy to the type
of dataset as well as to the pruning procedure. In other
words, the ability of a decision tree to detect cases and
take right decisions can be evaluated by a simple mea-
sure, but it also requires a specific induction procedure.
Likewise, in statistical inference, where the power of a
testing procedure is judged with respect to changes of the
alternative hypotheses, decision tree induction strongly
TEAM LinG

depends on both the hypotheses to verify and their
alternatives. For instance, in classification trees, the num-
ber of response classes and the prior distribution of cases
among the classes influence the quality of the final deci-
sion rule. In the credit-scoring example, an induction
procedure using a sample of 80% of good clients and 20%
of bad clients likely will provide reliable rules to identify
good clients and unreliable rules to identify bad ones.
MAIN THRUST
Exploratory trees can be fruitfully used to investigate
the data structure, but they cannot be used straightfor-
wardly for induction purposes. The main reason is that
exploratory trees are accurate and effective with re-
spect to the training data used for growing the tree, but
they might perform poorly when applied to classifying/
predicting fresh cases that have not been used in the
growing phase.
DTI Main Tasks
DTI definitely has an important purpose represented by
understandability: the tree structure for induction needs
to be simple and not large; this is a difficult task since a
predictor may reappear (even though in a restricted
form) many times down a branch. At the same time, a
further requirement is given by the identification issue:
on one hand, terminal branches of the expanded tree
reflect particular features of the training set, causing
over-fitting; on the other hand, over-pruned trees neces-
sarily do not allow identification of all the response
classes/values (under-fitting).
Tree Model Building
Simplification method performance in terms of accu-
racy depends on the partitioning criterion used in the
tree-growing procedure (Buntine & Niblett, 1992). Thus,
exploratory trees become an important preliminary step
for DTI. In tree model building, it is worth distinguish-
ing between the optimality criterion for tree pruning
(simplification method) and the criterion for selecting
the best decision rule (decision rule selection). These
criteria often use independent datasets (training set and
test set). In addition, a validation set can be required to
assess the quality of the final decision rule (Hand,
1997). In this respect, segmentation with pruning and
assessment can be viewed as stages of any computa-
tional model-building process based on a supervised
learning algorithm. Furthermore, growing the tree struc-
ture using a Fast Algorithm for Splitting Trees (FAST)
354
Decision Tree Induction
(Mola & Siciliano, 1997) becomes a fundamental step to
speed up the overall DTI procedure.
Tree Simplification: Pruning Algorithms
A further step is required for DTI relying on the hypoth-
esis of uncertainty in the data due to noise and residual
variation. Simplifying trees is necessary to remove the
most unreliable branches and improve understandabil-
ity. Thus, the goal of simplification is inferential (i.e.,
to define the structural part of the tree and reduce its
size while retaining its accuracy). Pruning methods
consist in simplifying trees in order to remove the most
unreliable branches and improve the accuracy of the rule
for classifying fresh cases.
The pioneer approach of simplification was pre-
sented in the Automatic Interaction Detection (AID) of
Morgan and Sonquist (1963). It was based on arresting
the recursive partitioning procedure according to some
stopping rule (pre-pruning).
Alternative procedures consist in pruning algorithms
working either from the bottom to the top of the tree
(post-pruning) or vice versa (pre-pruning). CART
(Breiman et al., 1984) introduced the idea to grow the
totally expanded tree for removing retrospectively some
of the branches (post-pruning). This results in a set of
optimally pruned trees for the selection of the final
decision rule.
The main issue of pruning algorithms is the defini-
tion of a complexity measure that takes account of both
the tree size and accuracy through a penalty parameter
expressing the gain/cost of pruning tree branches. The
training set is often used for pruning, whereas the test
set for selecting the final decision rule. This is the case of
both the error-complexity pruning of CART and the criti-
cal value pruning (Mingers, 1989). Nevertheless, some
methods require only the training set. This is the case of
the pessimistic error pruning and the error-based pruning
(Quinlan, 1987, 1993) as well as the minimum error pruning
(Cestnik & Bratko, 1991) and the CART cross-validation
method. Instead, other methods use only the test set,
such as the reduced error pruning (Quinlan, 1987). These
latter pruning algorithms yield to just one best pruned
tree, which represents in this way the final rule.
In DTI, accuracy refers to the predictive ability of the
decision tree to classify/predict an independent set of test
data. In classification trees, the error rate, measured by the
number of incorrect classifications of the tree on test data,
does not reflect accuracy of predictions for classes that
are not equally likely, and those with few cases are usually
badly predicted. As an alternative to the CART pruning,
Cappelli, et al. (1998) provided a pruning algorithm based
on the impurity-complexity measure to take account of the
distribution of the cases over the classes.
TEAM LinG
Decision Tree Induction
Decision Rule Selection
Given a set of optimally-pruned trees, a simple method
to choose the optimal decision rule consists in selecting
the pruned tree producing the minimum misclassification
rate on an independent test set (0-SE rule) or the most
parsimonious pruned tree whose error rate on the test set
is within one standard error of the minimum (1-SE rule)
(Breiman et al., 1984). Once the sequence of pruned
trees is obtained, a statistical testing pruning also could
be performed to achieve the most reliable decision rule
(Cappelli et al., 2002).
Tree Averaging and Compromise Rules
A more recent approach is to generate a set of candidate
decision trees that are aggregated in a suitable way to
provide a more accurate final rule. This strategy, which is
an alternative to the selection of one final tree, consists
in the definition of either a compromise rule or a con-
sensus rule using a set of trees rather than the single one.
One approach is known as tree averaging, which classi-
fies a new object by averaging over a set of trees using a
set of weights (Oliver & Hand, 1995). It requires the
definition of the set of trees (since it is impractical to
average over every possible pruned tree), the calculation
of the weights, and the independent data set to classify.
An alternative selection method for classification prob-
lems consists in summarizing the information of each
tree by table cross-classifying terminal nodes and re-
sponse classes, so that evaluating the predictability power
or the degree of homogeneity through a statistical index
yields finding out the best rule (Siciliano, 1998).
Ensemble Methods
Ensemble methods are based on a combination, expressed
by a weighted or non-weighted aggregation, of single
induction estimators able to improve the overall accu-
racy of any single induction method.
Bagging (Bootstrap Aggregating) (Breiman, 1996) is
obtained by a bootstrap replication of the sample units of
the training sample, each having the same probability to
be included in the bootstrap sample, in order to generate
single prediction/classification rules that are aggregated,
providing a final decision rule consisting in either the
average (for regression problems) or the modal class
(for classification problems) of the single estimates.
Definitions of bias and variance for a classifier as com-
ponents of the test set error have been used by Breiman
(1998). Unstable classifiers can have low bias on a large
range of data sets. Their problem is high variance.
Adaptive Boosting (AdaBoost) (Freud & Schapire,
1996; Schapire et al., 1998) adopts an iterative bootstrap ,
replication of the sample units of the training sample
such that at any iteration, misclassified/worse predicted
cases have higher probability to be included in the
current bootstrap sample, and the final decision rule is
obtained by majority voting.
Random Forest (Breiman, 1999) is instead an en-
semble of unpruned trees obtained by introducing two
bootstrap resampling schema, one on the objects and
another one on the predictors, such that an out-of-bag
sample provides the estimation of the test set error and
suitable measures of predictor importance are derived
for the final interpretation.
FUTURE TRENDS
The use of trees for both exploratory and decision
purposes will receive more and more attention over
both the statistical and the machine-learning communi-
ties, with novel contributions highlighting the idea of
decision rules as a preliminary or final tool for data
analysis (Conversano et al., 2001).
Combining Trees With Other Statistical
Tools
Some examples are given by the joint use of statistical
modeling and tree-structures for exploratory analysis
as well as for confirmatory analysis. For the former,
examples are given by the procedures called multi-
budget trees and two-stage discriminant trees, which
have been recently overviewed by Siciliano, Aria, and
Conversano (2004). For the latter, an example is given
by the stepwise model tree induction method (Malerba
et al., 2004), which associates multiple regression
models to leaves of the tree in order to define a suitable
data-driven procedure for tree induction.
Trees in Regression Smoothing
It is also worth mentioning the idea of using DTI for
regression smoothing purposes. Conversano (2002)
introduced a novel class of semiparametric models
named Generalized Additive Multi-Mixture Models
(GAM-MM) for performing statistical learning for
both classification and regression tasks. These models
are similar to generalized additive models and work
using an iterative estimation algorithm evaluating the
predictive power of suitable mixtures of smoothers/clas-
sifiers associated with each predictor entering the model.
355
TEAM LinG

Trees are part of the set of alternative smoothers or
classifiers defining the estimations functions associated
with each predictor on the basis of bagged scoring mea-
sures taking into account the trade-off between estima-
tion accuracy and model complexity.
Trees for Data Imputation and Data
Validation
A different approach is based on the use of DTI to solve
data quality problems. It is well known that the extreme
flexibility of DTI allows handling missing data in an easy
way when performing data modeling. But DTI does not
permit performing missing data imputation in a straight-
forward manner when dealing with multivariate data.
Conversano and Siciliano (2004) defined an incremen-
tal approach for missing data imputation based on deci-
sion trees. The basic idea is inspired by the principle of
statistical learning for information retrieval; namely,
observed data can be used to impute missing observa-
tions in an incremental manner, starting from the sub-
groups of observations presenting the less proportion
of missingness (with respect to one set of variables) up
to the subgroups presenting the maximum number of
them (with respect to one set of variables). The imputa-
tion is made in each step of the algorithm by performing
a segmentation of the complete cases (using as re-
sponse the variable presenting missing values). The
final decision rule is derived using either cross-valida-
tion or boosting, and it is used to impute missing values
in a given subgroup.
CONCLUSION
In the last two decades, computational enhancements
highly contributed to the increase in popularity of tree
structures. Partitioning algorithms and decision tree
induction are two faces of one medal. While the compu-
tational time consuming has been rapidly reducing, the
statistician is making use of more computational inten-
sive procedures to find out an unbiased and accurate
decision rule for new objects. Nevertheless, DTI cannot
result in finding out just a number (the error rate) with
its accuracy. Trees are much more than a decision rule.
Software enhancements through interactive guide
user interface and custom routines should empower the
final user of tree structures making possible to achieve
further important results in the directions of interpret-
ability, identification, and robustness.
356
Decision Tree Induction
REFERENCES
Breiman, L. (1996). Bagging predictors. Machine Learn-
ing, 24, 123-140.
Breiman, L. (1999). Random forestRandom features [tech-
nical report]. Berkeley, CA: University of California.
Breiman, L., Friedman, J.H., Olshen, R.A., & Stone, C.J.
(1984). Classification and regression trees. Belmont,
CA: Wadsworth.
Buntine, W., & Niblett, T. (1992). A further comparison
of splitting rules for decision-tree induction. Machine
Learning, 8, 75-85.
Cappelli, C., Mola, F., & Siciliano, R. (1998). An alter-
native pruning method based on the impurity-complex-
ity measure Proceedings in Computational Statistics:
13th Symposium of COMPSTAT. London: Springer.
Cappelli, C., Mola, F., & Siciliano, R. (2002). A statis-
tical approach to growing a reliable honest tree. Compu-
tational Statistics and Data Analysis, 38, 285-299.
Cestnik, B., & Bratko, I. (1991). On estimating prob-
abilities in tree pruning. Proceedings of the EWSL-91.
Berlin: Springer.
Conversano, C. (2002). Bagged mixture of classifiers
using model scoring criteria. Patterns Analysis & Ap-
plications, 5(4), 351-362.
Conversano, C., Mola, F., & Siciliano, R. (2001). Parti-
tioning algorithms and combined model integration for
data mining. Computational Statistics, 16, 323-339.
Conversano, C., & Siciliano, R. (2004). Incremental
tree-based missing data imputation with lexicographic
ordering. Proceedings of Interface 2003, Fairfax, USA.
Freud, Y., & Schapire, R. (1996). Experiments with a
new boosting algorithm. Proceedings of Machine Learn-
ing, Thirteenth International Conference. Berlin:
Springer.
Hand, D.J., Mannila, H., & Smyth, P. (2001). Principles
of data mining. Cambridge, USA: MIT Press.
Hastie, T., Friedman, J.H., & Tibshirani, R. (2001). The
elements of statistical learning: Data mining, infer-
ence and prediction. New York: Springer Verlag.
Malerba, D., Esposito, F., Ceci, M., & Appice, A. (2004),
Top-down induction of model trees with regression and
splitting nodes. IEEE Transactions on Pattern Analy-
sis and Machine Intelligence, 26, 6.
TEAM LinG
Decision Tree Induction
Mingers, J. (1989a). An empirical comparison of selection
measures for decision tree induction. Machine Learning,
3, 319-342.
Mingers, J. (1989b). An empirical comparison of prun-
ing methods for decision tree induction. Machine Learn-
ing, 4, 227-243.
Mola, F., & Siciliano, R. (1997). A fast splitting algo-
rithm for classification trees. Statistics and Comput-
ing, 7, 209-216.
Morgan, J.N., & Sonquist, J.A. (1963). Problem in the
analysis of survey data and a proposal. Journal of the
American Statistical Association, 58, 415-434.
Oliver, J.J., & Hand, D.J. (1995). On pruning and aver-
aging decision trees. Proceedings of Machine Learn-
ing, the 12th International Workshop. Berlin: Springer.
Quinlan, J.R. (1986). Induction of decision tree. Ma-
chine Learning, 1, 86-106.
Quinlan, J.R. (1987). Simplifying decision tree. Internat.
J. Man-Mach. Studies, 27, 221-234.
Schapire, R.E., Freund, Y., Barlett, P., & Lee, W.S.
(1998). Boosting the margin: A new explanation for the
effectiveness of voting methods. The Annals of Statis-
tics, 26(5), 1998.
Siciliano, R. (1998). Exploratory versus decision trees.
R. Payne, & P. Green (Eds.), Proceedings in computa-
tional statistics (pp. 113-124). Physica-Verlag. London:
Springer.
Siciliano, R., Aria, M., & Conversano, C. (2004). Tree
harvest: Methods, software and applications. In J. Antoch
(Ed.), COMPSTAT 2004 Proceedings (pp. 1807-1814).
Berlin: Springer.
Zhang, H., & Singer, B. (1999). Recursive partitioning
in the health sciences. New York: Springer Verlag.
KEY TERMS
Adaptive Boosting (AdaBoost): An iterative boot-
strap replication of the sample units of the training
sample such that at any iteration, misclassified/worse
predicted cases have higher probability to be included in
the current bootstrap sample, and the final decision rule
is obtained by majority voting.
Bagging (Bootstrap Aggregating): A bootstrap
replication of the sample units of the training sample,
each having the same probability to be included in the
bootstrap sample to generate single prediction/classifi-
cation rules that being aggregated provides a final deci- ,
sion rule consisting in either the average (for regression
problems) or the modal class (for classification problems)
among the single estimates.
Classification Tree: An oriented tree structure ob-
tained by a recursive partitioning of a sample of cases on
the basis of a sequential partitioning of the predictor
space such to obtain internally homogenous groups and
externally heterogeneous groups of cases with respect to
a categorical variable.
Decision Rule: The result of an induction proce-
dure providing the final assignment of a response class/
value to a new object so that only the predictor measure-
ments are known. Such rule can be drawn in the form of
decision tree.
Ensemble: A combination, typically weighted or
unweighted aggregation, of single induction estimators
able to improve the overall accuracy of any single
induction method.
Exploratory Tree: An oriented tree graph formed
by internal nodes and terminal nodes, the former allow-
ing the description of the conditional interaction paths
between the response variable and the predictors,
whereas the latter are labeled by a response class/value.
FAST (Fast Algorithm for Splitting Trees): A
splitting procedure to grow a binary tree using a suitable
mathematical property of the impurity proportional re-
duction measure to find out the optimal split at each
node without trying out necessarily all candidate splits.
Partitioning Tree Algorithm: A recursive algo-
rithm to form disjoint and exhaustive subgroups of
objects from a given group in order to build up a tree
structure.
Production Rule: A tree path characterized by a
sequence of predictor interactions yielding to a spe-
cific label class/value of the response variable.
Pruning: A top-down or bottom-up selective algo-
rithm to reduce the dimensionality of a tree structure in
terms of the number of its terminal nodes.
Random Forest: An ensemble of unpruned trees
obtained by introducing two bootstrap resampling
schema, one on the objects and another one on the
predictors, such that an out-of-bag sample provides the
estimation of the test set error, and suitable measures of
predictor importance are derived for the final interpre-
tation.
357
TEAM LinG
 Decision Tree Induction

Regression Tree: An oriented tree structure obtained

by a recursive partitioning of a sample on the basis of a
sequential partitioning of the predictor space such to
obtain internally homogenous groups and externally het-
erogeneous groups of cases with respect to a numerical
response variable.

358

TEAM LinG

Diabetic Data Warehouses
Joseph L. Breault
Ochsner Clinic Foundation, USA
INTRODUCTION
The National Academy of Sciences convened in 1995
for a conference on massive data sets. The presentation
on health care noted that massive applies in several
dimensions . . . the data themselves are massive, both in
terms of the number of observations and also in terms of
the variables . . . there are tens of thousands of indicator
variables coded for each patient (Goodall, 1995, para-
graph 18). We multiply this by the number of patients in
the United States, which is hundreds of millions.
Diabetic registries have existed for decades. Data-
mining techniques have recently been applied to them in
an attempt to predict diabetes development or high-risk
cases, to find new ways to improve outcomes, and to
detect provider outliers in quality of care or in billing
services (Breault, 2001; He, Koesmarno, Van, & Huang,
2000; Hsu, Lee, Liu, & Ling, 2000; Kakarlapudi, Sawyer, &
Staecker, 2003; Stepaniuk, 1999; Tafeit, Moller, Sudi, &
Reibnegger, 2000).
Diabetes is a major health problem. The long history
of diabetic registries makes it a realistic and valuable
target for data mining.
BACKGROUND
In-depth examination of one such diabetic data ware-
house developed a method of applying data-mining tech-
niques to this type of database (Breault, Goodall, & Fos,
2002). There are unique data issues and analysis prob-
lems with medical transactional databases. The lessons
learned will be applicable to any diabetic database and
perhaps to broader medical databases.
Methods for translating a complex relational medi-
cal database with time series and sequencing informa-
tion to a flat file suitable for data mining are challeng-
ing. We used the classification tree approach with a
binary target variable. While many data mining methods
(neural networks, logistic regression, etc.) could be
used, classification trees have been noted to be appeal-
ing to physicians because much of medical diagnosis
training operates in a fashion similar to classification
trees.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
359
,
MAIN THRUST
Three major challenges are reviewed here: a) understand-
ing and converting the diabetic databases into a data-
mining data table, b) the data mining, and c) utilizing
results to assist clinicians and managers in improving the
health of the population studied.
The Diabetic Database
The diabetic data warehouse we studied included 30,383
diabetic patients during a 42-month period with hun-
dreds of fields per patient.
Understanding the data requires awareness of its
limitations. These data were obtained for purposes other
than research. Clinicians will be aware that billing codes
are not always precise, accurate, and comprehensive.
However, the codes are widely used in outcomes mod-
eling. Epidemiologists and clinicians will be aware that
important predictors of diabetic outcomes are missing
from the database, such as body mass index, family
history of diabetes, time since the onset of diabetes,
diet, and exercise habits. These variables were not elec-
tronically stored and would require going to the paper
chart and patient interviews to obtain.
Developing the Data-Mining Data Table
The major challenge is transforming the data from the
relational structure of the diabetic data warehouse with
its multiple tables to a form suitable for data mining
(Nadeau, Sullivan, Teorey, & Feldman, 2003). Data-min-
ing algorithms are most often based on a single table,
within which is a record for each individual, and the fields
contain variable values specific to the individual. We call
this the data-mining data table. The most portable format
for the data-mining data table is a flat file, with one line for
each individual record.
SQL statements on the data warehouse create the flat
file output that the data-mining software then reads. The
steps are as follows:
Review each table of the relational database and
select the fields to export.
TEAM LinG

Determine the interactions between the tables in the
relational database.
Define the layout of the data-mining data table.
Specify patient inclusion and exclusion criteria.
What is the time interval? What are the minimum
and maximum number of records (e.g., clinic vis-
its or outcome measures) each patient must have
to be included? What relevant fields can be miss-
ing and still include the individual in the data-
mining data table?
Extract data, including the stripping of patient
identifiers to protect human subjects.
Determine how to handle missing values (Duhamel,
Nuttens, Devos, Picavet, & Beuscart, 2003)
Perform sanity checks on the data-mining data
table, for example, that the minimum and maxi-
mum of each variable make clinical sense.
Handling time series medical data is challenging for
data-mining software. One example in our study is the
HgbA1c, the key measure of glycemic control. This is
closely related to clinical outcomes and complication
rates in diabetes. Health care costs increase markedly
with each 1% increase in baseline HgbA1c; patients with
an HgbA1c of 10% versus 6% had a 36% increase in 3-
year medical costs (Blonde, 2001). How should this
time series variable be transformed from the relational
database to a vector (column) in the data-mining data
table? A given diabetic patient may have many of these
HgbA1c results. We could pick the last one, the first, a
median or mean value. Because the trend over time for
this variable is important, we could choose the slope of
its regression line over time. However, a linear function
may be a good representation for some patients, but a
very bad one for others that may be better represented by
an upside down U curve. This difficulty is a problem for
most repeated laboratory tests. Some information will
be lost in the creation of the data-mining data table.
We used the average HgbA1c for a given patient and
excluded patients who did not have at least two HgbA1c
results in the data warehouse. We repartitioned this
average HgbA1c into a binary variable based on a mean-
ingful clinical cut-point of 9.5%. Experts agree that an
HgbA1c >9.5% is a bad outcome, or a medical quality
error, no matter what the circumstances (American
Medical Association, Joint Commission on Accredita-
tion of Healthcare Organizations, & National Commit-
tee for Quality Assurance, 2001).
Our final data-mining data table had 15,902 patients
(rows). Mean HgbA1c > 9.5% was the target variable,
and the 10 predictors were age, sex, emergency depart-
ment visits, office visits, comorbidity index,
dyslipidemia, hypertension, cardiovascular disease, re-
tinopathy, and end stage renal disease. All these patients
360
Diabetic Data Warehouses
had at least two HgbA1c tests and at least two office
visits, the criteria we used for minimal continuity in this
42-month period.
Data-Mining Technique
We used the classification tree approach as standard-
ized in the CART software by Salford Systems. As
detailed in Hand, Mannila, and Smyth (2001), the prin-
ciple behind all tree models is to recursively partition
the input variable space to maximize purity in the termi-
nal tree nodes. The partitioning split in any cell is done
by searching each possible threshold for each variable
to find the threshold split that leads to the greatest
improvement in the purity score of the resultant nodes.
Hence, this is a monothetic process, which may be a
limitation of this method in some circumstances.
In CARTs defaults, the Gini splitting criteria are
used, although other methods are options. This could
recursively continue to the point of perfect purity,
which would sometimes mean only one patient in a
terminal node. But overfitting of the data does not help
in accurately classifying another data set. Therefore, we
divide the data randomly into learning and test sets. The
number of trees generated is halted or pruned back by
how accurately the classification tree created from the
learning set can predict classification in the test set.
Cross-validation is another option for doing this, though
in the CART softwares defaults this is limited to n =
3000. This could be changed higher to use our full data
set, but some CART consultants note, The n-fold cross-
validation technique is designed to get the most out of
datasets that are too small to accommodate a hold-out or
test sample. Once you have 3,000 records or more, we
recommend that a separate test set be used (Timberlake-
Consultants, 2001). The original CART creators recom-
mended dividing the data into test and learning samples
whenever there were more than 1,000 cases, with cross-
validation being preferable in smaller data sets (Breiman,
Friedman, Olshen, & Stone, 1984).
The 10 predictor variables were used with the binary
target variable of the HgbA1c average (cut-point of
9.5%) in an attempt to find interesting patterns that may
have management or clinical importance and are not
already known.
The variables that are most important to classifica-
tion in the optimal CART tree were age (100, where the
most important variable is arbitrarily given a relative
score of 100), number of office visits (51), comorbidity
index (CMI) (44), cardiovascular disease (16), choles-
terol problems (17), number of emergency room visits
(7), and hypertension (0.6).
CART can be used for multiple purposes. Here we
want to find clusters of deviance from glycemic control.
TEAM LinG
Diabetic Data Warehouses
With no analysis, the rate of bad glycemic control in the
learning sample is 13.2%. We want to find nodes that
have higher rates of bad glycemic control. The first split
in node 1 of the tree is based on an age of 65.6 (CART
sends all cases less than or equal to the cut-point to the
left and greater than the cut-point to the right). In node 2
(<=65.6 years of age), we have bad glycemic control in
19.4%.
If we look at the tree to find terminal nodes (TN)
where the percentage of all the patients in those nodes
having a bad HgbA1c is greater than the 19.4% true of
those <= 65.6 years of age, we identify 4 of the 10 TNs
in the learning sample.
The purer the nodes we limit ourselves to, the less
percentage of the overall population with bad glycemic
control we get. With no analysis in the learning set, we
can capture all 1052 patients with bad glycemic control
but must target the entire group of 7953. When we limit
ourselves to the purer node 2, we capture 74% of those
with bad glycemic control by targeting only 50% of the
population. If we limit ourselves to TN1, we capture 49%
of those with bad glycemic control by targeting only
27% of the population. If we use more complicated
criteria by combining the 4 TNs with worst glycemic
control, we capture 54% of those with bad glycemic
control by targeting only 30% of the population.
The classification errors in the learning and test
samples are substantial, as a quarter of the bad glycemic
control patients are missed in the CART analysis. CART
is doing a good job with the 10 predictor variables it is
given, but more accurate prediction requires additional
variables not in our database.
Adjustment to defaults in CART can give better re-
sults defined as capturing a larger percentage of those
with bad glycemic control within a smaller percentage of
the population. However, the complexity of the formulas
to identify the population is difficult for managers to
use. Not only must managers identify the persons, but
they must get enough of a feel for what the population
characteristics are to know what interventions are likely
to be helpful. This is more intuitive for those who are
younger than 55 or younger than 65 than it is for those
who satisfy 0.451*(AGE) + 0.893*(CMI) <= 32.5576.
Results
From this CART analysis, the most important variable
associated with a bad HgbA1c score is age less than 65.
Those less than 65.6 years old are almost three times as
likely to have bad glycemic control than those who are
older. The odds ratio that someone is less than 65.6 years
old if they have a bad HgbA1c (average reading > 9.5%)
is 3.18 (95% CI: 2.87, 3.53) (Fos & Fine, 2000). Simi-
larly, the odds ratio that someone is younger than 55.2
years rather than older than 65.6 if they have a bad
glycemic control is 4.11 (95% CI:3.60, 4.69). This is ,
surprising information to most clinicians. Similar find-
ings were recently reported with vascular endpoints in
diabetic patients (Miyaki, Takei, Watanabe, Nakashina,
& Omae, 2002).
The case may be that those with the worst glycemic
control die young and never make it to the older group.
Although this is an interesting theoretical explanation,
nevertheless, these numbers represent real patients
that need help with their glycemic control now.
If we want to target diabetics with bad HgbA1c
values, the odds of finding them are 3.2 times as high in
diabetic patients younger than 65.6 years than those
who are older and 4.1 times as high in those who are
younger than 55 than those over 65. This information is
clinically important because the younger group has so
many more years of life left to develop diabetic com-
plications from bad glycemic control. This is espe-
cially helpful because it tells us which population to
target interventions at even before we have the HgbA1c
values to show us. Health maintenance organizations
and public health workers may want to explore what
educational interventions can be successfully directed
to younger diabetics (younger than 65, especially
younger than 55) who are much more likely to have bad
glycemic control than the geriatric patients.
FUTURE TRENDS
Areas that need further work to fully utilize data mining
in health care include time-series issues, sequencing
information, data-squashing technologies, and a tight
integration of domain expertise and data-mining skills.
We have already discussed time-series issues. This
has been investigated but needs further exploration in
health care data mining (Bellazzi, Larizza, Magni,
Montani, & Stefanelli, 2000; Goodall, 1999; Tsien,
2000).
The sequence of various events may hold meaning
important to a study. For example, a patient may have
better glycemic control, manifested in improved
HgbA1c values, especially when the patient had an
office visit with a physician within a month of a previ-
ous HgbA1c. Perhaps this is meaningful information
that implies that the proper sequence of physician
visits relative to HgbA1c measurements is an important
predictor of good outcomes. All this information is
located in the relational database, but we must ferret it
out by having in advance an idea that a sequence of this
sort may be important and then searching for such
associations. There may be many such sequences in-
volving interactions between hospital, clinic, pharmacy,
361
TEAM LinG

and laboratory variables. Regrettably, no amount of data
mining will be able to extract sequence associations where
we have not thought to extract the prerequisite variables
from the relational database into the data-mining data
table. In the ideal data-mining scenario, software could
interface directly with the relational database and extract
all possibly meaningful sequences for us to review, and
domain experts would then sort through the list. This
issue has begun to get attention (Deroski & Lavra , 2001)
and will need to be addressed in future health care data
mining.
It has been shown that using a data-squashing algo-
rithm to reduce a massive data set is more powerful and
accurate than using a random sample (DuMouchel,
Volinsky, Johnson, Cortes, & Pregibon, 1999). Squash-
ing is a form of lossy compression that attempts to
preserve statistical information (DuMouchel, 2001).
These newer data-squashing techniques may be a better
approach than random sampling in massive data sets.
These techniques also protect human subjects privacy.
Transactional health care data mining, exemplified
in the diabetic data warehouses discussed previously,
involves a number of tricky data transformations that
require close collaboration between domain experts
and data miners (Breault & Goodall, 2002). Even with
ideal collaboration or overlapping expertise, we need to
develop new ways to extract variables from relational
databases containing time-series and sequencing infor-
mation. Part of the answer lies in collaborative groups
that can have additional insights. Part of the answer lies
in the further development of data-mining tools that act
directly on a relational database without transformation
to explicit data arrays. In some circumstances, it may be
useful to produce several data-mining data tables to
independently data mine and then combine the results.
This may be particularly useful when different
granularities of attributes are glossed over by the use of
a single data-mining data table.
CONCLUSION
Data mining is valuable in discovering novel associa-
tions in diabetic databases that can prove useful to
clinicians and administrators. This may also be the case
for many other health care problems.
REFERENCES
American Medical Association, Joint Commission on
Accreditation of Healthcare Organizations, & National
Committee for Quality Assurance. (2001). Coordi-
362
Diabetic Data Warehouses
nated performance measurement for the management of
adult diabetes. Retreived March 24, 2005, from http://
www.ama-assn.org/ama/upload/mm/370/nr.pdf
Bellazzi, R., Larizza, C., Magni, P., Montani, S., &
Stefanelli, M. (2000). Intelligent analysis of clinical
time series: An application in the diabetes mellitus
domain. Artificial Intelligence Med, 20(1), 37-57.
Blonde, L. (2001). Epidemiology, costs, consequences,
and pathophysiology of type 2 diabetes: An American
epidemic. Ochsner Journal, 3(3), 126-131.
Breault, J. L. (2001). Data mining diabetic databases: Are
rough sets a useful addition? In E. Wegman, A. Braverman,
A. Goodman, & P. Smyth (Eds.), Computing science and
statistics (pp. 597-606). Interface Foundation of North
America, Inc, 33, Fairfax Station, VA.
Breault, J. L., & Goodall, C. R. (2002, January). Mathemati-
cal challenges of variable transformations in data mining
diabetic data warehouses. Paper presented at the Math-
ematical Challenges in Scientific Data Mining Confer-
ence, Los Angeles, CA.
Breault, J. L., Goodall, C. R., & Fos, P. J. (2002). Data
mining a diabetic data warehouse. Artificial Intelli-
gence Med, 26(1-2), 37-54.
Breiman, L. (1984). Classification and regression
trees. Belmont, CA: Wadsworth International.
Duhamel, A., Nuttens, M.C., Devos, P., Picavet, M., &
Beuscart, R. (2003). A preprocessing method for improv-
ing data mining techniques: Application to a large medical
diabetes database. Stud Health Technol Inform, 95, 269-
274.
DuMouchel, W. (2001). Data squashing: Constructing
summary data sets. In E. Wegman, A. Braverman, A.
Goodman, & P. Smyth (Eds.), Computing science and
statistics. Interface Foundation of North America, Inc.,
Fairfax Station, VA.
DuMouchel, W., Volinsky, C., Johnson, T., Cortes, C., &
Pregibon, D. (1999, August). Squashing flat files flatter.
Proceedings of the Fifth ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining,
USA.
Deroski, S., & Lavra , N. (2001). Relational data mining.
New York: Springer.
Fos, P. J., & Fine, D. J. (2000). Designing health care
for populations: Applied epidemiology in health care
administration. San Francisco: Jossey-Bass.
Goodall, C. (1995). Massive data sets in healthcare. In
(Chair: Jon R. Kettenring), Massive data sets. Paper pre-
TEAM LinG
Diabetic Data Warehouses
sented at the meeting of the Committee on Applied and
Theoretical Statistics at the National Academy of Sciences,
National Research Council, Washington, DC. Retrieved
March 29, 2005, from http://bob.nap.edu/html/massdata/
media/cgoodall-t.html
Goodall, C. R. (1999). Data mining of massive datasets in
healthcare. Journal of Computational and Graphical
Statistics, 8(3), 620-634.
Hand, D. J., Mannila, H., & Smyth, P. (2001). Principles
of data mining. Cambridge, MA: MIT Press.
He, H., Koesmarno, Van, & Huang. (2000). Data mining
in disease management: A diabetes case study. In R.
Mizoguchi & J. K. Slaney (Eds.), Proceedings of the
Sixth Pacific Rim International Conference on Artifi-
cial Intelligence: Topics in artificial intelligence (p.
799). New York: Springer.
Hsu, W. (2000, August). Exploration mining in diabetic
patients databases: Findings and conclusions. Proceed-
ings of the Sixth ACM SIGKDD International Confer-
ence on Knowledge Discovery and Data Mining, USA.
Kakarlapudi, V., Sawyer, R., & Staecker, H. (2003). The
effect of diabetes on sensorineural hearing loss. Otol
Neurotol, 24(3), 382-386.
Miyaki, K., (2002). Novel statistical classification
model of type 2 diabetes mellitus patients for tailor-
made prevention using data mining algorithm. J
Epidemiol, 12(3), 243-248.
Nadeau, T. P., (2003). Applying database technology to
clinical and basic research bioinformatics projects. J
Integr Neurosci, 2(2), 201-217.
Stepaniuk, J. (1999, June). Rough set data mining of
diabetes data. In Z. Ras & A. Skowron (Eds.), Proceed-
ings of the 11th International Symposium onVol. Foun-
dations of intelligent systems (pp. 457-465). New York:
Springer.
Tafeit, E., Moller, Sudi, & Reibnegger. (2000). ROC
and CART analysis of subcutaneous adipose tissue to-
pography (SAT-Top) in type-2 diabetic women and
healthy females. American Journal of Human Biology,
12, 388-394.
Timberlake-Consultants. (2001). CART frequently asked
questions. Retrieved October 21, 2001, from http:// ,
www.timberlake.co.uk/software/cart/cartfaq1.htm#q23
Tsien, C. L. (2000). Event discovery in medical time-series
data. Proceedings of the AMIA Symposium, 858-862.
KEY TERMS
Co-Morbidity Index: A composite variable that
gives a measure of how many other medical problems
someone has in addition to the one being studied.
Data Mining Data Table: The flat file constructed
from the relational database that is the actual table used
by the data-mining software.
Glycemic Control: Tells how well controlled are
the sugars of a diabetic patient. Usually measured by
HgbA1c.
HgbA1c: Blood test that measures the percent of
receptors on a red blood cell that are saturated with
glucose. This is translated into a measure of how sugars
have averaged over the last few months. Normal is less
than 6, depending on the laboratory standards.
Medical Transactional Database: The database
created from the billing and required reporting transac-
tions of a medical practice. Clinical experience is some-
times required to understand gaps and inadequacies in
the collected data.
Monothetic Process: In a classification tree, when
data at each node are split on just one variable rather than
several variables.
Recursive Partitioning: The method used to di-
vide data at each node of a classification tree. At the top
node, every variable is examined at every possible value
to determine which variable split will produce the maxi-
mum and minimum amounts of the target variable in the
daughter nodes. This is recursively done for each addi-
tional node.
363
TEAM LinG
364
Discovering an Effective Measure in Data
Mining
Takao Ito
Ube National College of Technology, Japan
INTRODUCTION
One of the most important issues in data mining is to
discover an implicit relationship between words in a large
corpus and labels in a large database. The relationship
between words and labels often is expressed as a function
of distance measures. An effective measure would be
useful not only for getting the high precision of data
mining, but also for time saving of the operation in data
mining. In previous research, many measures for calculat-
ing the one-to-many relationship have been proposed,
such as the complementary similarity measure, the mutual
information, and the phi coefficient. Some research showed
that the complementary similarity measure is the most
effective. The author reviewed previous research related
to the measures in one-to-many relationships and pro-
posed a new idea to get an effective one, based on the
heuristic approach in this article.
BACKGROUND
Generally, the knowledge discover in databases (KDD)
process consists of six stages: data selection, cleaning,
enrichment, coding, data mining, and reporting (Adriaans
& Zantinge, 1996). Needless to say, data mining is the
most important part in the KDD. There are various tech-
niques, such as statistical techniques, association rules,
and query tools in a database, for different purposes in
data mining. (Agrawal, Mannila, Srikant, Toivonen &
Verkamo, 1996; Berland & Charniak, 1999; Caraballo, 1999;
Fayyad, Piatetsky-Shapiro, Smyth & Uthurusamy, 1996;
Han & Kamber, 2001).
When two words or labels in a large database have
some implicit relationship with each other, one of the
different purposes is to find out the two relative words or
labels effectively. In order to find out relationships be-
tween words or labels in a large database, the author
found the existence of at least six distance measures after
reviewing previously conducted research.
The first one is the mutual information proposed by
Church and Hanks (1990). The second one is the confi-
dence proposed by Agrawal and Srikant (1995). The third
one is the complementary similarity measure (CSM) pre-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
sented by Hagita and Sawaki (1995). The fourth one is the
dice coefficient. The fifth one is the Phi coefficient. The
last two are both mentioned by Manning and Schutze
(1999). The sixth one is the proposal measure (PM) sug-
gested by Ishiduka, Yamamoto, and Umemura (2003). It is
one of the several new measures developed by them in
their paper.
In order to evaluate these distance measures, formulas
are required. Yamamoto and Umemura (2002) analyzed
these measures and expressed them in four parameters of
a, b, c, and d (Table 1).
Suppose that there are two words or labels, x and y, and
they are associated together in a large database. The mean-
ings of these parameters in these formulas are as follows:
a. The number of documents/records that have x and
y both.
b. The number of documents/records that have x but
not y.
c. The number of documents/records that do not have
x but do have y.
d. The number of documents/records that do not have
either x or y.
n. The total number of parameters a, b, c, and d.
Umemura (2002) pointed out the following in his paper:
Occurrence patterns of words in documents can be
expressed as binary. When two vectors are similar, the
two words corresponding to the vectors may have some
implicit relationship with each other. Yamamoto and
Umemura (2002) completed their experiment to test the
validity of these indexes under Umemuras concept. The
result of the experiment of distance measures without
noisy pattern from their experiment can be seen in Figure
1 (Yamamoto & Umemura, 2002).
The experiment by Yamamoto and Umemura (2002)
showed that the most effective measure is the CSM. They
indicated in their paper as follows: All graphs showed
that the most effective measure is the complementary
similarity measure, and the next is the confidence and the
third is asymmetrical average mutual information. And the
least is the average mutual information (Yamamoto and
Umemura, 2002). They also completed their experiments
with noisy pattern and found the same result (Yamamoto
& Umemura, 2002).
TEAM LinG
Discovering an Effective Measure in Data Mining

Table 1. Kind of distance measures and their formulas

No Kin d of Dista n ce M ea sur es F or m ul a

,
an
1 th e m u tu a l in for m a t ion I ( x1 ; y1 ) = log
(a + b)(a + c)
a
2 th e c on fiden ce conf (Y | X ) =
a+c
ad bc
3 th e comp leme n ta r y si mi la r it y me a su r e Sc (F , T ) =
(a + c)(b + d )
2a
4 th e d ice coefficien t S d (F ,T ) =
( a + b ) + (a + c )
ad bc
5 th e P h i co effici en t DE =
(a + b)(a + c)(b + d )(c + d )
a 2b
6 th e pr oposal m eas u r e S (F , T ) =
1+ c

Figure 1. Result of the experiment of distance measures without noisy pattern

MAIN THRUST effective measure in this article (Aho, Kernighan &

Weinberger, 1995).
How to select a distance measure is a very important issue,
because it has a great influence on the result of data RESULT OF THE THREE KINDS OF
mining (Fayyad, Piatetsky-Shapiro & Smyth, 1996; EXPERIMENTS
Glymour, Madigan, Pregibon & Smyth, 1997) The author
completed the following three kinds of experiments, based All of these three kinds of experiments are executed under
upon the heuristic approach, in order to discover an the following conditions. In order to discover an effective

365

TEAM LinG
 Discovering an Effective Measure in Data Mining

measure, the author selected actual data of a places name,
such as the name of prefecture and the name of a city in
Japan from the articles of a nationally circulated newspa-
per, the Yomiuri. The reasons for choosing a places name
are as follows: first, there are one-to-many relationships
between the name of a prefecture and the name of a city;
second, the one-to-many relationship can be checked
easily from the maps and telephone directory. Generally
speaking, the first name, such as the name of a prefecture,
consists of another name, such as the name of a city. For
instance, Fukuoka City is geographically located in Fukuoka
Prefecture, and Kitakyushu City also is included in Fukuoka
Prefecture, so there are one-to-many relationships be-
tween the name of the prefecture and the name of the city.
The distance measure would be calculated with a large
database in the experiments. The experiments were ex-
ecuted as follows:
Step 1. Establish the database of the newspaper.
Step 2. Choose the prefecture name and city name from the
database mentioned in step 1.
Step 3. Count the number of parameters a, b, c, and d from
the newspaper articles.
Step 4. Then, calculate the distance measure adopted.
Step 5. Sort the result calculated in step 4 in descent order
upon the distance measure.
Step 6. List the top 2,000 from the result of step 5.
Step 7. Judge the one-to-many relationship whether it is
correct or not.
Step 8. List the top 1,000 as output data and count its
number of correct relationships.
Step 9. Finally, an effective measure will be found from the
result of the correct number.
To uncover an effective one, two methods should be
considered. The first one is to test various combinations
of each variable in distance measure and to find the best
combination, depending upon the result. The second one
is to assume that the function is a stable one and that only
part of the function can be varied.
The first one is the experiment with the PM.
The total number of combinations of five variables is
3,125. The author calculated all combinations, except for
the case when the denominator of the PM becomes zero.
The result of the top 20 in the PM experiment, using a
years amount of articles from the Yomiuri in 1991, is
calculated as follows.
In Table 2, the No. 1 function of a11c1 has a highest
a 1 1 a
correct number, which means S ( F , T ) = = .
c +1 1+ c
1 a 1 a
The No. 11 function of 1a1c0 means S ( F , T ) = = .
c+0 c
The rest function in Table 2 can be read as mentioned
previously.
This result appears to be satisfactory. But to prove
whether it is satisfactory or not, another experiment
should be done. The author adopted the Phi coefficient
and calculated its correct number. To compare with the
result of the PM experiment, the author completed the
experiment of iterating the exponential index of denomi-
nator in the Phi coefficient from 0 to 2, with 0.01 step
based upon the idea of fractal dimension in the complex
theory instead of fixed exponential index at 0.5. The result
of the top 20 in this experiment, using a years amount of
articles from the Yomiuri in 1991, just like the first experi-
ment, can be seen in Table 3.
Compared with the result of the PM experiment men-
tioned previously, it is obvious that the number of cor-
rect relationships is less than that of the PM experiment;
therefore, it is necessary to uncover a new, more effective
measure. From the previous research done by Yamamoto
and Umemura (2002), the author found that an effective
measure is the CSM.
The author completed the third experiment using the
CMS. The experiment began by iterating the exponential
index of the denominator in the CSM from 0 to 1, with 0.01
steps just like the second experiment. Table 4 shows the

Table 2. Result of the top 20 in the PM experiment

No Function Correct Number No Function Correct Number

1 a11c1 789 11 1a1c0 778
2 a111c 789 12 1a10c 778
3 1a1c1 789 13 11acc 778
4 1a11c 789 14 11ac0 778
5 11ac1 789 15 11a0c 778
6 11a1c 789 16 aa1cc 715
7 a11cc 778 17 aa1c0 715
8 a11c0 778 18 aa10c 715
9 a110c 778 19 a1acc 715
10 1a1cc 778 20 a1ac0 715

366

TEAM LinG
Discovering an Effective Measure in Data Mining

Table 3. Result of the top 20 in the Phi coefficient experiment

No Exponential Index Correct Number No Exponential index Correct Number

,
1 0.26 499 11 0.20 465
2 0.25 497 12 0.31 464
3 0.27 495 13 0.19 457
4 0.24 493 14 0.32 445
5 0.28 491 15 0.18 442
6 0.23 488 16 0.17 431
7 0.29 480 17 0.16 422
8 0.22 478 18 0.15 414
9 0.21 472 19 0.14 410
10 0.30 470 20 0.13 403

Table 4. Result of the top 20 in the CSM experiment

1991 1992 1993 1994 1995 1996 1997

E.I. C.N. E.I. C.N. E.I. C.N. E.I. C.N. E.I. C.N. E.I. C.N. E.I. C.N.
0.73 850 0.75 923 0.79 883 0.81 820 0.85 854 0.77 832 0.77 843
0.72 848 0.76 922 0.80 879 0.82 816 0.84 843 0.78 828 0.78 842
0.71 846 0.74 920 0.77 879 0.8 814 0.83 836 0.76 826 0.76 837
0.70 845 0.73 920 0.81 878 0.76 804 0.86 834 0.75 819 0.79 829
0.74 842 0.72 917 0.78 878 0.75 800 0.82 820 0.74 818 0.75 824
0.63 841 0.77 909 0.82 875 0.79 798 0.88 807 0.71 818 0.73 818
0.69 840 0.71 904 0.76 874 0.77 798 0.87 805 0.73 812 0.74 811
0.68 839 0.78 902 0.75 867 0.78 795 0.89 803 0.70 803 0.72 811
0.65 837 0.79 899 0.74 864 0.74 785 0.81 799 0.72 800 0.71 801
0.64 837 0.70 898 0.73 859 0.73 769 0.90 792 0.69 800 0.80 794
0.62 837 0.69 893 0.72 850 0.72 761 0.80 787 0.79 798 0.81 792
0.66 835 0.80 891 0.71 833 0.83 751 0.91 777 0.68 788 0.83 791
0.67 831 0.81 884 0.83 830 0.71 741 0.79 766 0.67 770 0.82 790
0.75 828 0.68 884 0.70 819 0.70 734 0.92 762 0.80 767 0.70 789
0.61 828 0.82 883 0.69 808 0.84 712 0.93 758 0.66 761 0.84 781
0.76 824 0.83 882 0.83 801 0.69 711 0.78 742 0.81 755 0.85 780
0.60 818 0.84 872 0.68 790 0.85 706 0.94 737 0.65 749 0.86 778
0.77 815 0.67 870 0.85 783 0.86 691 0.77 730 0.82 741 0.87 776
0.58 814 0.66 853 0.86 775 0.68 690 0.76 709 0.83 734 0.69 769
0.82 813 0.85 852 0.67 769 0.87 676 0.95 701 0.64 734 0.68 761

Note: C.N. means the correct number, and E.I. means the exponential index.

result of the top 20 in this experiment, using a years
amount of articles from the Yomiuri from 1991-1997.
It is obvious by these results that the CSM is more
effective than the PM and the Phi coefficient. The relation-
ship of the complete result of the exponential index of the
denominator in the CSM and the correct number can be
seen as in Figure 2.
The most effective exponential index of the denomina-
tor in the CSM is from 0.73 to 0.85, but not as 0.5, as many
researchers believe. It would be hard to get the best result
with the usual method using the CSM.
Determine the Most Effective
Exponential Index of the Denominator
in the CSM
To discover the most effective exponential index of the
denominator in the CSM, a calculation about the relation-
ship between the exponential index and the total number
of documents of n was carried out, but none was found.
In fact, it is hard to consider that the exponential index
would vary with the difference of the number of documents.

367

TEAM LinG

Figure 2. Relationships between the exponential indexes of denominator in the CSM and the correct number
The C orrect num ber
1000
900
800
700
600
500
400
300
200
100
0
0 0.08 0.16 0.24
The ExponentialIndex of Denom inator In C SM
The details of the four parameters of a, b, c, and d in
Table 4 are listed in Table 5.
The author adopted the average of the cumulative
exponential index to find the most effective exponential
index of the denominator in the CSM. Based upon the
results in Table 4, calculations for the average of the
cumulative exponential index of the top 20 and those
results are presented in Figure 3.
From the results in Figure 3, it is easy to understand
that the exponential index converges at a constant value
of about 0.78. Therefore, the exponential index could be
fixed at a certain value, and it will not vary with the size of
documents.
In this article, the author puts the exponential index at
0.78 to forecast the correct number. The gap between the
calculation result forecast by the revised method and the
maximum result of the third experiment is illustrated in
Figure 4.
FUTURE TRENDS
Many indexes have been developed for the discovery of
an effective measure to determine an implicit relationship
between words in a large corpus or labels in a large
database. Ishiduka, Yamamoto, and Umemura (2003) re-
ferred to part of them in their research paper. Almost all of
these approaches were developed by a variety of math-
ematical and statistical techniques, a conception of neural
network, and the association rules.
368
Discovering an Effective Measure in Data Mining
1991
1992
1993
1994
1995
1996
1997
0.32 0.4 0.48 0.56 0.64 0.72 0.8 0.88 0.96
For many things in the world, part of the economic
phenomena obtain a normal distribution referred to in
statistics, but others do not, so the author developed the
CSM measure from the idea of fractal dimension in this
article. Conventional approaches may be inherently inac-
curate, because usually they are based upon linear math-
ematics. The events of a concurrence pattern of correlated
pair words may be explained better by nonlinear math-
ematics. Typical tools of nonlinear mathematics are com-
plex theories, such as chaos theory, cellular automaton,
percolation model, and fractal theory. It is not hard to
predict that many more new measures will be developed
in the near future, based upon the complex theory.
CONCLUSION
Based upon previous research of the distance measures,
the author discovered an effective measure of distance
measure in one-to-many relationships with the heuristic
approach. Three kinds of experiments were conducted,
and it was confirmed that an effective measure is the CSM.
In addition, it was discovered that the most effective
exponential of the denominator in the CSM is 0.78, not
0.50, as many researchers believe.
A great deal of work still needs to be carried out, and
one of them is the meaning of the 0.78. The meaning of the
most effective exponential of denominator 0.78 in the CSM
should be explained and proved mathematically, although
its validity has been evaluated in this article.
TEAM LinG
Discovering an Effective Measure in Data Mining

Table 5. Relationship between the maximum correct number and their parameters
,
Year Exponential Index Correct Number a b c d n
1991 0.73 850 2,284 46,242 3,930 1,329 53,785
1992 0.75 923 453 57,636 1,556 27 59,672
1993 0.79 883 332 51,649 1,321 27 53,329
1994 0.81 820 365 65,290 1,435 36 67,126
1995 0.85 854 1,500 67,914 8,042 190 77,646
1996 0.77 832 636 56,529 2,237 2,873 62,275

Figure 3. Relationships between the exponential index and the average of the number of the cumulative exponential
index

T he exponential index
0.795

0.79

0.785

0.78

0.775

0.77

0.765

0.76
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
T he number of the cumulative exponential index

ACKNOWLEDGMENTS ings of the ACM SIGMOD Conference on Management

The author would like to express his gratitude to the
following: Kyoji Umemura, Professor, Toyohashi Univer-
sity of Technology; Taisuke Horiuchi, Associate Profes-
sor, Nagano National College of Technology; Eiko
Yamamoto, Researcher, Communication Research Labo-
ratories; Yuji Minami, Associate Professor, Ube National
College of Technology; Michael Hall, Lecturer, Nakamura-
Gakuen University; who suggested a large number of
improvements, both in content and computer program.
REFERENCES
Adriaans, P., & Zantinge, D. (1996). Data mining. Addison
Wesley Longman Limited.
Agrawal, R., & Srikant, R. (1995). Mining of association
rules between sets of items in large databases. Proceed-
of Data.
Agrawal, R. et al. (1996). Fast discovery of association
rules, advances in knowledge discovery and data min-
ing. Cambridge, MA: MIT Press.
Aho, A.V., Kernighan, B.W., & Weinberger, P.J. (1989).
The AWK programming language. Addison-Wesley Pub-
lishing Company, Inc.
Berland, M., & Charniak, E. (1999). Finding parts in very
large corpora. Proceedings of the 37th Annual Meeting of
the Association for Computational Linguistics.
Caraballo, S.A. (1999). Automatic construction of a
hypernym-labeled noun hierarchy from text. Proceedings
of the Association for Computational Linguistics.
Church, K.W., & Hanks, P. (1990). Word association
norms, mutual information, and lexicography. Computa-
tional Linguistics, 16(1), 22-29.

369

TEAM LinG

Figure 4. Gaps between the maximum correct number and the correct number forecasted by the revised method
1000
900
800
700
600
500
400
300
200
100
0
1991 1992 1993 1994 1995
Fayyad, U.M. et al. (1996). Advances in knowledge dis-
covery and data mining.AAAI Press/MIT Press.
Fayyad, U.M., Piatetsky-Shapiro, G., & Smyth, P.
(1996). The KDD process for extracting useful knowl-
edge from volumes of data. Communications of the
ACM, 39(11), 27-34.
Glymour, C. et al. (1997). Statistical themes and lessons for
data mining. Data Mining and Knowledge Discover, 1(1),
11-28.
Hagita, N., & Sawaki, M. (1995). Robust recognition of
degraded machine-printed characters using complimen-
tary similarity measure and error-correction learning. Pro-
ceedings of the SPIEThe International Society for
Optical Engineering.
Han, J., & Kamber, M. (2001). Data mining. Morgan
Kaufmann Publishers.
Ishiduka, T., Yamamoto, E., & Umemura, K. (2003). Evalu-
ation of a function presumes the one-to-many relation-
ship [unpublished research paper] [Japanese edition]).
Manning, C.D., & Schutze, H. (1999). Foundations of
statistical natural language processing. MIT Press.
Umemura, K. (2002). Selecting the most highly correlated
pairs within a large vocabulary. Proceedings of the
COLING Workshop SemaNet02, Building and Using
Semantic Networks.
370
Discovering an Effective Measure in Data Mining
m axim um correct num ber
correct num ber focasted by revised
m ethod
1996 1997
Yamamoto, E., & Umemura, K. (2002). A similarity measure
for estimation of one-to-many relationship in corpus
[Japanese edition]. Journal of Natural Language Pro-
cessing, 9, 45-75.
KEY TERMS
Complementary Similarity Measurement: An index
developed experientially to recognize a poorly printed
character by measuring the resemblance of the correct
pattern of the character expressed in a vector. The author
calls this a diversion index to identify the one-to-many
relationship in the concurrence patterns of words in a
large corpus or labels in a large database in this article.
Confidence: An asymmetric index that shows the
percentage of records for which A occurred within the
group of records and for which the other two, X and Y,
actually occurred under the association rule of X, Y => A.
Correct Number: For example, if the city name is
included in the prefecture name geographically, the au-
thor calls it correct. So, in this index, the correct number
indicates the total number of correct one-to-many rela-
tionship calculated on the basis of the distance measures.
Distance Measure: One of the calculation techniques
to discover the relationship between two implicit words in
a large corpus or labels in a large database from the
viewpoint of similarity.
TEAM LinG
Discovering an Effective Measure in Data Mining
Mutual Information: Shows the amount of informa-
tion that one random variable x contains about another y.
In other words, it compares the probability of observing
x and y, together with the probabilities of observing x and
y independently.
Phi Coefficient: One metric for corpus similarity mea-
sured upon the chi square test. It is an index to calculate
the frequencies of the four parameters of a, b, c, and d in
documents based upon the chi square test with the
frequencies expected for independence. ,
Proposal Measure: An index to measure the concur-
rence frequency of the correlated pair words in docu-
ments, based upon the frequency of two implicit words
that appear. It will have high value, when the two implicit
words in a large corpus or labels in a large database occur
with each other frequently.
371
TEAM LinG
372

Discovering Knowledge from XML Documents

Richi Nayak
Queensland University of Technology, Australia

INTRODUCTION BACKGROUND: WHAT IS XML

XML is the new standard for information exchange and
retrieval. An XML document has a schema that defines
the data definition and structure of the XML document
(Abiteboul et al., 2000). Due to the wide acceptance of
XML, a number of techniques are required to retrieve
and analyze the vast number of XML documents. Auto-
matic deduction of the structure of XML documents for
storing semi-structured data has been an active subject
among researchers (Abiteboul et al., 2000; Green et al.,
2002). A number of query languages for retrieving data
from various XML data sources also has been developed
(Abiteboul et al., 2000; W3c, 2004). The use of these
query languages is limited (e.g., limited types of inputs
and outputs, and users of these languages should know
exactly what kinds of information are to be accessed).
Data mining, on the other hand, allows the user to search
out unknown facts, the information hidden behind the
data. It also enables users to pose more complex queries
(Dunham, 2003).
Figure 1 illustrates the idea of integrating data min-
ing algorithms with XML documents to achieve knowl-
edge discovery. For example, after identifying similari-
ties among various XML documents, a mining technique
can analyze links between tags occurring together within
the documents. This may prove useful in the analysis of
e-commerce Web documents recommending personal-
ization of Web pages.
MINING?
XML mining includes mining of structures as well as
contents from XML documents, depicted in Figure 2
(Nayak et al., 2002). Element tags and their nesting therein
dictate the structure of an XML document (Abiteboul et
al., 2000). For example, the textual structure enclosed by
<author> </author> is used to describe the author tuple
and its corresponding text in the document. Since XML
provides a mechanism for tagging names with data, knowl-
edge discovery on the semantics of the documents be-
comes easier for improving document retrieval on the
Web. Mining of XML structure is essentially mining of schema
including intrastructure mining, and interstructure mining.
Intrastructure Mining
Concerned with the structure within an XML document.
Knowledge is discovered about the internal structure of
XML documents in this type of mining. The following
mining tasks can be applied.
The classification task of data mining maps a new
XML document to a predefined class of documents. A
schema is interpreted as a description of a class of XML
documents. The classification procedure takes a collec-
tion of schemas as a training set and classifies new XML
documents according to this training set.

Figure 1. XML mining scheme Figure 2. A taxonomy of XML mining

XML
Mining

XML XML
Structure Content
Mining Mining

Intra- Inter-
Content Structure
Structure Structure Analysis
Mining Mining Clarification

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
Discovering Knowledge from XML Documents
The clustering task of data mining identifies similari-
ties among various XML documents. A clustering algo-
rithm takes a collection of schemas to group them together
on the basis of self-similarity. These similarities are then
used to generate new schema. As a generalization, the
new schema is a superclass to the training set of schemas.
This generated set of clustered schemas can now be used
in classifying new schemas. The superclass schema also
can be used in integration of heterogeneous XML docu-
ments for each application domain. This allows users to
find, collect, filter, and manage information sources more
effectively on the Internet.
The association data mining describes relationships
between tags that tend to occur together in XML docu-
ments that can be useful in the future. By transforming
the tree structure of XML into a pseudo-transaction, it
becomes possible to generate rules of the form if an
XML document contains a <craft> tag, then 80% of the
time it also will contain a <licence> tag. Such a rule
then may be applied in determining the appropriate
interpretation for homographic tags.
Interstructure Mining
Concerned with the structure between XML documents.
Knowledge is discovered about the relationship be-
tween subjects, organizations, and nodes on the Web in
this type of mining. The following mining tasks can be
applied.
Clustering schemas involves identifying similar
schemas. The clusters are used in defining hierarchies
of schemas. The schema hierarchy overlaps instances
on the Web, thus discovering authorities and hubs
(Garofalakis et al. 1999). Creators of schema are iden-
tified as authorities, and creators of instances are hubs.
Additional mining techniques are required to identify
all instances of schema present on the Web. The follow-
ing application of classification can identify the most
likely places to mine for instances. Classification is
applied with namespaces and URIs (Uniform Resource
Identifiers). Having previously associated a set of
schemas with a particular namespace or URI, this infor-
mation is used to classify new XML documents origi-
nating from these places.
Content is the text between each start and end tag in
XML documents. Mining for XML content is essen-
tially mining for values (an instance of a relation),
including content analysis and structural clarification.
Content Analysis
Concerned with analysing texts within XML documents.
The following mining tasks can be applied to contents.
Classification is performed on XML content, labeling
new XML content as belonging to a predefined class. To ,
reduce the number of comparisons, pre-existing schemas
classify the new documents schema. Then, only the
instance classifications of the matching schemas need to
be considered in classifying a new document.
Clustering on XML content identifies the potential
for new classifications. Again, consideration of schemas
leads to quicker clustering; similar schemas are likely
to have a number of value sets. For example, all schemas
concerning vehicles have a set of values representing
cars, another set representing boats, and so forth. How-
ever, schemas that appear dissimilar may have similar
content. Mining XML content inherits some problems
faced in text mining and analysis. Synonymy and polysemy
can cause difficulties, but the tags surrounding the
content usually can help resolve ambiguities.
Structural Clarification
Concerned with distinguishing the similar structured
documents based on contents. The following mining
tasks can be performed.
Content provides support for alternate clustering of
similar schemas. Two distinctly structured schemas
may have document instances with identical content.
Mining these avails new knowledge. Vice versa, schemas
provide support for alternate clustering of content. Two
XML documents with distinct content may be clustered
together, given that their schemas are similar.
Content also may prove important in clustering
schemas that appear different but have instances with
similar content. Due to heterogeneity, the incidence of
synonyms is increased. Are separate schemas actually
describing the same thing, only with different terms?
While thesauruses are vital, it is impossible for them to
be exhaustive for the English language, let alone handle
all languages. Conversely, schemas appearing similar
actually are completely different, given homographs.
The similarity of the content does not distinguish the
semantic intention of the tags. Mining, in this case,
provides probabilities of a tag having a particular mean-
ing or a relationship between meaning and a URI.
METHODS OF XML STRUCTURE
MINING
Mining of structures from a well-formed or valid docu-
ment is straightforward, since a valid document has a
schema mechanism that defines the syntax and structure
of the document. However, since the presence of schema
is not mandatory for a well-formed XML document, the
373
TEAM LinG

document may not always have an accompanying schema.
To describe the semantic structure of the documents,
schema extraction tools are needed to generate schema for
the given well-formed XML documents.
DTD Generator (Kay, 2000) generates the DTD for a
given XML document. However, the DTD generator yields
a distinct DTD for every XML document; hence, a set of
DTDs is defined for a collection of XML documents rather
than an overall DTD. Thus, the application of data mining
operations will be difficult in this matter. Tools such as
XTRACT (Garofalakis, 2000) and DTD-Miner (Moh et al.,
2000) infer an accurate and semantically meaningful DTD
schema for a given collection of XML documents. How-
ever, these tools depend critically on being given a rela-
tively homogeneous collection of XML documents. In
such heterogeneous and flexible environment as the Web,
it is not reasonable to assume that XML documents related
to the same topic have the same document structure.
Due to a number of limitations using DTDs as an
internal structure, such as limited set of data types, loose
structure constraints, and limitation of content to tex-
tual, many researchers propose the extraction of XML
schema as an extension to XML DTDs (Feng et al., 2002;
Vianu, 2001). In Chidlovskii (2002), a novel XML schema
extraction algorithm is proposed, based on the Extended
Context-Free Grammars (ECFG) with a range of regular
expressions. Feng et al. (2002) also presented a seman-
tic network-based design to convey the semantics car-
ried by the XML hierarchical data structures of XML
documents and to transform the model into an XML
schema. However, both of these proposed algorithms are
very complex.
Mining of structures from ill-formed XML docu-
ments (that lack any fixed and rigid structure) are per-
formed by applying the structure extraction approaches
developed for semi-structured documents. But not all of
these techniques can effectively support the structure
extraction from XML documents that is required for
further application of data mining algorithms. For in-
stance, the NoDoSe tool (Adelberg & Denny, 1999)
determines the structure of a semi-structured document
and then extracts the data. This system is based primarily
on plain text and HTML files, and it does not support
XML. Moreover, in Green, et al. (2002), the proposed
extraction algorithm considers both structure and con-
tents in semi-structured documents, but the purpose is to
query and build an index. They are difficult to use without
some alteration and adaptation for the application of data
mining algorithms.
An alternative method is to approach the document as
the Object Exchange Model (OEM) (Nestorov et al.
1999; Wang et al. 2000) data by using the corresponding
374
Discovering Knowledge from XML Documents
data graph to produce the most specific data guide
(Nayak et al., 2002). The data graph represents the inter-
actions between the objects in a given data domain.
When extracting a schema from a data graph, the goal is
to produce the most specific schema graph from the
original graph. This way of extracting schema is more
general than using the schema for a guide, because most
of the XML documents do not have a schema, and some-
times, if they have a schema, they do not conform to it.
METHODS OF XML CONTENT
MINING
Before knowledge discovery in XML documents oc-
curs, it is necessary to query XML tags and content to
prepare the XML material for mining. An SQL-based
query can extract data from XML documents. There are
a number of query languages, some specifically de-
signed for XML and some for semi-structured data, in
general. Semi-structured data can be described by the
grammar of SSD (semi-structured data) expressions.
The translation of XML to SSD expression is easily
automated (Abiteboul et al., 2000). Query languages
for semi-structured data exploit path expressions. In
this way, data can be queried to an arbitrary depth. Path
expressions are elementary queries with the results
returned as a set of nodes. However, the ability to
return results as semi-structured data is required, which
path expressions alone cannot do. Combining path ex-
pressions with SQL-style syntax provides greater flex-
ibility in testing for equality, performing joins, and
specifying the form of query results. Two such lan-
guages are Lorel (Abiteboul et al., 2000) and Unstruc-
tured Query Language (UnQL) (Farnandez et al., 2000).
UnQL requires more precision and is more reliant on
path expressions.
XML-QL, XML-GL, XSL, and Xquery are designed
specifically for querying XML (W3c, 2004). XML-QL
(Garofalsaki et al., 1999) and Xquery bring together
regular path expressions, SQL-style query techniques,
and XML syntax. The great benefit is the construction
of the result in XML and, thus, transforming XML data
from one schema to another. Extensible Stylesheet
Language (XSL) is not implemented as a query lan-
guage but is intended as a tool for transforming XML to
HTML. However, XSL_S select pattern is a mecha-
nism for information retrieval and, as such, is akin to a
query (W3c, 2004). XML-GL (Ceri et al., 1999) is a
graphical language for querying and restructuring XML
documents.
TEAM LinG
Discovering Knowledge from XML Documents
FUTURE TRENDS
There has been extensive effort to devise new technolo-
gies to process and integrate XML documents, but a lot
of open possibilities still exist. For example, integration of
data mining, XML data models and database languages
will increase the functionality of relational database prod-
ucts, data warehouses, and XML products. Also, to
satisfy the range of data mining users (from naive to expert
users), future work should include mining user graphs
that are structural information of Web usages, as well as
visualization of mined data. As data mining is applied to
large semantic documents or XML documents, extraction
of information should consider rights management of
shared data. XML mining should have the authorization
level to empower security to restrict only appropriate
users to discover classified information.
CONCLUSION
XML has proved effective in the process of transmitting
and sharing data over the Internet. Companies want to
bring this advantage into analytical data, as well. As
XML material becomes more abundant, the ability to
gain knowledge from XML sources decreases due to
their heterogeneity and structural irregularity; the idea
behind the XML data mining looks like a solution to put
to work. Using XML data in the mining process has
become possible through new Web-based technologies
that have been developed. Simple Object Access Proto-
col (SOAP) is a new technology that has enabled XML
to be used in data mining. For example, vTag Web
Mining Server aims at monitoring and mining of the
Web with the use of information agents accessed by
SOAP (vtag, 2003). Similarly, XML for Analysis de-
fines a communication structure for an application pro-
gramming interface, which aims at keeping client pro-
gramming independent from the mechanics of data trans-
port but, at the same time, providing adequate informa-
tion concerning the data and ensuring that it is properly
handled (XMLanalysis, 2003). Another development,
YALE, is an environment for machine learning experi-
ments that uses XML files to describe data mining
experiments setup (Yale, 2004). The Data Miners AR-
CADE also uses XML as the target language for all data
mining tools within its environment (Arcade, 2004).
REFERENCES
Abiteboul, S., Buneman, P., & Suciu, D. (2000). Data on
the Web: From relations to semistructured data and
XML. San Francisco, CA: Morgan Kaumann.
Adelberg, B., & Denny, M. (1999). Nodose version 2.0.
Proceedings of the ACM SIGMOD Conference on ,
Management of Data, Seattle, Washington.
Arcade. (2004). http://datamining.csiro.au/arcade.html
Ceri, S. et al. (1999). XMLGl: A graphical language for
querying and restructuring XML documents. Proceed-
ings of the 8th International WWW Conference, Toronto,
Canada.
Chidlovskii, B. (2002). Schema extraction from XML col-
lections. Proceedings of the 2nd ACM/IEEE-CS Joint
Conference on Digital Libraries, Portland, Oregon.
Dunham, M.H. (2003). Data mining: Introductory and
advanced topics. Upper Saddle River, NJ: Prentice Hall
Farnandez, M., Buneman, P., & Suciu, D. (2000). UNQL: A
query language and algebra for semistructured data based
on structural recursion. VLDB JOURNAL: Very Large
Data Bases, 9(1), 76-110.
Feng, L., Chang, E., & Dillon, T. (2002). A semantic
network-based design methodology for XML documents.
ACM Transactions of Information Systems (TOIS),
20(4), 390-421.
Garofalakis, M. et al. (1999). Data mining and the Web:
Past, present and future. Proceedings of the Second
International Workshop on Web Information and Data
Management, Kansas City, Missouri.
Garofalakis, M.N. et al. (2000). XTRACT: A system for
extracting document type descriptors from XML docu-
ments. Proceedings of the 2000 ACM SIGMOD Inter-
national Conference on Management of Data, Dallas,
Texas.
Green, R., Bean, C.A., & Myaeng, S.H. (2002). The seman-
tics of relationships: An interdisciplinary perspective.
Boston: Kluwer Academic Publishers.
Kay, M. (2000). SAXON DTD generatorA tool to gen-
erate XML DTDs. Retrieved January 2, 2003, from http:/
/home.iclweb.com/ic2/mhkay/dtdgen.html
Moh, C.-H., & Lim, E.-P. (2000). DTD-miner: A tool for
mining DTD from XML documents. Proceedings of the
Second International Workshop on Advanced Issues of
E-Commerce and Web-Based Information Systems, Cali-
fornia.
Nayak, R., Witt, R., & Tonev, A. (2002, June). Data mining
and XML documents. Proceedings of the 2002 Interna-
tional Conference on Internet Computing, Nevada.
Nestorov, S. et al. (1999). Representative objects: Concise
representation of semi-structured, hierarchical data. Pro-
375
TEAM LinG

ceedings of the IEEE Proc on Management of Data,
Seattle, Washington.
Vianu, V. (2001). A Web odyssey: from Codd to XML.
Proceedings of the 20 th ACM SIGMOD-SIGACT-
SIGART Symposium on Principles of Database Systems,
California.
Vtag. (2003). http://www.connotate.com/csp.asp
W3c. (2004). XML query (Xquery). Retrieved March 18,
2004, from http://www.w3c.org/XML/Query
Wang, Q., Yu, X.J., & Wong, K. (2000). Approximate graph
scheme extraction for semi-structured data. Proceedings
of the 7th International Conference on Extending Data-
base Technology, Konstanz.
XMLanalysis. (2003). http://www.intelligenteai.com/fea-
ture/011004/editpage.shtml
Yale. (2004). http://yale.cs.uni-dortmund.de/
KEY TERMS
Ill-Formed XML Documents: Lack any fixed and
rigid structure.
Valid XML Document: To be valid, an XML docu-
ment additionally must conform (at least) to an explic-
itly associated document schema definition.
376
Discovering Knowledge from XML Documents
Well-Formed XML Documents: To be well-formed, a
pages XML must have properly nested tags, unique
attributes (per element), one or more elements, exactly one
root element, and a number of schema-related constraints.
Well-formed documents may have a schema, but they do
not conform to it.
XML Content Analysis Mining: Concerned with
analysing texts within XML documents.
XML Interstructure Mining: Concerned with the
structure between XML documents. Knowledge is dis-
covered about the relationship among subjects, organi-
zations, and nodes on the Web.
XML Intrastructure Mining: Concerned with the
structure within an XML document(s). Knowledge is
discovered about the internal structure of XML docu-
ments.
XML Mining: Knowledge discovery from XML
documents (heterogeneous and structural irregular).
For example, clustering data mining techniques can
group a collection of XML documents together accord-
ing to the similarity of their structures. Classification
data mining techniques can classify a number of hetero-
geneous XML documents into a set of predefined clas-
sification of schemas to improve XML document han-
dling and achieve efficient searches.
XML Structural Clarification Mining: Concerned
with distinguishing the similar structured documents
based on contents.
TEAM LinG

Discovering Ranking Functions for
Information Retrieval
Weiguo Fan
Virginia Polytechnic Institute and State University, USA
Praveen Pathak
University of Florida, USA
INTRODUCTION
The field of information retrieval deals with finding
relevant documents from a large document collection
or the World Wide Web in response to a users query
seeking relevant information. Ranking functions play a
very important role in the retrieval performance of such
retrieval systems and search engines. A single ranking
function does not perform well across different user
queries, and document collections. Hence it is neces-
sary to discover a ranking function for a particular
context. Adaptive algorithms like genetic programming
(GP) are well suited for such discovery.
BACKGROUND
In an information retrieval system (IR), given a user
query a matching function matches the information in
the query with that in the documents to rank the docu-
ments in decreasing order of their predicted relevance
to the user. The top ranked documents are then pre-
sented to the user as a response to her query. To facili-
tate this relevance estimation process both the docu-
ments and the queries need to be transformed in a form
that can be easily processed by computers. Vector Space
Model (VSM) (Salton, 1989) is one of the most suc-
cessful models to represent the documents and queries.
We choose this model as the underlying model in our
research due to the ease of interpretation of the model
and the tremendous success in retrieval performance
studies. Moreover most existing search engines and IR
systems are based on this model.
In VSM, both documents and queries are represented
as vectors of terms. Suppose there are t terms in the
collection, then a document D and a query Q are repre-
sented as:
D = (wd1, wd2, , wdt)
Q = (wq1, wq2, , wqt)
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
377
,
where wdi, wqi (for i=1 to t) are weights assigned to different
terms in the document and the query respectively. The
similarity between the two vectors is calculated as the
cosine of the angle between the two vectors. It is ex-
pressed as (Salton & Buckley, 1988):
t
w qi wdi
Similarity (Q, D) = i =1
t t
(1)
(w
i =1
qi ) 2 * ( wdi ) 2
i =1
The score, called retrieval status value (RSV), is
calculated for each document in the collection and the
documents are ordered and presented to the user in the
decreasing order of RSV. Various content based fea-
tures are available in VSM to compute the term weights.
The most common ones are the term frequency (tf) and
the inverse document frequency (idf). Term frequency
measures the number of times a term appears in the
document or the query. The higher this number, the
more important the term is assumed to be in describing
the document. Inverse document frequency is calcu-
lated as log( N / DF ) , where N is the total number of
documents in the collection, and DF is the number of
documents in which the term appears. A high value of idf
means the term appears in a relatively few number of
documents and hence the term is assumed to be impor-
tant in describing the document. A lot of similar content
based features are available in literature (Salton, 1989;
Salton & Buckley, 1988). The features can be combined
(e.g. tf*idf) to generate a variety of new composite
features that can be used in term weighting.
Equation (1) suggests that in order to discover a good
ranking function, we need to discover the optimal way of
assigning weights to document and query keywords. Change
in term weighting strategy will essentially change the behav-
ior of a ranking function. In this chapter we describe a method
to discover an optimal ranking function.
TEAM LinG

Figure 1. A sample tree representation for a ranking
function
+ is given in Figure 1. We will use such a tree based
- lo g
tf * n df
df N Finally, it has been empirically found that GP discovers
MAIN THRUST
In this chapter we present a systematic and automatic
discovery process to discover ranking functions. The
process is based on an artificial intelligence technique
called genetic programming (GP). GP is based on ge-
netic algorithms (GA) (Goldberg, 1989; Holland, 1992).
Because of the intrinsic parallel search mechanism and
powerful global exploration capability in high-dimen-
sional space, both GA and GP have been used to solve a
wide range of hard optimization problems. They are
used in various optimal design and data-mining applica-
tions (Koza, 1992).
GP represents the solution to a problem as a chro-
mosome (or an individual) in a population pool. It
evolves the population of chromosomes in successive
generations by following the genetic transformation
operations such as reproduction, crossover, and muta-
tion to discover chromosomes with better fitness val-
ues. A fitness function assigns a fitness value for each
chromosome that represents how good the chromosome
is at solving the problem at hand.
We use GP for discovering ranking functions be-
cause of four reasons. First, in GP there is no stringent
requirement for an objective function to be continuous.
All that is needed is that the objective function should be
able to differentiate good solutions from the bad ones.
This property allows us to use common IR performance
measures, like average precision (P_Avg), which are
non-linear in nature as objective functions. Second, GP
is well suited to represent the common tree based
Table 1. Essential GP components
GP Parameters
Terminals
Functions
Fitness Function
Reproduction and Crossover
378
Discovering Ranking Functions for Information Retrieval
representations for the solutions. A tree-based represen-
tation allows for easier parsing and implementation. An
example of a term weighting formula using tree structure
representation in this chapter. Third, GP is very effective
for non-linear function and structure discovery problems
where traditional optimization methods do not seem to
work well (Banzhaf, Nordin, Keller, & Francone, 1998).
better solutions than those obtained by conventional
heuristic algorithms.
Ranking function discovery as presented in this chap-
ter is different from classification in that we seek to
find a function that will be used for ranking or prioritiz-
ing documents. There are efforts in IR that treat this as
a classification problem in which a classifier or dis-
criminant function is used for ranking. But evidence
shows that ranking function discovery has yielded better
retrieval results than the results obtained by treating this
as a classification problem using Support Vector Ma-
chines and Neural Networks (Fan, Gordon, Pathak,
Wensi, & Fox, 2004; Fuhr & Pfeifer, 1994). We now
proceed to describe the discovery process using GP.
Ranking Function Discovery by GP
In order to apply GP in our context we need to define
several components for it. We use the tree structure as
shown in Figure 1 to represent term weighting formula.
Components needed for such a representation are given
in Table 1.
For the purpose of our discovery framework we will
define these parameters as follows:
An individual in the population is expressed in
term of a tree which represents one possible rank-
ing function. A population in a generation con-
sists of P such trees.
Terminals: We use the features mentioned in
Table 2 and real constants as the terminals.
Functions: We use +, -, *, /, and log as the
functions allowed
Meaning
Leaf nodes in the tree data structure
Non-leaf nodes used to combine the leaf nodes. Typically
numerical operations
The objective function that needs to be optimized
Genetic operators used to copy fit solutions from one generation
to another and to introduce diversity in the population
TEAM LinG
Discovering Ranking Functions for Information Retrieval

Fitness Function: We use the P_Avg as the fitness
function which is defined in equation
i and a validation set. This two dataset training method-
D r (d j )
r (d ) * j =1
i =1
i
i
(2)
P _ Avg =
T Re l
where r(di) (0,1) is the relevance score assigned to
a document, it being 1 if the document is relevant and
0 otherwise. |D| is the total number of retrieved
documents. TRel is the total number of relevant
documents for the query. This equation incorpo-
rates both the standard retrieval measures of preci-
sion and recall. It also takes into account the order-
ing of the relevant retrieved documents. For ex-
ample, if there are 20 documents retrieved and only
5 of them are relevant then P_Avg score is higher if
the relevant documents are higher up in the order
(say top 5 retrieved documents are relevant) as
compared to the P_Avg score with relevant docu-
ments that are lower in the retrieval order (say the
16th to 20th document). This property is very im-
portant in many retrieval scenarios where the user
is willing to see only the top few retrieved docu-
ments. P_Avg is the most widely used measure in
retrieval studies for comparing performance of
different systems.
Reproduction: Reproduction copies the top (in
terms of fitness) trees in the population into the next
population. If P is the population size and reproduc-
tion rate is rate_r then top rate_r * P trees are
copied into next generation. rate_r is set to 0.1.
Crossover: We use tournament selection to se-
lect, with replacement, 6 random trees from the
population. The top two among the six trees (in
terms of fitness) are selected for crossover and
they exchange sub-trees to form trees for the next
generation.
The detailed process for the ranking function discov-
ery is as given in Figure 2. It is an iterative process. First ,
a population of random ranking functions is created. The
training documents are divided into a set of training set
ology has been commonly used in machine learning
experiments (Mitchell, 1997). Relevance judgments for a
query for each of the documents in the training and
validation set are known. The random ranking functions
are evaluated using relevance information for the train-
ing set. The performance measure used is given in Equa-
tion (2). The topmost ranking function in terms of perfor-
mance is noted. The population is subjected to the
genetic operations of selection, reproduction, and cross-
over to generate the next generation. The process is
repeated for each generation. At the end of 30 genera-
tions the thirty ranking functions are applied to the
validation set and the best performing ranking function
is chosen as the discovered ranking function for the
particular query.
It is to be noted that over the 30 generations the best
ranking function in each generation does successively
improve retrieval performance on the training data set.
This improvement in retrieval performance is as ex-
pected. However, to avoid overfitting problems that are
very common in these techniques, we apply the best
ranking function from each of the 30 generations to the
unseen validation data set and choose the best perform-
ing ranking function on the validation data set to be
applied to the test data set. Thus the final ranking
function that we choose for applying to the test data set
need not necessarily come from the last generation.
Experiments and Results
We have applied the discovery process described above
on various TREC (Harman, 1996; Hawking, 2000; Hawk-
ing & Craswell, 2001) document collections. The TREC
datasets were divided into training, validation, and test
datasets. The discovered ranking function after the train-
ing and validation phase was applied to the test dataset.
The performance results of our method were compared

Table 2. Features used in the discovery process

Features Used Statistical Meaning
Tf Number of times a term appears in a document
tf_max Maximum tf in the entire document collection
tf_avg Average tf for a document
Df Number of documents in the collection the term appeared in
df_max Maximum df across all terms
N Number of documents in the entire text collection
Length Length of a document
Length_avg Average length of document in the entire collection
N Number of unique terms in the document

379

TEAM LinG
 Discovering Ranking Functions for Information Retrieval
Figure 2. Discovery process
IN P U T : Q u ery, T raining d ocu m en ts
O U TP U T: D isco vered ra nkin g fu nctio n
P rocess:
T raining docum ents divided into a training set and a
validation set
Initial population of random ranking functions
generated
A pply the follow ing on training dataset for 30
generations
Use the individual ranking function in the generation to rank
documents in the training set
Create a new population by applying genetic operators of
reproduction and crossover on the existing population
T he top perform ing individual from each generation is
applied to the docum ents in the validation set and the
best perform ing ranking function is selected
T his is the D iscovered ranking function
with the results obtained by applying well known ranking
functions in the literature (OKAPI, Pivoted TFIDF, and
INQUERY) (Singhal, Salton, Mitra, & Buckley, 1996) to the
same test dataset. These well known functions essentially
differ based on their term weighting strategies i.e. the way
they combine various weighting features shown in Table
2. We have found that retrieval performance has signifi-
cantly increased by using our method of discovering
ranking functions. The performance improvement, in terms
of P_Avg, has been anywhere from 11% to 75%, with the
results being statistically significant. More details about
the performance comparisons are available elsewhere
(Fan, Gordon, & Pathak, 2004a, 2004b).
FUTURE TRENDS
We believe the ranking functions discovery as outlined
in this chapter is just the beginning of a promising
avenue of research. This discovery process can be ap-
plied to either the routing search task (search specific
to a particular query) or the ad-hoc search task (a gen-
eralized search for a set of queries). Thus the process
could be made more personalized for an individual user
with persistent information requirements or to a group of
users (for example in a department in an organization)
with similar, but not same, information requirements.
Another promising avenue of research can deal with
intelligently leveraging both the structural as well as
statistical information available in documents, specifi-
cally HTML and XML documents. The work presented
here utilizes just the statistical information in the docu-
ments in terms of frequencies of terms in documents or
across documents. But structural information such as
where the terms appear e.g. in the title, anchor, body, or
abstract of the document can also be leveraged using the
discovery process described in the chapter. Such leveraging,
380
we believe, will yield even better retrieval performance.
Finally, we believe the technique mentioned in this
chapter can be combined with data-fusion techniques to
combine successful traits from various algorithms to
yield better retrieval performance.
CONCLUSION
In this chapter we have presented a ranking function
discovery process to discover new ranking functions.
Ranking functions match the information in documents
with that in the queries to rank the documents in the
decreasing order of predicted relevance of the docu-
ments to the user. Although there are well known rank-
ing functions in the IR literature we believe even better
ranking functions can be discovered for specific queries
or a set of queries. The discovery of ranking functions
was accomplished using Genetic Programming, an arti-
ficial intelligence algorithm based on evolutionary
theory. The results of GP based retrieval have been
found to significantly outperform the results obtained
by well known ranking functions. We believe this line of
research will be potentially very rewarding in terms of
much improved retrieval performance.
REFERENCES
Banzhaf, W., Nordin, P., Keller, R., & Francone, F.
(1998). Genetic programming: An introduction - On
the automatic evolution of computer programs and its
applications. San Francisco, CA: Morgan Kaufmann
Publishers.
Fan, W., Gordon, M., & Pathak, P. (2004a). Discovery of
context-specific ranking functions for effective informa-
tion retrieval using genetic programming. IEEE Transac-
tions on Knowledge and Data Engineering, 16(4), 523-
527.
Fan, W., Gordon, M., & Pathak, P. (2004b). A generic
ranking function discovery framework by genetic pro-
gramming for information retrieval. Information Pro-
cessing and Management, 40(4), 587-602.
Fan, W., Gordon, M. D., Pathak, P., Wensi, X., & Fox, E.
(2004). Ranking function optimization for effective
web search by genetic programming: an empirical study,
Proceedings of 37th Hawaii International Confer-
ence on System Sciences, Big Island, Hawaii. IEEE.
Fuhr, N., & Pfeifer, U. (1994). Probabilistic information
retrieval as combination of abstraction inductive learning
and probabilistic assumptions. ACM Transactions on
Information Systems, 12, 92-115.
TEAM LinG
Discovering Ranking Functions for Information Retrieval
Goldberg, D. E. (1989). Genetic algorithms in search,
optimization and machine learning. Addison-Wesley.
Harman, D. K. (1996). Overview of the fourth text retrieval
conference (TREC-4). In D. K. Harman (Ed.), Proceedings
of the Fourth Text Retrieval Conference (Vol. 500-236, pp.
1-24). NIST Special Publication.
Hawking, D. (2000). Overview of the TREC-9 Web track.
In E. Voorhees & H. D. (Eds.), Ninth Text Retrieval
Conference (Vol. 500-249, pp. 86-102). NIST Special Pub-
lication.
Hawking, D., & Craswell, N. (2001). Overview of the TREC-
2001 Web track. In E. Voorhees & D. K. Harman (Eds.),
Proceedings of the Tenth Text Retrieval Conference (Vol.
500-250, pp. 61-67). NIST.
Holland, J. H. (1992). Adaptation in natural and artificial
systems (2nd ed.). MIT Press.
Koza, J. R. (1992). Genetic programming: On the pro-
gramming of computers by means of natural selection.
Cambridge, MA: MIT Press.
Mitchell, T. M. (1997). Machine learning. McGraw Hill).
Salton, G. (1989). Automatic text processing. Reading,
MA: Addison-Wesley Publishing Co.
Salton, G., & Buckley, C. (1988). Term weighting ap-
proaches in automatic text retrieval. Information pro-
cessing and management, 24(5), 513-523.
Singhal, A., Salton, G., Mitra, M., & Buckley, C. (1996).
Document length normalization. Information process-
ing and management, 32(5), 619-633.
KEY TERMS
,
Average Precision (P_Avg): This is a well known
measure of retrieval performance. It is the average of the
precision scores calculated every time a new relevant
document is found, normalized by the total number of
relevant documents in the collection.
Document Cut-OFF Value (DCV): The number of
documents that the user is willing to see as a response to
the query.
Document Frequency: The number of documents
in the document collection that the term appears in.
Genetic Programming: A stochastic search algo-
rithm based on evolutionary theory, with the aim to
optimize structure or functional form. A tree structure
is commonly used for representation of solutions.
Precision: The ratio of the number of relevant
documents retrieved to the total number of documents
retrieved.
Ranking Function: A function that matches the
information in documents with that in the user query to
assign a score for each document in the collection.
Recall: The ratio of the number of relevant docu-
ments retrieved to the total number of relevant docu-
ments in the document collection.
Term Frequency: The number of times a term
appears in a document.
Vector Space Model (VSM): A common IR model
where both documents and queries are represented as
vectors of terms.
381
TEAM LinG
382
Discovering Unknown Patterns in Free Text
Jan H. Kroeze
University of Pretoria, South Africa
INTRODUCTION
A very large percentage of business and academic data is
stored in textual format. With the exception of metadata,
such as author, date, title and publisher, these data are
not overtly structured like the standard, mainly numeri-
cal, data in relational databases. Parallel to data mining,
which finds new patterns and trends in numerical data,
text mining is the process aimed at discovering unknown
patterns in free text. Owing to the importance of com-
petitive and scientific knowledge that can be exploited
from these texts, text mining has become an increas-
ingly popular and essential theme in data mining (Han
& Kamber, 2001, p. 428).
Text mining has a relatively short history: Unlike
search engines and data mining that have a longer history
and are better understood, text mining is an emerging
technical area that is relatively unknown to IT profes-
sions (Chen, 2001, p. vi).
BACKGROUND
Definitions of text mining vary a great deal, from views
that it is an advanced form of information retrieval (IR)
to those that regard it as a sibling of data mining:
Text mining is the discovery of texts.
Text mining is the exploration of available texts.
Text mining is the extraction of information from
text.
Text mining is the discovery of new knowledge in
text.
Text mining is the discovery of new patterns,
trends and relations in and among texts.
Han & Kamber (2001, pp. 428-435), for example,
devote much of their rather short discussion of text
mining to information retrieval. However, one should
differentiate between text mining and information re-
trieval. Text mining does not consist of searching through
metadata and full-text databases to find existing infor-
mation. The point of view expressed by Nasukawa &
Nagano (2001, p. 969), to wit that text mining is a text
version of generalized data mining, is correct. Text
mining should focus on finding valuable patterns and
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
rules in text that indicate trends and significant features
about specific topics (Nasukawa & Nagano, 2001, p. 967).
MAIN THRUST
Like data mining, text mining is a proactive process that
automatically searches data for new relationships and
anomalies to serve as a basis for making business deci-
sions aimed at gaining competitive advantage (cf., Rob
& Coronel, 2004, p. 597). Although data mining can
require some interaction between the investigator and
the data-mining tool, it can be considered as an auto-
matic process because data-mining tools automati-
cally search the data for anomalies and possible
relationships, thereby identifying problems that have
not yet been identified by the end user, while mere
data analysis relies on the end users to define the
problem, select the data, and initiate the appropriate
data analyses to generate the information that helps
model and solve problems those end-users uncover
(Rob & Coronel, 2004, p. 597). The same distinction is
valid for text mining. Therefore, text-mining tools should
also initiate analyses to create knowledge (Rob &
Coronel, 2004, p. 598).
In practice, however, the borders between data analy-
sis, information retrieval and text mining are not always
quite so clear. Montes-y-Gmez et al. (2004) proposed
an integrated approach, called contextual exploration,
which combines robust access (IR), non-sequential navi-
gation (hypertext) and content analysis (text mining).
According to Smallheiser (2001, pp. 690-691), text
mining approaches can be divided into two main types:
Macro analyses perform data-crunching operations over
a large, often global set of papers encompassing one or
more fields, in order to identify large-scale trends or to
classify and organize the literature. In contrast, micro
analyses pose a sharply focused question, in which one
searches for complementary information that links two
small, pre-specified fields of inquiry.
The Need for Text Mining
Text mining can be used as an effective business intel-
ligence tool for gaining competitive advantage through
TEAM LinG
Discovering Unknown Patterns in Free Text
the discovery of critical, yet hidden, business informa-
tion. In the field of academic research, text mining can be
used to scan large numbers of publications in order to
select the most relevant literature and to propose new
links between independent research results. Text min-
ing is also needed to formulate and assess hypotheses
arising in biomedical research and for helping
make policy decisions regarding technical innovation
(Smallheiser, 2001, p. 690). Another application of text
mining is in medical science, to discover gene interac-
tions, functions and relations, or to build and structure
medical knowledge bases, and to find undiscovered
relations between diseases and medications (De Bruijn
& Martin, 2002, p. 8).
Types of Text Mining
Keyword-Based Association Analysis
Association analysis looks for correlations between
texts based on the occurrence of related keywords or
phrases. Texts with similar terms are grouped together.
The pre-processing of the texts is very important and
includes parsing and stemming, and the removal of
words with minimal semantic content. Another issue is
the problem of compounds and non-compounds should
the analysis be based on singular words or should word
groups be accounted for? (cf., Han & Kamber, 2001, p.
433). Kostoff et al. (2002), for example, have measured
the frequencies and proximities of phrases regarding
electrochemical power to discover central themes and
relationships among them. This knowledge discovery,
combined with the interpretation of human experts, can
be regarded as an example of knowledge creation through
intelligent text mining.
Automatic Document Classification
Electronic documents are classified according to a pre-
defined scheme or training set. The user compiles and
refines the classification parameters, which are then
used by a computer program to categorise the texts in
the given collection automatically (cf., Sullivan, 2001,
p. 198). Classification can also be based on the analysis
of collocation [the juxtaposition or association of a
particular word with another particular word or words
(The Oxford Dictionary, 1995)]. Words that often
appear together probably belong to the same class (Lopes
et al., 2004). According to Perrin & Petry (2003) use-
ful text structure and content can be systematically
extracted by collocational lexical analysis with statis-
tical methods. Text classification can be used by busi-
nesses, for example, to categorise customers e-mails
automatically and suggest the appropriate reply tem-
plates (Weng & Liu, 2004). ,
Similarity Detection
Texts are grouped according to their own content into
categories that were not previously known. The docu-
ments are analysed by a clustering computer program,
often a neural network, but the clusters still have to be
interpreted by a human expert (Hearst, 1999). Docu-
ment pre-processing (tagging of parts of speech,
lemmatisation, filtering and structuring) precedes the
actual clustering phase (Iiritano et al., 2004). The clus-
tering program finds similarities between documents,
for example, common author, same themes, or informa-
tion from common sources. The program does not need
a training set or taxonomy, but generates it dynamically
(cf., Sullivan, 2001, p. 201). One example of the use of
text clustering is found in the work of Fattori et al.
(2003), whose text-mining tool processes patent docu-
ments into dynamic clusters to discover patenting trends,
which constitutes information that can be used as com-
petitive intelligence.
Link Analysis
Link analysis is the process of building up networks of
interconnected objects through relationships in order
to expose patterns and trends (Westphal & Blaxton,
1998, p. 202). In text databases, link analysis is the finding
of meaningful, high levels of correlations between text
entities. The user can, for example, suggest a broad
hypothesis and then analyse the data in order to prove or
disprove this hunch. It can also be an automatic or semi-
automatic process, in which a surprisingly high number of
links between two or more nodes may indicate relations
that have hitherto been unknown. Link analysis can also
refer to the use of algorithms to build and exploit networks
of hyperlinks in order to find relevant and related docu-
ments on the Web (Davison, 2003). Yoon & Park (2004)
use link analysis to construct a visual network of pat-
ents, which facilitates the identification of a patents
relative importance: The coverage of the application is
wide, ranging from new idea generation to ex post facto
auditing (p. 49). Text mining is also used to identify
experts by finding and evaluating links between persons
and areas of expertise (Ibrahim, 2004).
Sequence Analysis
A sequential pattern is the arrangement of a number of
elements, in which the one leads to the other over time
(Wong et al., 2000). Sequence analysis is the discovery
383
TEAM LinG

of patterns that are related to time frames, for example, the
origin and development of a news thread (cf., Montes-y-
Gmez et al., 2001), or tracking a developing trend in
politics or business. It can also be used to predict recurring
events.
Anomaly Detection
Anomaly detection is the finding of information that
violates the usual patterns, for example, a book that refers
to a unique source, or a document lacking typical informa-
tion. An example of anomaly detection is the detection of
irregularities in news reports or different topic profiles in
newspapers (Montes-y-Gmez et al., 2001).
Hypertext Analysis
Text mining is about looking for patterns in natural
language text. Web mining is the slightly more general
case of looking for patterns in hypertext and often ap-
plies graph theoretical approaches to detect and utilise
the structure of web sites (New Zealand Digital Library,
2002). Marked-up language, especially XML tags, fa-
cilitates text mining because the tags can often be used
to simulate database attributes and to convert data-cen-
tric documents into databases, which can then be ex-
ploited (Tseng & Hwung, 2002). Mark-up tags also make
it possible to create artificial structures [that] help us
understand the relationship between documents and docu-
ment components (Sullivan, 2001, p. 51).
CRITICAL ISSUES
Many sources on text mining refer to text as unstruc-
tured data. However, it is a fallacy that text data are
unstructured. Text is actually highly structured in terms
of morphology, syntax, semantics and pragmatics. On
the other hand, it must be admitted that these structures
are not directly visible: text represents factual infor-
mation in a complex, rich, and opaque manner
Figure 1. A differentiation between information retrieval, standard and intelligent metadata mining, and
standard and intelligent text mining (abbreviated from Kroeze, Matthee, & Bothma, 2003)
Novelty level: Non-novel
investigation -
Data type: information retrieval
Metadata (overtly Information retrieval
structured) of metadata
Free text (covertly Information retrieval
structured) of full texts
384
Discovering Unknown Patterns in Free Text
(Nasukawa & Nagano, 2001, p. 967).
Authors also differ on the issue of natural language
processing within text mining. Some prefer a more statis-
tical approach (cf., Hearst, 1999), while others feel that
linguistic parsing is an essential part of text mining.
Sullivan (2001, p. 37) regards the representation of mean-
ing by means of syntactic-semantic representations as
essential for text mining: Text processing techniques,
based on morphology, syntax, and semantics, are pow-
erful mechanisms for extracting business intelligence
information from documents. We can scan text for
meaningful phrase patterns and extract key features and
relationships. According to De Bruijn & Martin (2002,
p. 16), [l]arge-scale statistical methods will continue to
challenge the position of the more syntax-semantics
oriented approaches, although both will hold their own
place.
In the light of the various definitions of text mining,
it should come as no surprise that authors also differ on
what qualifies as text mining and what does not. Build-
ing on Hearst (1999), Kroeze, Matthee, & Bothma
(2003) use the parameters of novelty and data type to
distinguish between information retrieval, standard text
mining and intelligent text mining (see Figure 1).
Halliman (2001, p. 7) also hints at a scale of new-
ness of information: Some text mining discussions
stress the importance of discovering new knowledge.
And the new knowledge is expected to be new to every-
body. From a practical point of view, we believe that
business text should be mined for information that is
new enough to give a company a competitive edge
once the information is analyzed.
Another issue is the question of when text mining
can be regarded as intelligent. Intelligent behavior is
the ability to learn from experience and apply knowl-
edge acquired from experience, handle complex situa-
tions, solve problems when important information is
missing, determine what is important, react quickly and
correctly to a new situation, understand visual images,
process and manipulate symbols, be creative and imagi-
native, and use heuristics (Stair & Reynolds, 2001, p.
Semi-novel Novel investigation -
investigation - knowledge creation
knowledge discovery
Standard metadata Intelligent metadata
mining mining
Standard text mining Intelligent text mining
TEAM LinG
Discovering Unknown Patterns in Free Text
421). Intelligent text mining should therefore refer to the
interpretation and evaluation of discovered patterns.
FUTURE TRENDS
Mack and Hehenberger (2002, p. S97) regards the automa-
tion of human-like capabilities for comprehending com-
plicated knowledge structures as one of the frontiers of
text-based knowledge discovery. Incorporating more
artificial intelligence abilities into text-mining tools will
facilitate the transition from mainly statistical procedures
to more intelligent forms of text mining.
CONCLUSION
Text mining can be regarded as the next frontier in the
science of knowledge discovery and creation, enabling
businesses to acquire sought-after competitive intelli-
gence, and helping scientists of all academic disciplines
to formulate and test new hypotheses. The greatest
challenges will be to select the most appropriate tech-
nology for specific problems and to popularise these
new technologies so that they become instruments that
are generally known, accepted and widely used.
REFERENCES
Chen, H. (2001). Knowledge management systems: A
text mining perspective. Tucson, AZ: University of
Arizona (Knowledge Computing Corporation).
Davison, B.D. (2003). Unifying text and link analysis. In
Text-Mining & Link-Analysis Workshop of the 18 th
International Joint Conference on Artificial Intelli-
gence. Retrieved from http://www-2.cs.cmu.edu/
~dunja/TextLink2003/
De Bruijn, B., & Martin, J. (2002). Getting to the (c)ore
of knowledge: Mining biomedical literature. Interna-
tional Journal of Medical Informatics, 67(1-3), 7-18.
Fattori, M., Pedrazzi, G., & Turra, R. (2003). Text
mining applied to patent mapping: A practical business
case. World Patent Information, 25(4), 335-342.
Halliman, C. (2001). Business intelligence using smart
techniques: Environmental scanning using text min-
ing and competitor analysis using scenarios and
manual simulation. Houston, TX: Information Uncover.
Han, J., & Kamber, M. (2001). Data mining: Concepts
and techniques. San Francisco, CA: Morgan Kaufmann.
Hearst, M.A. (1999). Untangling text data mining. In
Proceedings of ACL99: the 37th Annual Meeting of the ,
Association for Computational Linguistics. University
of Maryland, June 20-26 (invited paper). Retrieved from
http://www.ai.mit.edu/people/jimmylin/papers/
Hearst99a.pdf
Ibrahim, A. (2004). Expertise location: Can text mining
help? In N.F.F. Ebecken, C.A. Brebbia, & A. Zanas (Eds.),
Data mining IV (pp. 109-118). Southampton: WIT Press.
Iiritano, S., Ruffolo, M., & Rullo, P. (2004). Preprocessing
method and similarity measures in clustering-based text
mining: A preliminary study. In N.F.F. Ebecken, C.A.
Brebbia, & A. Zanas (Eds.), Data mining IV (pp. 73-79).
Southampton: WIT Press.
Kostoff, R.N., Tshiteya, R., Pfeil, K.M., & Humenik,
J.A. (2002). Electrochemical power text mining using
bibliometrics and database tomography. Journal of
Power Sources, 110(1), 163-176.
Kroeze, J.H., Matthee, M.C., & Bothma, T.J.D. (2003).
Differentiating data- and text-mining terminology. In IT
Research in Developing Countries - Proceedings of
SAICSIT 2003 (Annual Research Conference of the
South African Institute of Computer Scientists and
Information Technologists) (pp. 93-101), September 17-
19. Pretoria. SAICSIT.
Lopes, M.C.S., Terra, G.S., Ebecken, N.F.F., & Cunha,
G.G. (2004). Mining text databases on clients opinion
for oil industry. In N.F.F. Ebecken, C.A. Brebbia, & A.
Zanas (eds.), Data Mining IV (pp. 139-147).
Southampton: WIT Press.
Mack, R., & Hehenberger, M. (2002). Text-based knowl-
edge discovery: Search and mining of life-science docu-
ments. Drug Discovery Today, 7(11) (Suppl.), S89-S98.
Montes-y-Gmez, M., Gelbukh, A., & Lpez-Lpez, A.
(2001). Mining the news: Trends, associations, and devia-
tions. Computacin y Sistemas, 5(1). Retrieved from http:/
/ccc.inaoep.mx/~mmontesg/publicaciones/2001/
NewsMining-CyS01.pdf
Montes-y-Gmez, M., Prez-Coutio, M., Villaseor-
Pineda, L., & Lpez-Lpez, A. (2004). Contextual ex-
ploration of text collections. Lecture Notes in Computer
Science (Vol. 2945) Berlin: Springer-Verlag. Retrieved
from http://ccc.inaoep.mx/~mmontesg/publicaciones/
2004/ContextualExploration-CICLing04.pdf
Nasukawa, T., & Nagano, T. (2001). Text analysis and
knowledge mining system. IBM Systems Journal, 40(4),
967-984.
385
TEAM LinG

New Zealand Digital Library, University of Waikato. (2002).
Text mining. Retrieved from http://www.cs.waikato.ac.nz/
~nzdl/textmining/
Perrin, P., & Petry, F.E. (2003). Extraction and repre-
sentation of contextual information for knowledge dis-
covery in texts. Information Sciences, 151, 125-152.
Rob, P., & Coronel, C. (2004). Database systems: Design,
implementation, and management (6th ed.). Boston, MA:
Course Technology.
Smallheiser, N.R. (2001). Predicting emerging tech-
nologies with the aid of text-based data mining: The
micro approach. Technovation, 21(10), 689-693.
Stair, R.M., & Reynolds, G.W. (2001). Principles of
information systems: A managerial approach (5th ed.).
Boston, MA: Course Technology.
Sullivan, D. (2001). Document warehousing and text
mining: Techniques for improving business opera-
tions, marketing, and sales. New York, NY: John Wiley.
Tseng, F.S.C., & Hwung, W.J. (2002). An automatic
load/extract scheme for XML documents through ob-
ject-relational repositories. Journal of Systems and
Software, 64(3), 207-218.
Weng, S.S., & Liu, C.K. (2004). Using text classifica-
tion and multiple concepts to answer e-mails. Expert
Systems with Applications, 26(4), 529-543.
Westphal, C., & Blaxton, T. (1998). Data mining solu-
tions: Methods and tools for solving real-world prob-
lems. New York, NY: John Wiley.
Wong, P.K., Cowley, W., Foote, H., Jurrus, E., & Tho-
mas, J. (2000). Visualizing sequential patterns for text
mining. In Proceedings of the IEEE Symposium on
Information Visualization 2000 (p. 105). Retrieved from
http://portal.acm.org/citation.cfm
Yoon, B., & Park, Y. (2004). A text-mining-based patent
network: Analytical tool for high-technology trend. The
Journal of High Technology Management Research,
15(1), 37-50.
KEY TERMS
Business Intelligence: Any information that re-
veals threats and opportunities that can motivate a com-
pany to take some action (Halliman, 2001, p. 3).
Competitive Advantage: The head start a business has
owing to its access to new or unique information and
knowledge about the market in which it is operating.
386
Discovering Unknown Patterns in Free Text
Hypertext: A collection of texts containing links to
each other to form an interconnected network (Sullivan,
2001, p. 46).
Information Retrieval: The searching of a text
collection based on a users request to find a list of
documents organised according to its relevance, as
judged by the retrieval engine (Montes-y-Gmez et al.,
2004). Information retrieval should be distinguished
from text mining.
Knowledge Creation: The evaluation and interpre-
tation of patterns, trends or anomalies that have been
discovered in a collection of texts (or data in general),
as well as the formulation of its implications and conse-
quences, including suggestions concerning reactive busi-
ness decisions.
Knowledge Discovery: The discovery of patterns,
trends or anomalies that already exist in a collection of
texts (or data in general), but have not yet been identi-
fied or described.
Mark-Up Language: Tags that are inserted in free
text to mark structure, formatting and content. XML
tags can be used to mark attributes in free text and to
transform free text into an exploitable database (cf.,
Tseng & Hwung, 2002).
Metadata: Information regarding texts, for example,
author, title, publisher, date and place of publication,
journal or series, volume, page numbers, key words, etc.
Natural Language Processing (NLP): The automatic
analysis and/or processing of human language by com-
puter software, focussed on understanding the contents
of human communications. It can be used to identify
relevant data in large collections of free text for a data
mining process (Westphal & Blaxton, 1998, p. 116).
Parsing: A (NLP) process that analyses linguistic
structures and breaks them down into parts, on the
morphological, syntactic or semantic level.
Stemming: Finding the root form of related words,
for example singular and plural nouns, or present and
past tense verbs, to be used as key terms for calculating
occurrences in texts.
Text Mining: The automatic analysis of a large text
collection in order to identify previously unknown pat-
terns, trends or anomalies, which can be used to derive
business intelligence for competitive advantage or to
formulate and test scientific hypotheses.
TEAM LinG

Discovery Informatics
William W. Agresti
Johns Hopkins University, USA
INTRODUCTION
Discovery informatics is an emerging methodology that
brings together several threads of research and practice
aimed at making sense out of massive data sources. It is
defined as the study and practice of employing the full
spectrum of computing and analytical science and tech-
nology to the singular pursuit of discovering new infor-
mation by identifying and validating patterns in data
(Agresti, 2003).
BACKGROUND
In this broad-based conceptualization, discovery
informatics may be seen as taking shape by drawing on
more of the following established disciplines:
Database Management: Data models, data analy-
sis, data structures, data management, federation
of databases, data warehouses, database manage-
ment systems.
Pattern Recognition: Statistical processes, clas-
sifier design, image data analysis, similarity mea-
sures, feature extraction, fuzzy sets, clustering
algorithms.
Information Storage and Retrieval: Indexing,
content analysis, abstracting, summarization, elec-
tronic content management, search algorithms,
query formulation, information filtering, relevance
and recall, storage networks, storage technology.
Knowledge Management: Knowledge sharing,
knowledge bases, tacit and explicit knowledge,
relationship management, content structuring,
knowledge portals, collaboration support systems.
Artificial Intelligence: Learning, concept for-
mation, neural nets, knowledge acquisition, intel-
ligent systems, inference systems, Bayesian meth-
ods, decision support systems, problem solving,
intelligent agents, text analysis, natural language
processing.
What distinguishes discovery informatics is that it
brings coherence across dimensions of technologies
and domains to focus on discovery. It recognizes and
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
387
,
builds upon excellent programs of research and practice
in individual disciplines and application areas. It looks
selectively across these boundaries to find anything (e.g.,
ideas, tools, strategies, and heuristics) that will help with
the critical task of discovering new information.
To help characterize discovery informatics, it may
be useful to see if there are any roughly analogous
developments elsewhere. Two examplesknowledge
management and core competencemay be instructive
as reference points.
Knowledge management, which began its evolution
in the early 1990s, is the practice of transforming the
intellectual assets of an organization into business value
(Agresti, 2000). Of course, before 1990, organizations,
to varying degrees, knew that the successful delivery of
products and services depended on the collective knowl-
edge of employees. However, KM challenged organiza-
tions to focus on knowledge and recognize its key role
in their success. They found value in addressing ques-
tions such as the following:
What is the critical knowledge that should be
managed?
Where is the critical knowledge?
How does knowledge get into products and services?
When C. K. Prahalad and Gary Hamel published their
highly influential paper, The Core Competence of the
Corporation (Prahalad & Hamel, 1990), companies
had some capacity to identify what they were good at.
However, as with KM, most organizations did not appre-
ciate how identifying and cultivating core competences
(CC) may make the difference between being competi-
tive or not. A core competence is not the same as what
you are good at or being more vertically integrated. It
takes dedication, skill, and leadership to effectively
identify, cultivate, and deploy core competences for
organizational success.
Both KM and CC illustrate the potential value of
taking on a specific perspective. By doing so, an organi-
zation will embark on a worthwhile reexamination of
familiar topicsits customers, markets, knowledge
sources, competitive environment, operations, and suc-
cess criteria. The claim of this article is that discovery
informatics represents a distinct perspective, one that is
potentially highly beneficial, because, like KM and CC,
TEAM LinG

it strikes at what is often an essential element for
success and progressdiscovery.
MAIN THRUST
Both the technology and application dimensions will be
explored to help clarify the meaning of discovery
informatics.
Discovery Across Technologies
The technology dimension is considered broadly to in-
clude automated hardware and software systems, theo-
ries, algorithms, architectures, techniques, methods, and
practices. Included here are familiar elements associated
with data mining and knowledge discovery, such as clus-
tering, link analysis, rule induction, machine learning,
neural networks, evolutionary computation, genetic al-
gorithms, and instance-based learning (Wang, 2003).
However, the discovery informatics viewpoint goes
further, to activities and advances that are associated with
other areas but should be seen as having a role in discov-
ery. Some of these activities, like searching or knowl-
edge sharing, are well known from everyday experiences.
Conducting searches on the Internet is a common
practice that needs to be recognized as part of a thread
of information retrieval. Because it is practiced essen-
tially by all Internet users and involves keyword search,
there is a tendency to minimize its importance. Search
technology is extremely sophisticated (Baeza-Yates &
Ribiero-Neto, 1999). People always have some starting
point for their searches. Often, it is not a keyword, but
a concept. So people are forced to perform the transfor-
mation from a notional concept of what is desired to a
list of one or more keywords. The net effect can be the
familiar many-thousand hits from the search engine.
Even though the responses are ranked for relevance (a
rich and research-worthy subject itself), people may
still find that the returned items do not match their
intended concepts.
Offering hope for improved search are advances in
concept-based search (Houston & Chen, 2004), more
intuitiveness to a persons sense of find me content
like this, where this can be a concept embodied in an
entire document or series of documents. For example,
a person may be interested in learning which parts of a
new process guideline are being used in practice in the
pharmaceutical industry. Trying to obtain that informa-
tion through keyword searches typically would involve
trial and error on various combinations of keywords.
What the person would like to do is to point a search tool
to an entire folder of multimedia electronic content and
ask the tool to effectively integrate over the folder
388
Discovery Informatics
contents and then discover new items that are similar.
Current technology can support this ability to associate
a fingerprint with a document (Heintze, 2004) in order
to characterize its meaning, thereby enabling concept-
based searching. Discovery informatics recognize that
advances in search and retrieval enhance the discovery
process.
This same semantic analysis can be exploited in
other settings, such as within organizations. It is pos-
sible now to have your e-mail system prompt you, based
on the content of messages you compose. When you
click send, the e-mail system may open a dialogue box
(e.g., Do you also want to send that to Mary?). The
system has analyzed the content of your message, deter-
mining that, for messages in the past having similar
content, you also have sent them to Mary. So the system
is now asking you if you have perhaps forgotten to
include her. While this feature can certainly be intrusive
and bothersome unless it is wanted, the point is that the
same semantic analysis advances are at work here as
with the Internet search example.
The informatics part of discovery informatics also
conveys the breadth of science and technology needed
to support discovery. There are commercially available
computer systems and special-purpose software dedi-
cated to knowledge discovery (see listings at http://
www.kdnuggets.com/). The informatics support includes
comprehensive hardware-software discovery platforms
as well as advances in algorithms and data structures,
which are core subjects of computer science. The latest
developments in data sharing, application integration,
and human-computer interfaces are used extensively in
the automated support of discovery. Particularly valu-
able, because of the voluminous data and complex rela-
tionships, are advances in visualization (Marakas, 2003).
Commercial visualization packages are used widely to
display patterns and to enable expert interaction and
manipulation of the visualized relationships.
Discovery Across Domains
Discovery informatics encourages a view that spans
application domains. Over the past decade, the term has
been associated most often with drug discovery in the
pharmaceutical industry, mining biological data. The
financial industry also was known for employing tal-
ented programmers to write highly sophisticated math-
ematical algorithms for analyzing stock trading data,
seeking to discover patterns that could be exploited for
financial gain. Retailers were prominent in developing
large data warehouses that enabled mining across inven-
tory, transaction, supplier, marketing, and demographic
databases. The situation (marked by drug discovery
informatics, financial discovery informatics, etc.) was
TEAM LinG
Discovery Informatics
evolving into one in which discovery informatics was
preceded by more and more words as it was being used in
an increasing number of domain areas. One way to see the
emergence of discovery informatics is to strip away the
domain modifiers and recognize the universality of ev-
ery application area and organization wanting to take
advantage of its data.
Discovery informatics techniques are very influen-
tial across professions and domains:
In the financial community, the FinCEN Artificial
Intelligence System (FAIS) uses discovery
informatics methods to help identify money laun-
dering and other crimes. Rule-based inference is
used to detect networks of relationships among
people and bank accounts so that human analysts
can focus their attention on potentially suspicious
behavior (Senator, Goldberg & Wooton, 1995).
In the education profession, there are exciting
scenarios that suggest the potential for discovery
informatics techniques. In one example, a knowl-
edge manager works with a special education teacher
to evaluate student progress. The data warehouse
holds extensive data on students, their performance,
teachers, evaluations, and course content. The edu-
cators are pleased that the aggregate performance
data for the school is satisfactory. However, with
discovery informatics capabilities, the story does
not need to end there. The data warehouse permits
finer granularity examination, and the discovery
informatics tools are able to find patterns that are
hidden by the summaries. The tools reveal that
examinations involving word problems show much
poorer performance for certain students. Further
analysis shows that this outcome was also true in
previous courses for these students. Now, the edu-
cators have the specific data enabling them to take
action in the form of targeted tutorials for specific
students to address the problem (Tsantis &
Castellani, 2001).
Bioinformatics requires the intensive use of infor-
mation technology and computer science to ad-
dress a wide array of challenges (Watkins, 2001).
One example illustrates a multi-step computational
method to predict gene models, an important activ-
ity that has been addressed to date by combinations
of gene prediction programs and human experts.
The new method is entirely automated, involving
optimization algorithms and the careful integration
of the results of several gene prediction programs,
including evidence from annotation software. Sta-
tistical scoring is implemented with decision trees
to show clearly the gene prediction results (Allen,
Pertea & Salzberg, 2004).
Common Elements of Discovery
Informatics ,
The only constant in discovery informatics is data and
an interacting entity with an interest in discovering new
information from it. What varies and has an enormous
effect on the ease of discovering new information is
everything else, notably the following:
Data
Volume: How much?
Accessibility: Ease of access and analysis?
Quality: How clean and complete is it? Can it be
trusted as accurate?
Uniformity: How homogeneous are the data?
Are they in multiple forms, structures, formats,
and locations?
Medium: Text, numbers, audio, video, image,
electronic, or magnetic signals or emanations?
Structure of the Data: Formatted rigidly, par-
tially, or not at all? If text, does it adhere to known
language?
Interacting Entity
Nature: Is it a person or intelligent agent?
Need, Question, or Motivation of the User:
What is prompting a person to examine these
data? How sharply defined is the question or
need? What expectations exist about what might
be found? If the motivation is to find something
interesting, what does that mean in context?
A wide variety of activities may be seen as varia-
tions on this scheme. An individual using a search
engine to search the Internet for a home mortgage
serves as an instance of query-driven discovery. A
retail company looking for interesting patterns in its
transaction data is engaged in data-driven discovery. An
intelligent agent examining newly posted content to the
Internet based on a persons profile is also engaged in
a process of discovery, only the interacting entity is not
a person.
FUTURE TRENDS
There are many indications that discovery informatics
will grow in relevance. Storage costs are dropping; for
example, the cost per gigabyte of magnetic disk storage
declined by a factor of 500 from 1990 to 2000 (Univer-
389
TEAM LinG

sity of California at Berkeley, 2003). Our stockpiles of
data are expanding rapidly in every field of endeavor.
Businesses at one time were comfortable with opera-
tional data summarized over days or even weeks. Increas-
ing automation led to point-of-decision data on transac-
tions. With online purchasing, it is now possible to know
the sequence of clickstreams leading up to the sale. So
the granularity of the data is becoming finer, as busi-
nesses are learning more about their customers and about
ways to become more profitable. This business analytics
is essential for organizations to be competitive.
A similar process of finer granular data exists in
bioinformatics. In the human body, there are 23 pairs of
human chromosomes, approximately 30,000 genes, and
more than 1,000,000 proteins (Watkins, 2001). The
advances in decoding the human genome are remark-
able, but it is proteins that ultimately regulate metabo-
lism and disease in the body (Watkins, 2001, p. 27). So,
the challenges for bioinformatics continue to grow
along with the data.
CONCLUSION
Discovery informatics is an emerging methodology that
promotes a crosscutting and integrative view. It looks
across both technologies and application domains to
identify and organize the techniques, tools, and models
that improve data-driven discovery.
There are significant research questions as this meth-
odology evolves. Continuing progress will be received
eagerly from efforts in individual strategies for knowl-
edge discovery and machine learning, such as the excel-
lent contributions in Koza, et al. (2003). An additional
opportunity is to pursue the recognition of unifying
aspects of practices now associated with diverse disci-
plines. While the anticipation of new discoveries is
exciting, the evolving practical application of discovery
methods needs to respect individual privacy and a di-
verse collection of laws and regulations. Balancing
these requirements constitutes a significant and persis-
tent challenge as new concerns emerge and as laws are
drafted.
Looking ahead to the challenges and opportunities
of the 21st century, discovery informatics is poised to
help people and organizations learn as much as possible
from the worlds abundant and ever-growing data assets.
REFERENCES
Agresti, W.W. (2000). Knowledge management. Ad-
vances in Computers, 53, 171-283.
390
Discovery Informatics
Agresti, W.W. (2003). Discovery informatics. Commu-
nications of the ACM, 46(8), 25-28.
Allen, J.E., Pertea, M., & Salzberg, S.L. (2004). Computa-
tional gene prediction using multiple sources of evidence.
Genome Research, 14, 142-148.
Baeza-Yates, R., & Ribiero-Neto, B. (1999). Modern infor-
mation retrieval. Reading, MA: Addison-Wesley.
Bergeron, B. (2003). Bioinformatics computing. Upper
Saddle River, NJ: Prentice Hall.
Heintze, N. (2004). Scalable document fingerprinting.
Carnegie Mellon University. Retrieved from http://
www2.cs.cmu.edu/afs/cs/user/nch/www/koala/
main.html
Houston, A., & Chen, H. (2004). A path to concept-based
information access: From national collaboratories to
digital libraries. University of Arizona. Retrieved from
http://ai.bpa.arizona.edu/go/intranet/papers/
Book7.pdf
Koza, J.R. et al. (Eds.) (2003). Genetic programming IV:
Routine human-competitive machine intelligence.
Dordrecht, The Netherlands: Kluwer Academic Publishers.
Marakas, G.M. (2003). Modern data warehousing,
mining, and visualization. Upper Saddle River, NJ:
Prentice Hall.
Prahalad, C.K., & Hamel, G. (1990). The core compe-
tence of the corporation. Harvard Business Review, 3,
79-91.
Senator, T.E., Goldberg, H.G., & Wooton, J. (1995). The
financial crimes enforcement network AI system (FAIS):
Identifying potential money laundering from reports of
large cash transactions. AI Magazine, 16, 21-39.
Tsantis, L., & Castellani, J. (2001). Enhancing learning
environments through solution-based knowledge dis-
covery tools: Forecasting for self-perpetuating sys-
temic reform. Journal of Special Education Technol-
ogy, 16, 39-52.
University of California at Berkeley. (2003). How much
information? School of Information Management and
Systems. Retrieved from http://www.sims.berkeley.edu/
research/projects/how-much-info/how-much-info.pdf
Wang, J. (2003). Data mining: Opportunities and chal-
lenges. Hershey, PA: Idea Group Publishing.
Watkins, K.J. (2001). Bioinformatics. Chemical & En-
gineering News, 79, 26-45.
TEAM LinG
Discovery Informatics
KEY TERMS
Clickstream: The sequence of mouse clicks ex-
ecuted by an individual during an online Internet session.
Data Mining: The application of analytical methods
and tools to data for the purpose of identifying patterns
and relationships such as classification, prediction, es-
timation, or affinity grouping.
Discovery Informatics: The study and practice of
employing the full spectrum of computing and analyti-
cal science and technology to the singular pursuit of
discovering new information by identifying and validat-
ing patterns in data.
Evolutionary Computation: Solution approach
guided by biological evolution, which begins with po-
tential solution models and then iteratively applies al-
gorithms to find the fittest models from the set to serve as
inputs to the next iteration, ultimately leading to a model ,
that best represents the data.
Knowledge Management: The practice of transform-
ing the intellectual assets of an organization into business
value.
Neural Networks: Learning systems, designed by
analogy with a simplified model of the neural connec-
tions in the brain, which can be trained to find nonlinear
relationships in data.
Rule Induction: Process of learning from cases or
instances the if-then rule relationships consisting of an
antecedent (i.e., if-part, defining the preconditions or
coverage of the rule) and a consequent (i.e., then-part,
stating a classification, prediction, or other expression of
a property that holds for cases defined in the antecedent).
391
TEAM LinG
392
Discretization for Data Mining
Ying Yang
Monash University, Australia
Geoffrey I. Webb
Monash University, Australia
INTRODUCTION
Discretization is a process that transforms quantitative
data into qualitative data. Quantitative data are commonly
involved in data mining applications. However, many
learning algorithms are designed primarily to handle quali-
tative data. Even for algorithms that can directly deal with
quantitative data, learning is often less efficient and less
effective. Hence research on discretization has long been
active in data mining.
BACKGROUND
Many data mining systems work best with qualitative
data, where the data values are discrete descriptive terms
such as young and old. However, lots of data are quan-
titative, for example, with age being represented by a
numeric value rather a small number of descriptors. One
way to apply existing qualitative systems to such quan-
titative data is to transform the data.
Discretization is a process that transforms data con-
taining a quantitative attribute so that the attribute in
question is replaced by a qualitative attribute. A many to
one mapping function is created so that each value of the
original quantitative attribute is mapped onto a value of
the new qualitative attribute. First, discretization divides
the value range of the quantitative attribute into a finite
number of intervals. The mapping function associates all
of the quantitative values in a single interval to a single
qualitative value. A cut point is a value of the quantitative
attribute where a mapping function locates an interval
boundary. For example, a quantitative attribute recording
age might be mapped onto a new qualitative age attribute
with three values, pre-teen, teenage, and post-teen. The
cut points for such a discretization may be 13 and 18.
Values of the original quantitative age attribute that are
below 13 might get mapped onto the pre-teen value of the
new attribute, values from 13 to 18 onto teen, and values
above 18 onto post-teen.
Various discretization methods have been proposed.
Diverse taxonomies exist in literature to categorize
discretization methods. These taxonomies are comple-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
mentary, each relating to a different dimension along
which discretization methods may differ. Typically,
discretization methods can be either primary or compos-
ite. Primary methods accomplish discretization without
reference to any other discretization method. Composite
methods are built on top of some primary method(s).
Primary methods can be classified as per the following
taxonomies.
Supervised vs. Unsupervised (Dougherty, Kohavi,
& Sahami, 1995): Supervised methods are only
applicable when mining data that are divided into
classes. These methods refer to the class informa-
tion when selecting discretization cut points. Unsu-
pervised methods do not use the class information.
For example, when trying to predict whether a cus-
tomer will be profitable, the data might be divided
into two classes profitable and unprofitable. A
supervised discretization technique would take
account of how useful was the selected cut point for
identifying whether a customer was profitable. An
unsupervised technique would not. Supervised
methods can be further characterized as error-based,
entropy-based or statistics-based. Error-based
methods apply a learner to the transformed data and
select the intervals that minimize error on the train-
ing data. In contrast, entropy-based and statistics-
based methods assess respectively the class en-
tropy or some other statistic regarding the relation-
ship between the intervals and the class.
Parametric vs. Non-Parametric: Parametric
discretization requires the user to specify param-
eters for each discretization performed. An example
of such a parameter is the maximum number of
intervals to be formed. Non-parametric discretization
does not utilize user-specified parameters.
Hierarchical vs. Non-Hierarchical: Hierar-
chical discretization utilizes an incremental pro-
cess to select cut points. This creates an implicit
hierarchy over the value range. Hierarchical
discretization can be further characterized as ei-
ther split or merge (Kerber, 1992). Split
discretization starts with a single interval that
TEAM LinG
Discretization for Data Mining
encompasses the entire value range, then repeat-
edly splits it into sub-intervals until some stop-
ping criterion is satisfied. Merge discretization
starts with each value in a separate interval, then
repeatedly merges adjacent intervals until a stop-
ping criterion is met. It is possible to combine
both split and merge techniques. For example,
initial intervals may be formed by splitting. A
merge process is then applied to post-process
these initial intervals. Non-hierarchical
discretization creates intervals without forming a
hierarchy. For example, many methods forming
the intervals sequentially in a single scan through
the data.
Univariate vs. Multivariate (Bay, 2000): Univariate
methods discretize an attribute without reference to
attributes other than the class. In contrast, multi-
variate methods consider relationships among at-
tributes during discretization.
Disjoint vs. Non-Disjoint (Yang & Webb, 2002):
Disjoint methods discretize the value range of an
attribute into intervals that do not overlap. Non-
disjoint methods allow overlap between intervals.
Global vs. Local (Dougherty, Kohavi, & Sahami,
1995): Global methods create a single mapping
function that is applied throughout a given classi-
fication task. Local methods allow different map-
ping functions for a single attribute in different
classification contexts. For example, decision tree
learning may discretize a single attribute into differ-
ent intervals at different nodes of a tree (Quinlan,
1993). Global techniques are more efficient, because
one discretization is used throughout the entire
data mining process, but local techniques may re-
sult in the discovery of more useful cut points.
Eager vs. Lazy (Hsu, Huang, & Wong, 2000, 2003):
Eager methods generate the mapping function prior
to classification time. Lazy methods generate the
mapping function as it is needed during classifica-
tion time.
Ordinal vs. Nominal: Ordinal discretization forms a
mapping function from quantitative to ordinal quali-
tative data. It seeks to retain ordering information
implicit in quantitative attributes. In contrast, nomi-
nal discretization forms a mapping function from
quantitative to nominal qualitative data, thereby
discarding any ordering information. For example, if
the value range 0 29 were discretized into three
intervals 0 9, 10 19 and 20 29, if the intervals are
treated as nominal then a value in the interval 0 9
will be treated as dissimilar to one in 20 29 as it is
to one in 10 19. In contrast, while ordinal
discretization will treat the difference between 9 and
either 10 or 19 as equivalent, it retains the informa-
tion that this difference is less than the difference
between 9 and 29. ,
Fuzzy vs. Non-fuzzy (Ishibuchi, Yamamoto, &
Nakashima, 2001; Wu, 1999): Fuzzy discretization
creates a fuzzy mapping function. A value may
belong to multiple intervals, each with varying de-
grees of strength. Non-fuzzy discretization forms
exact cut points.
Composite methods first generate a mapping function
using an initial primary method. They then use other
primary methods to adjust the initial cut points.
MAIN THRUST
The main thrust of this chapter deals with how to select
a discretization method. This issue is particularly impor-
tant since there exist a large number of discretization
methods and no one can be universally optimal. When
selecting between discretization methods it is critical to
take account of the learning context, in particular, of the
learning algorithm, the nature of the data, and the learning
objectives. Different learning contexts have different char-
acteristics and hence have different requirements for
discretization. It is unrealistic to pursue a universally
optimal discretization approach that can be blind to its
learning context.
Many discretization techniques have been developed
primarily in the context of a specific type of learning
algorithm, such as decision tree learning, decision rule
learning, naive-Bayes learning, Bayes network learning,
clustering, and association learning. Different types of
learning have different characteristics and hence require
different strategies of discretization.
For example, decision tree learners can suffer from the
fragmentation problem. If an attribute has many values, a
split on this attribute will result in many branches, each of
which receives relatively few training instances, making
it difficult to select appropriate subsequent tests. Hence
they may benefit more than other learners from
discretization that results in few intervals. Decision rule
learners may require pure intervals (containing instances
dominated by a single class), while probabilistic learners
such as naive-Bayes do not. The relations between at-
tributes are key themes for association learning, and
hence multivariate discretization that can capture the
inter-dependencies among attributes is desirable. If
coupled with lazy discretization, lazy learners can further
save training effort. Non-disjoint discretization is not
applicable if the learning algorithm, such as decision tree
learning, requires disjoint attribute values.
In order to facilitate understanding this issue, we
contrast discretization strategies in two popular learning
393
TEAM LinG

contexts, decision tree learning and naive-Bayes learning.
Although both are commonly used for data mining appli-
cations, they have very different inductive biases and
learning mechanisms. As a result, they desire different
discretization methodologies.
Discretization in Decision Tree Learning
The learned concept is represented by a decision tree in
decision tree learning. Each non-leaf node tests an at-
tribute. Each branch descending from that node corre-
sponds to one of the attributes values. Each leaf node
assigns a class label. A decision tree classifies instances
by sorting them down the tree from the root to some leaf
node (Mitchell, 1997). Algorithms such as ID3 (Quinlan,
1986) and its successor C4.5 (Quinlan, 1993) are well known
exemplars.
Fayyad & Irani (1993) proposed multi-interval-entropy-
minimization discretization (MIEMD), which has been one
of the most popular discretization mechanisms for deci-
sion tree learning over years. Briefly speaking, MIEMD
discretizes a quantitative attribute by calculating the class
information entropy as if the classification only uses that
single attribute after discretization. This can be suitable
for the divide-and-conquer strategy of decision tree learn-
ing, which handles one attribute at a time. However, it is not
necessarily appropriate for other learning mechanisms
such as naive-Bayes learning, which involves all the at-
tributes simultaneously (Yang, 2003).
Furthermore, MIEMD uses the minimum description
length criterion (MDL) as its termination condition that
decides when to stop further partitioning a quantitative
attributes value range. As An and Cercone (1999) indi-
cate, this criterion has an effect to form qualitative at-
tributes with few values. This effect is desirable for deci-
sion tree learning, since it helps avoid the fragmentation
problem by minimizing the number of values of an attribute.
If an attribute has many values, forming a split on those
values fragments that data into small subsets with respect
to which it is difficult to perform further learning (Quinlan,
1993). However, this effect of minimization is not so wel-
come to naive-Bayes learning because it brings adverse
impact as we will detail in the next section.
Discretization in Naive-Bayes Learning
When classifying an instance, nave-Bayes learning ap-
plies Bayes theorem to calculate the probability of each
class given this instance. The most probable class is
chosen as the class of this instance. In order to simplify the
calculation, an attribute independence assumption is
made, assuming attributes conditionally independent of
each other given the class. Although this assumption is
394
Discretization for Data Mining
often violated in real-world applications, naive-Bayes
learning still achieves surprisingly good classification
performance. Domingos and Pazzani (1997) suggested
one reason is that the classification estimation under
zero-one loss is only a function of the sign of the
probability estimation. The classification accuracy can
remain high even while the assumption violation causes
poor probability estimation, so long as the highest
estimate relates to the correct class. Because they are
simple, effective, efficient, robust to noise, and sup-
port incremental training, nave-Bayes classifiers have
been employed in numerous classification tasks.
Appropriate discretization mechanisms for naive-
Bayes learning include fixed-frequency discretization
(Yang, 2003), proportional discretization (Yang, 2003)
and non-disjoint discretization (Yang, 2003). For ex-
ample, when discretizing a quantitative attribute, fixed-
frequency discretization (FFD) predefines a sufficient
interval frequency k. It then discretizes the sorted values
into intervals so that each interval has approximately the
same number k of training instances with adjacent (pos-
sibly identical) values. By this means, FFD fixes an
interval frequency that is not arbitrary but can ensure
that each interval contains sufficient instances to allow
reasonable probability estimates.
However, FFD may result in inferior performance for
decision tree learning. FFD first ensures that each inter-
val contains sufficient instances for estimating the na-
ive-Bayes probabilities. On top of that, FFD tries to
maximize the number of discretized intervals to reduce
discretization bias (Yang, 2003). If employed in decision
tree learning, this maximization effect of FFD tends to
cause a severe fragmentation problem.
The other way around, MIEMD is effective for deci-
sion tree learning but not for naive-Bayes learning.
Because of its attribute independence assumption, na-
ive-Bayes learning is not subject to the fragmentation
problem. MIEMDs tendency to minimize the number of
intervals has a strong potential to reduce the classifica-
tion variance but increase the classification bias. As the
data size becomes large, it is very likely that the loss
through bias increase will soon overshadow the gain
through variance reduction, resulting in inferior learning
performance (Yang, 2003). This impact is particularly
undesirable, since, due to its efficiency, naive-Bayes
learning is very popular for learning from large data.
Hence, MIEMD is not a desirable approach for
discretization in naive-Bayes learning.
Discretization in Association Rule
Discovery
Association rules (Agrawal, Imielinski, & Swami, 1993)
have quite distinct discretization requirements to the
TEAM LinG
Discretization for Data Mining
classification learning techniques discussed above. Be-
cause any attribute may appear in the consequent of an
association rule, there is no single class variable with
respect to which global supervised discretization might
be performed. Further, there is no clear evaluation crite-
rion for the sets of rules that are produced. Whereas it is
possible to assess the expected error rate of a decision
tree or a nave Bayes classifier, there is no corresponding
metric for comparing the quality of two alternative sets of
association rules. In consequence, it is not apparent how
one might assess the quality of two alternative
discretizations of an attribute. It is not even possible to
run the system with each alternative and evaluate the
quality of the respective results. For these reasons simple
global discretization techniques such as fixed frequency
discretization are often used.
In contrast, Srikant & Agrawal (1996) present a hybrid
global unsupervised and local multi-variate supervised
discretization technique. Each attribute is initially
discretized using fixed frequency discretization. Then, for
each rule a locally optimal discretization is generated by
considering new discretizations formed by joining neigh-
boring intervals. This technique is more computationally
demanding than a simple global approach, but may result
in more useful rules.
FUTURE TRENDS
Many problems still remain open with regard to
discretization.
The relationship between the nature of a learning task
and the appropriate discretization strategies requires fur-
ther investigation. One challenge is to identify what
aspects of a learning task are relevant to the selection of
discretization strategies. It would be useful to character-
ize tasks and discretization methods into abstract features
that facilitate the selection.
Discretization for time-series data is another interest-
ing topic. In time-series data, each instance is associated
with a time stamp. The concept that underlies the data may
drift over time, which implies that the appropriate
discretization cut points may change. It will be time
consuming if discretization has to be conducted from
scratch each time the data changes. In this case, incremen-
tal discretization that only needs the old cut points and
new data to form new cut points can be of great utility.
A third trend is discretization for stream data. A
fundamental difference between time-series data and
stream data is that for stream data, one only has access to
data in the current time window, but not any previous
data. The data may have large volume, change very fast
and require fast response, for example, exchange data
from the stock market and observation data from monitor-
ing sensors. The key theme here is to boost ,
discretizations efficiency (without any significant ac-
curacy loss) and to quickly incorporate discretizations
results into the learner.
CONCLUSION
The process that transforms quantitative data to qualita-
tive data is discretization. The real world is abundant in
quantitative data. In contrast, many learning algorithms
are more adept at learning from qualitative data. This gap
can be shrunk by discretization, which makes discretization
an important research area for knowledge discovery.
Numerous methods have been developed as under-
standing of discretization evolves. Different methods are
tuned to different learning tasks. When seeking to employ
an existing discretization method or develop a new
discretization mechanism, it is very important to under-
stand the learning context within which the discretization
lies. For example, what learning algorithm will make use of
the discretized values? Different learning algorithms have
different characteristics and require different discretization
strategies. There is no universally optimal discretization
solution.
There are significant research questions that still
remain open in discretization. Continuing progress is
both desirable and necessary.
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
associations between sets of items in massive databases.
Proceedings of the 1993 ACM-SIGMOD International
Conference on Management of Data (pp. 207-216).
An, A., & Cercone, N. (1999). Discretization of continuous
attributes for learning classification rules. Proceedings of
the 3rd Pacific-Asia Conference on Methodologies for
Knowledge Discovery and Data Mining (pp. 509-514).
Bay, S.D. (2000). Multivariate discretization of continu-
ous variables for set mining. Proceedings of the 6th ACM
SIGKDD International Conference on Knowledge Dis-
covery and Data Mining (pp. 315-319).
Bluman, A.G. (1992). Elementary statistics: A step by step
approach. Dubuque, IA: Wm.C.Brown Publishers.
Domingos, P., & Pazzani, M. (1997). On the optimality of
the simple Bayesian classifier under zero-one loss. Ma-
chine Learning, 29, 103-130.
395
TEAM LinG

Dougherty, J., Kohavi, R., & Sahami, M. (1995). Super-
vised and unsupervised discretization of continuous fea-
tures. Proceedings of the 12th International Confer-
ence on Machine Learning (pp. 94-202).
Fayyad, U.M., & Irani, K.B. (1993). Multi-interval
discretization of continuous-valued attributes for classi-
fication learning. Proceedings of the 13th International
Joint Conference on Artificial Intelligence (pp. 1022-
1027).
Hsu, C.N., Huang, H.J., & Wong, T.T. (2000). Why
discretization works for naive Bayesian classifiers. Pro-
ceedings of the 17th International Conference on Ma-
chine Learning (pp. 309-406).
Hsu, C.N., Huang, H.J., & Wong, T.T. (2003). Implications
of the Dirichlet assumption for discretization of continu-
ous variables in naive Bayesian classifiers. Machine
Learning, 53(3), 235-263.
Ishibuchi, H., Yamamoto, T., & Nakashima, T. (2001).
Fuzzy data mining: Effect of fuzzy discretization. Proceed-
ings of the 2001 IEEE International Conference on Data
Mining (pp. 241-248).
Kerber, R. (1992). Chimerge: Discretization for numeric
attributes. Proceedings of 10 th National Conference on
Artificial Intelligence (pp. 123-128).
Mitchell, T.M. (1997). Machine learning. New York, NY:
McGraw-Hill Companies.
Quinlan, J.R. (1986). Induction of decision trees. Machine
Learning, 1, 81-106.
Quinlan, J.R. (1993). C4.5: Programs for machine learn-
ing. San Francisco, CA: Morgan Kaufmann Publishers.
Samuels, M.L., & Witmer, J.A. (1999). Statistics for the life
sciences (2nd ed.). Prentice-Hall.
Srikant, R., & Agrawal, R. (1996). Mining quantitative
association rules in large relational tables. Proceedings of
the 1996 ACM-SIGMOD International Conference on
Management of Data (pp. 1-12).
Wu, X. (1999). Fuzzy interpretation of discretized inter-
vals. IEEE Transactions on Fuzzy Systems, 7(6), 753-759.
396
Discretization for Data Mining
Yang, Y., & Webb, G.I. (2002). Non-disjoint discretization
for naive-Bayes classifiers. Proceedings of the 19th Inter-
national Conference on Machine Learning (pp. 666-
673).
Yang, Y. (2003). Discretization for naive-Bayes learning.
PhD thesis, School of Computer Science and Software
Engineering, Monash University, Melbourne, Australia.
KEY TERMS
Turning to the authority of introductory statistical
textbooks (Bluman, 1992; Samuels & Witmer, 1999), the
following definitions are adopted.
Continuous Data: Can assume all values on the num-
ber line within their value range. The values are obtained
by measuring. An example is: temperature.
Discrete Data: Assume values that can be counted.
The data cannot assume all values on the number line
within their value range. An example is: number of children
in a family.
Discretization: A process that transforms quantita-
tive data to qualitative data.
Nominal Data: Classified into mutually exclusive (non-
overlapping), exhaustive categories in which no meaning-
ful order or ranking can be imposed on the data. An
example is: blood type of a person: A, B, AB, O.
Ordinal Data: Classified into categories that can be
ranked. However, the differences between the ranks can-
not be calculated by arithmetic. An example is: assign-
ment evaluation: fail, pass, good, excellent.
Qualitative Data: Also often referred to as categorical
data, are data that can be placed into distinct categories.
Qualitative data sometimes can be arrayed in a meaningful
order. But no arithmetic operations can be applied to them.
Quantitative data can be further classified into two groups,
nominal or ordinal.
Quantitative Data: Numeric in nature. They can be
ranked in order. They also admit to meaningful arithmetic
operations. Quantitative data can be further classified
into two groups, discrete or continuous.
TEAM LinG

Discretization of Continuous Attributes
Fabrice Muhlenbach
EURISE, Universit Jean Monnet - Saint-Etienne, France
Ricco Rakotomalala
ERIC, Universit Lumire - Lyon 2, France
INTRODUCTION
In the data-mining field, many learning methods such
as association rules, Bayesian networks, and induction
rules (Grzymala-Busse & Stefanowski, 2001) can
handle only discrete attributes. Therefore, before the
machine-learning process, it is necessary to re-encode
each continuous attribute in a discrete attribute consti-
tuted by a set of intervals. For example, the age attribute
can be transformed in two discrete values representing
two intervals: less than 18 (a minor) and 18 or greater.
This process, known as discretization, is an essential
task of the data preprocessing not only because some
learning methods do not handle continuous attributes,
but also for other important reasons. The data trans-
formed in a set of intervals are more cognitively rel-
evant for a human interpretation (Liu, Hussain, Tan, &
Dash, 2002); the computation process goes faster with
a reduced level of data, particularly when some at-
tributes are suppressed from the representation space
of the learning problem if it is impossible to find a
relevant cut (Mittal & Cheong, 2002); the discretization
can provide nonlinear relations for example, the
infants and the elderly people are more sensitive to
illness. This relation between age and illness is then not
linear which is why many authors propose to discretize
the data even if the learning method can handle continu-
ous attributes (Frank & Witten, 1999). Lastly,
discretization can harmonize the nature of the data if it
is heterogeneous for example, in text categorization,
the attributes are a mix of numerical values and occur-
rence terms (Macskassy, Hirsh, Banerjee, & Dayanik,
2001).
An expert realizes the best discretization because he
can adapt the interval cuts to the context of the study and
can then make sense of the transformed attributes. As
mentioned previously, the continuous attribute age
can be divided in two categories. Take basketball as an
example; what is interesting about this sport is that it has
many categories: mini-mite (under 7), mite (7 to 8),
squirt (9 to 10), peewee (11 to 12), bantam (13 to
14), midget (15 to 16), junior (17 to 20), and
senior (over 20). Nevertheless, this approach is not
feasible in the majority of machine-learning problem
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
397
,
cases because there are no experts available, no a priori
knowledge on the domain, or, for a big dataset, the
human cost would be prohibitive. It is then necessary to
be able to have an automated method to discretize the
predictive attributes and find the cut-points that are
better adapted to the learning problem.
Discretization was little studied in statistics ex-
cept by some rather old articles considering it as a
special case of the one-dimensional clustering (Fisher,
1958) but from the beginning of the 1990s, the
research expanded very quickly with the development of
supervised methods (Dougherty, Kohavi, & Sahami,
1995; Liu et al., 2002). Lately, the applied discretization
has affected other fields: An efficient discretization can
also improve the performance of discrete methods such
as the association rule construction (Ludl & Widmer,
2000a) or the machine learning of a Bayesian network
(Friedman & Goldsmith, 1996).
In this article, we will present the discretization as a
preliminary condition of the learning process. The pre-
sentation will be limited to the global discretization
methods (Frank & Witten, 1999), because in a local
discretization, the cutting process depends on the par-
ticularities of the model construction for example,
the discretization in rule induction associated with ge-
netic algorithms (Divina, Keijzer, & Marchiori, 2003)
or lazy discretization associated with nave Bayes clas-
sifier induction (Yang & Webb, 2002). Moreover, even
if this article presents the different approaches to
discretize the continuous attributes, whatever the learn-
ing method may be used, in the supervised learning
framework, only discretizing the predictive attributes
will be presented. The cutting of the attributes to be
predicted depends a lot on the particular properties of
the problem to treat. The discretization of the class
attribute is not realistic because this pretreatment, if
effectuated, would be the learning process itself.
BACKGROUND
The discretization of a continuous-valued attribute con-
sists of transforming it into a finite number of intervals
and to re-encode, for all instances, each value on this
TEAM LinG

attribute by associating it with its corresponding inter-
val. There are many ways to realize this process.
One of these ways consists of realizing a
discretization with a fixed number of intervals. In this
situation, the user must choose the appropriate number
a priori: Too many intervals will be unsuited to the
learning problem, and too few intervals can risk losing
some interesting information. A continuous attribute
can be divided in intervals of equal width (see Figure 1)
or equal frequency (see Figure 2). Other methods exist to
constitute the intervals based on the clustering prin-
ciples, for example, k-means clustering discretization
(Monti & Cooper, 1999).
Nevertheless, for supervised learning, these
discretization methods ignore an important source of
information: the instance labels of the class attribute.
By contrast, the supervised discretization methods
handle the class label repartition to achieve the differ-
ent cuts and find the more appropriate intervals. Fig-
ure 3 shows a situation where it is more efficient to have
only two intervals for the continuous attribute instead of
three: It is not relevant to separate two bordering inter-
vals if they are composed of the same class data. There-
fore, the supervised or unsupervised quality of a
discretization method is an important criterion to take
into consideration.
Another important criterion to qualify a method is
the fact that a discretization either processes on the
different attributes one by one or takes into account the
whole set of attributes for doing an overall cutting. The
second case, called multivariate discretization, is par-
ticularly interesting when some interactions exist be-
tween the different attributes. In Figure 4, a supervised
discretization attempts to find the correct cuts by taking
into account only one attribute independently of the
others. This will fail: It is necessary to represent the
data with the attributes X1 and X2 together to find the
appropriate intervals on each attribute.
MAIN THRUST
The two criteria mentioned in the previous section
unsupervised/supervised and univariate/multivariate
will characterize the major discretization method fami-
Figure 1. Equal width discretization
Figure 2. Equal frequency discretization
398
Discretization of Continuous Attributes
lies. In the following sections, we use these criteria to
distinguish the particularities of each discretization
method.
Univariate Unsupervised Discretization
The simplest discretization methods make no use of the
instance labels of the class attribute. For example, the
equal width interval binning consists of observing the
values of the dataset to identify the minimum and the
maximum values observed and to divide the continuous
attribute into the number of intervals chosen by the user
(Figure 1). Nevertheless, in this situation, if uncharac-
teristic extreme values exist in the dataset (outliers),
the range will be changed, and the intervals will be
misappropriated. To avoid this problem, divide the con-
tinuous attribute into intervals containing the same num-
ber of instances (Figure 2): This method is called the
equal frequency discretization method.
The unsupervised discretization can be grasped as a
problem of sorting and separating intermingled prob-
ability laws (Potzelberger & Felsenstein, 1993). The
existence of an optimum analysis was studied by Teicher
(1963) and Yakowitz and Spragins (1968). Neverthe-
less, these methods are limited in their application in
data mining due to too strong of statistical hypotheses
seldom checked with real data.
Univariate Supervised Discretization
To improve the quality of a discretization in supervised
data-mining methods, it is important to take into ac-
count the instance labels of the class attribute. Figure 3
shows the problem of constituting intervals without the
information of the class attribute. The intervals that are
the better adapted to a discrete machine-learning method
are the pure intervals containing only instances of a
given class. To obtain such intervals, the supervised
discretization methods such as the state-of-the-art
method Minimum Description Length Principle Cut
(MDLPC) are based on statistical or information-theo-
retical criteria and heuristics (Fayyad & Irani, 1993).
In a particular case, even if one supervised method can
give better results than another (Kurgan & Krysztof,
2004), with real data, the improvements of one method
TEAM LinG
Discretization of Continuous Attributes
Figure 3. Supervised and unsupervised discretizations
compared to the other supervised methods are insignifi-
cant. Moreover, the performance of a discretization method
is difficult to estimate without a learning algorithm. In
addition, the final results can arise from the discretization
processing, the learning processing, or the combination of
both. Because the discretization is realized in an ad hoc
way, independently of the learning algorithm characteris-
tics, there is no guarantee that the interval cut will be
optimal for the learning method. Only a little work showed
the relevance and the optimality of the global discretization
for a very specific classifier such as nave Bayes (Hsu,
Huang, & Wong, 2003; Yang & Webb, 2003). The super-
vised discretization methods can be distinguished de-
pending on the way the algorithm proceeds: bottom-up
(each value represents an interval, which is merged pro-
gressively with the others to constitute the appropriate
number of intervals) or top-down (the whole dataset rep-
resents an interval and is progressively cut to constitute
the appropriate number of intervals). However, there are
no significant performance differences between these two
latest approaches (Zighed, Rakotomalala, & Feschet, 1997).
In brief, the different supervised (univariate) methods
can be characterized by (a) the particular statistical crite-
rion used to evaluate the degree of an intervals purity, (2)
the top-down or bottom-up strategy to find the cut-points
used to determine the intervals.
Multivariate Unsupervised
Discretization
Association rules constitute an unsupervised learning
method that needs discrete attributes. For such a method,
the discretization of a continuous attribute can be real-
ized in an univariate way but also in a multivariate way. In
the latter case, each attribute is cut in relation to the
other attributes of the database; this approach can then
provide some interesting improvements when unsuper-
vised univariate discretization methods do not yield satis-
factory results.
Figure 4. Interaction between the attributes X1 and X2
X2
,
X1
The multivariate unsupervised discretizations can be
performed by clustering techniques using all attributes
globally. It is also possible to consider each cluster
obtained as a class and improve the discretization quality
by using (univariate) supervised discretization methods
(Chmielewski & Grzymala-Busse, 1994).
An approach called multisupervised discretization
(Ludl & Widmer, 2000a) can be seen as a particular
unsupervised multivariate discretization. This method
starts with the temporary univariate discretization of
all attributes. Then the final cutting of a given attribute
is based on the univariate supervised discretization of
all other attributes previously and temporarily
discretized. These attributes play the role of a class
attribute one after another. Finally, the smallest inter-
vals are merged. For supervised learning problems, a
paving of the representation space can be done by
cutting each continuous attribute into intervals. The
discretization process consists in merging the border-
ing intervals in which the data distribution is the same
(Bay, 2001). Nevertheless, even if this strategy can
introduce the class attribute in the discretization pro-
cess, it cannot give a particular role to the class at-
tribute and can induce the discretization to
nonimpressive results in the predictive model.
Multivariate Supervised Discretization
When the learning problem is supervised and the in-
stance labels are scattered in the representation space
with interactions between the continuous predictive
attributes (as presented in Figure 4), the methods pre-
viously seen will not give satisfactory results.
HyperCluster Finder is a method that will fix this
problem by combining the advantages of the supervised
and multivariate approaches (Muhlenbach &
Rakotomalala, 2002). This method is based on clusters
constituted as sets of same-class instances that are
closed in the representation space. The clusters are
identified on a multivariate and supervised way: First, a
neighborhood graph is built by using all predictive at-
tributes to determine which instances are close to others;
second, the edges connecting two instances belonging
399
TEAM LinG

to different classes are cut on the graph to constitute the
clusters; third, the minimal and maximal values of each
relevant cluster are used as cut-points on each predictive
attribute. The intervals found by this method have the
characteristic to be pure on a pavement of the whole
representation space, even if the purity is not guaranteed
for an independent attribute. The combination of all
predictive attribute intervals is what will provide pure
areas in the representation space.
FUTURE TRENDS
Today, the discretization field is well studied in the
supervised and unsupervised cases for a univariate pro-
cess. However, there is little work in the multivariate
case. A related problem exists in the feature selection
domain, which needs to be combined with the aforemen-
tioned multivariate case. This should bring improved
and more pertinent progress. It is virtually certain that
better results can be obtained for a multivariate
discretization if all attributes of the representation
space are relevant for the learning problem.
CONCLUSION
In a data-mining task, for a supervised or unsupervised
learning problem, the discretization turned out to be an
essential preprocessing step on which the performance
of the learning algorithm that uses discretized attributes
will depend.
Many methods, supervised or not, multivariate or
not, exist to perform this pretreatment, more or less
adapted to a given dataset and learning problem. Further-
more, a supervised discretization can also be applied in
a regression problem when the attribute to be predicted
is continuous (Ludl & Widmer, 2000b). The choice of a
particular discretization method depends on (a) its algo-
rithmic complexity (complex algorithms will take more
computation time and will be unsuited to very large
datasets), (b) its efficiency (the simple unsupervised
univariate discretization methods are inappropriate to
complex learning problems), and (c) its appropriate
combination with the learning method using the
discretized attributes (a supervised discretization is
better adapted to the supervised learning problem). For
the last point, it is also possible to significantly improve
the performance of the learning method by choosing an
appropriate discretization, for instance, a fuzzy
discretization for the nave Bayes algorithm (Yang &
Webb, 2002). Nevertheless, it is unnecessary to employ a
sophisticated discretization method if the learning method
400
Discretization of Continuous Attributes
does not benefit from the discretized attributes
(Muhlenbach & Rakotomalala, 2002).
ACKNOWLEDGMENT
Edited with the aid of Christopher Yukna.
REFERENCES
Bay, S. D. (2001). Multivariate discretization for set
mining. Knowledge and Information Systems, 3(4), 491-
512.
Chmielewski, M. R., & Grzymala-Busse, J. W. (1994).
Global discretization of continuous attributes as pre-
processing for machine learning. Proceedings of the
Third International Workshop on Rough Sets and Soft
Computing (pp. 294-301).
Divina, F., Keijzer, M., & Marchiori, E. (2003). A method
for handling numerical attributes in GA-based inductive
concept learners. Proceedings of the Genetic and Evolu-
tionary Computation Conference (pp. 898-908).
Dougherty, K., Kohavi, R., & Sahami, M. (1995). Super-
vised and unsupervised discretization of continuous
features. Proceedings of the 12th International Con-
ference on Machine Learning (pp. 194-202).
Fayyad, U. M., & Irani, K. B. (1993). The attribute selection
problem in decision tree generation. Proceedings of the
13th International Joint Conference on Artificial Intel-
ligence (pp. 1022-1027).
Fisher, W. D. (1958). On grouping for maximum homo-
geneity. Journal of the American Statistical Society, 53,
789-798.
Frank, E., & Witten, I. (1999). Making better use of global
discretization. Proceedings of the 16th International
Conference on Machine Learning (pp. 115-123).
Friedman N., & Goldszmidt, M. (1996). Discretization of
continuous attributes while learning Bayesian networks
from mixed data. Proceedings of the 13th International
Conference on Machine Learning (pp. 157-165).
Grzymala-Busse, J. W., & Stefanowski, J. (2001). Three
discretization methods for rule induction. International
Journal of Intelligent Systems, 16, 29-38.
Hsu, H., Huang, H., & Wong, T. (2003). Implication of
dirichlet assumption for discretization of continuous
variables in nave Bayes classifiers. Machine Learning,
53(3), 235-263.
TEAM LinG
Discretization of Continuous Attributes
Kurgan, L., & Krysztof, J. (2004). CAIM discretization
algorithm. IEEE Transactions on Knowledge and Data
Engineering, 16(2), 145-153.
Liu, H., Hussain, F., Tan, C., & Dash, M. (2002).
Discretization: An enabling technique. Data Mining
and Knowledge Discovery, 6(4), 393-423.
Ludl, M., & Widmer, G. (2000a). Relative unsupervised
discretization for association rule mining. Principles
of the Fourth European Conference on Data Mining
and Knowledge Discovery (pp. 148-158).
Ludl, M., & Widmer, G. (2000b). Relative unsupervised
discretization for regression problem. Proceedings of
the 11th European Conference on Machine Learning
(pp. 246-253).
Macskassy, S. A., Hirsh, H., Banerjee, A., & Dayanik, A.
A. (2001). Using text classifiers for numerical classifi-
cation. Proceedings of the 17th International Joint
Conference on Artificial Intelligence (pp. 885-890).
Mittal, A., & Cheong, L. (2002). Employing discrete Bayes
error rate for discretization and feature selection tasks.
Proceedings of the First IEEE International Conference
on Data Mining (pp. 298-305).
Monti, S., & Cooper, G. F. (1999). A latent variable model
for multivariate discretization. Proceedings of the Sev-
enth International Workshop on Artificial Intelligence
and Statistics
Muhlenbach, F., & Rakotomalala, R. (2002). Multivariate
supervised discretization: A neighborhood graph ap-
proach. Proceedings of the First IEEE International
Conference on Data Mining (pp. 314-321).
Potzelberger, K., & Felsenstein, K. (1993). On the fisher
information of discretized data. Journal of Statistical
Computation and Simulation, 46(3-4), 125-144.
Teicher, H. (1963). Identifiability of finite mixtures. Ann.
Math. Statist., 34, 1265-1269.
Yakowitz, S. J., & Spragins, J. D. (1968). On the identifiabil-
ity of finite mixtures. Ann. Math. Statist., 39, 209-214.
Yang, Y., & Webb, G. (2002). Non-disjoint discretization
for nave Bayes classifiers. Proceedings of the 19th
International Conference on Machine Learning (pp.
666-673).
Yang, Y., & Webb, G. (2003). On why discretization works
for nave Bayes classifiers. Proceedings of the 16th Aus-
tralian Joint Conference on Artificial Intelligence (pp.
440-452).
Zighed, D., Rakotomalala, R., & Feschet, F. (1997). Optimal
multiple intervals discretization of continuous attributes ,
for supervised learning. Proceedings of the Third Inter-
national Conference on Knowledge Discovery in Data-
bases (pp. 295-298).
KEY TERMS
Cut-Points: A cut-point (or split-point) is a value
that divides an attribute into intervals. A cut-point has to
be included in the range of the continuous attribute to
discretize. A discretization process can produce none
or several cut-points.
Discrete/Continuous Attributes: An attribute is a
quantity describing an example (or instance); its domain
is defined by the attribute type, which denotes the values
taken by an attribute. An attribute can be discrete (or
categorical, indeed symbolic) when the number of val-
ues is finite. A continuous attribute corresponds to real
numerical values (for instance, a measurement). The
discretization process transforms an attribute from con-
tinuous to discrete.
Instances: An instance is an example (or record) of
the dataset; it is often a row of the data table. Instances
of a dataset are usually seen as a sample of the whole
population (the universe). An instance is described by
its attribute values, which can be continuous or discrete.
Number of Intervals: The number of intervals cor-
responds to the different values of a discrete attribute
resulting from the discretization process. The number
of intervals is equal to the number of cut-points plus 1.
The minimum number of intervals of an attribute is equal
to 1, and the maximum number of intervals is equal to
the number of instances.
Representation Space: The representation space is
formed with all the attributes of a learning problem. In
the supervised learning, it consists of the representation
of the labeled instances in a multidimensional space,
where all predictive attributes play the role of a dimen-
sion.
Supervised/Unsupervised: A supervised learning
algorithm searches a functional link between a class-
attribute (or dependent attribute or attribute to be pre-
dicted) and predictive attributes (the descriptors). The
supervised learning process aims to produce a predictive
model that is as accurate as possible. In an unsupervised
learning process, all attributes play the same role; the
unsupervised learning method tries to group instances in
clusters, where instances in the same cluster are similar,
and instances in different clusters are dissimilar.
401
TEAM LinG
 Discretization of Continuous Attributes

Univariate/Multivariate: A univariate (or monothetic)

method processes a particular attribute independently of
the others. A multivariate (or polythetic) method pro-
cesses all attributes of the representation space, so it can
fix some problems related to the interactions among the
attributes.

402

TEAM LinG

Distributed Association Rule Mining
Mafruz Zaman Ashrafi
Monash University, Australia
David Taniar
Monash University, Australia
Kate A. Smith
Monash University, Australia
INTRODUCTION
Data mining is an iterative and interactive process that
explores and analyzes voluminous digital data to dis-
cover valid, novel, and meaningful patterns (Mohammed,
1999). Since digital data may have terabytes of records,
data mining techniques aim to find patterns using
computationally efficient techniques. It is related to a
subarea of statistics called exploratory data analysis.
During the past decade, data mining techniques have
been used in various business, government, and scien-
tific applications.
Association rule mining (Agrawal, Imielinsky & Sawmi,
1993) is one of the most studied fields in the data-mining
domain. The key strength of association mining is com-
pleteness. It has the ability to discover all associations
within a given dataset. Two important constraints of
association rule mining are support and confidence
(Agrawal & Srikant, 1994). These constraints are used to
measure the interestingness of a rule. The motivation of
association rule mining comes from market-basket analy-
sis that aims to discover customer purchase behavior.
However, its applications are not limited only to market-
basket analysis; rather, they are used in other applica-
tions, such as network intrusion detection, credit card
fraud detection, and so forth.
The widespread use of computers and the advances in
network technologies have enabled modern organiza-
tions to distribute their computing resources among
different sites. Various business applications used by
such organizations normally store their day-to-day data
in each respective site. Data of such organizations in-
creases in size everyday. Discovering useful patterns
from such organizations using a centralized data mining
approach is not always feasible, because merging datasets
from different sites into a centralized site incurs large
network communication costs (Ashrafi, David & Kate,
2004). Furthermore, data from these organizations are
not only distributed over various locations, but are also
fragmented vertically. Therefore, it becomes more dif-
ficult, if not impossible, to combine them in a central
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
403
,
location. Therefore, Distributed Association Rule Min-
ing (DARM) emerges as an active subarea of data-
mining research.
Consider the following example. A supermarket may
have several data centers spread over various regions
across the country. Each of these centers may have
gigabytes of data. In order to find customer purchase
behavior from these datasets, one can employ an asso-
ciation rule mining algorithm in one of the regional data
centers. However, employing a mining algorithm to a
particular data center will not allow us to obtain all the
potential patterns, because customer purchase patterns
of one region will vary from the others. So, in order to
achieve all potential patterns, we rely on some kind of
distributed association rule mining algorithm, which
can incorporate all data centers.
Distributed systems, by nature, require communica-
tion. Since distributed association rule mining algo-
rithms generate rules from different datasets spread
over various geographical sites, they consequently re-
quire external communications in every step of the
process (Ashrafi, David & Kate, 2004; Assaf & Ron, 2002;
Cheung, Ng, Fu & Fu, 1996). As a result, DARM algo-
rithms aim to reduce communication costs in such a way
that the total cost of generating global association rules
must be less than the cost of combining datasets of all
participating sites into a centralized site.
BACKGROUND
DARM aims to discover rules from different datasets
that are distributed across multiple sites and intercon-
nected by a communication network. It tries to avoid the
communication cost of combining datasets into a cen-
tralized site, which requires large amounts of network
communication. It offers a new technique to discover
knowledge or patterns from such loosely coupled dis-
tributed datasets and produces global rule models by
using minimal network communication. Figure 1 illus-
trates a typical DARM framework. It shows three par-
TEAM LinG
 Distributed Association Rule Mining

Figure 1. A distributed data mining framework CHALLENGES OF DARM

Final All of the DARM algorithms are based on sequential

Models
association mining algorithms. Therefore, they inherit
Global Model Communication Channel
all drawbacks of sequential association mining. How-
Generation
ever, DARM not only deals with the drawbacks of it but
also considers other issues related to distributed com-
Local
Model
Local
Model
Local
Model
puting. For example, each site may have different plat-
forms and datasets, and each of those datasets may have
Algorithm Algorithm Algorithm different schemas. In the following paragraphs, we dis-
cuss a few of them.

Frequent Itemset Enumeration

Data
Repository
Data
Repository
Repository
Site 1 Site 2
ticipating sites, where each site generates local models
from its respective data repository and exchanges local
models with other sites in order to generate global
models.
Typically, rules generated by DARM algorithms are
considered interesting, if they satisfy both minimum
global support and confidence threshold. To find inter-
esting global rules, DARM generally has two distinct
tasks: (i) global support count, and (ii) global rules
generation.
Let D be a virtual transaction dataset comprised of
D1, D2, D3 D m geographically distributed datasets;
let n be the number of items and I be the set of items such
that I = {a1, a 2, a3, an}, where ai n. Suppose N is
the total number of transactions and T = {t1, t2, t3,
t N} is the sequence of transaction, such that t i D. The
support of each element of I is the number of transac-
tions in D containing I and for a given itemset A I; we
can define its support as follows:
A ti
Support ( A) = (1)
N performs better when support threshold is high. On the
Itemset A is frequent if and only if Support(A)
minsup, where minsup is a user-defined global support
threshold. Once the algorithm discovers all global fre-
quent itemsets, each site generates global rules that
have user-specified confidence. It uses frequent itemsets
to find the confidence of a rule R1 and can be calculated
by using the following formula:
Support ( F1 F2 )
Confidence( R) = (2)
Support ( F1 )
Data
Frequent itemset enumeration is one of the main associa-
Site 3 tion rule mining tasks (Agrawal & Srikant, 1993; Zaki,
2000; Jiawei, Jian & Yiwen, 2000). Association rules are
generated from frequent itemsets. However, enumerating
all frequent itemsets is computationally expansive. For
example, if a transaction of a database contains 30 items,
one can generate up to 230 itemsets. To mitigate the enu-
meration problem, we found two basic search approaches
in the data-mining literature. The first approach uses
breadth-first searching techniques that search through the
iterate dataset by generating the candidate itemsets. It
works efficiently when user-specified support threshold is
high. The second approach uses depth-first searching
techniques to enumerate frequent itemsets. This search
technique performs better when user-specified support
threshold is low, or if the dataset is dense (i.e., items
frequently occur in transactions). For example, Eclat (Zaki,
2000) determines the support of k-itemsets by intersecting
the tidlists (Transaction ID) of the lexicographically first
two (k-1) length subsets that share a common prefix.
However, this approach may run out of main memory when
there are large numbers of transactions.
However, DARM datasets are spread over various
sites. Due to this, it cannot take full advantage of those
searching techniques. For example, breath-first search
other hand, in DARM, the candidate itemsets are gener-
ated by combining frequent items of all datasets; hence,
it enumerates those itemsets that are not frequent in a
particular site. As a result, DARM cannot utilize the
advantage of breath-first techniques when user-speci-
fied support threshold is high. In contrast, if a depth-
first search technique is employed in DARM, then it
needs large amounts of network communication. There-
fore, without very fast network connection, depth-first
search is not feasible in DARM.

404

TEAM LinG
Distributed Association Rule Mining
Communication
A fundamental challenge for DARM is to develop mining
techniques without communicating data from various
sites unnecessarily. Since, in DARM, each site shares its
frequent itemsets with other sites to generate unambigu-
ous association rules, each step of DARM requires com-
munication. The cost of communication increases pro-
portionally to the size of candidate itemsets. For ex-
ample, suppose there are three sites, such as S1, S2, and
S3, involved in distributed association mining. Suppose
that after the second pass, site S1 has candidate 2-
itemsets equal to {AB, AC, BC, BD}, S2 = {AB, AD, BC,
BD}, and S3 = {AC, AD, BC, BD}. To generate global
frequent 2-itemsets, each site sends its respective candi-
date 2-itemsets to other sites and receives candidate 2-
itemsets from others. If we calculate the total number of
candidate 2-itemsets that each site receives, it is equal to
8. But, if each site increases the number of candidate 2-
itemsets by 1, each site will receive 10 candidate 2-
itemsets, and, subsequently, this increases communica-
tion cost. This cost will further increase when the number
of sites is increased. Due to this reason, message optimi-
zation becomes an integral part of DARM algorithms.
The basic message exchange technique (Agrawal &
Shafer, 1996) incurs massive communication costs, es-
pecially when the number of local frequent itemsets is
large. To reduce message exchange cost, we found sev-
eral message optimization techniques (Ashrafi et al.,
2004; Assaf & Ron, 2002; Cheung et al., 1996). Each of
these optimizations focuses on the message exchange
size and tries to reduce it in such a way that the overall
communication cost is less than the cost of merging all
datasets into a single site. However, those optimization
techniques are based on some assumptions and, there-
fore, are not suitable in many situations. For example,
DMA (Cheung et al., 1996) assumes the number of
disjoint itemsets among different sites is high. But this
is only achievable when the different participating sites
have vertical fragmented datasets. To mitigate these
problems, we found another framework (Ashrafi et al.,
2004) in data-mining literature. It partitions the differ-
ent participating sites into two groups, such as sender
and receiver, and uses itemsets history to reduce mes-
sage exchange size. Each sender site sends its local
frequent itemsets to a particular receiver site. When
receiver sites receive all local frequent itemsets, it
generate global frequent itemsets for that iteration and
sends back those itemsets to all sender sites
Privacy
Association rule-mining algorithms discover patterns
from datasets based on some statistical measurements.
However those statistical measurements have signifi-
cant meaning and may breach privacy (Evfimievski, ,
Srikant, Agrawal & Gehrke, 2002; Rizvi & Haritsa, 2002).
Therefore, privacy becomes a key issue of association
rule mining. Distributed association rule-mining algo-
rithms discover association rules beyond the organiza-
tion boundary. For that reason, the chances of breaching
privacy in distributed association mining are higher than
the centralized association mining (Vaidya & Clifton,
2002; Ashrafi, David & Kate, 2003), because distributed
association mining accomplishes the final rule model by
combining various local patterns. For example, there are
three sites, S1, S2, and S3, each of which has datasets
DS1, DS2, and DS3. Suppose A and B are two items
having global support threshold; in order to find rule
AB or BA, we need to aggregate the local support of
itemsets AB from all participating sites (i.e., site S1, S2,
and S3). When we do such aggregation, all sites learn the
exact support count of other sites. However, participat-
ing sites generally are reluctant to disclose the exact
support of itemset AB to other sites, because support
counts of an itemset has a statistical meaning, and this
may thread data privacy.
For the above-mentioned reason, we need secure
multi-party computation solutions to maintain privacy
of distributed association mining (Kantercioglu & Clifton,
2002). The goal of secure multi-party computation (SMC)
in distributed association rule mining is to find global
support of all itemsets using a function where multiple
parties hold their local support counts, and, at the end, all
parties know about the global support of all itemsets but
nothing more. And finally, each participating site uses this
global support for rule generation.
Partition
DARM deals with different possibilities of data distri-
bution. Different sites may contain horizontally or
vertically partitioned data. In horizontal partition, the
dataset of each participating site shares a common set
of attributes. However, in vertical partition, the datasets
of different sites may have different attributes. Figure
2 illustrates horizontal and vertical partition dataset in
a distributed context. Generating rules by using DARM
becomes more difficult when different participating
sites have vertically partitioned datasets. For example,
if a participating site has a subset of items of other
sites, then that site may have only a few frequent
itemsets and may finish the process earlier than others.
However, that site cannot move to the next iteration,
because it needs to wait for other sites to generate
global frequent itemsets of that iteration.
Furthermore, the problem becomes more difficult
when the dataset of each site does not have the same
405
TEAM LinG
 Distributed Association Rule Mining

Figure 2. Distributed (a) homogeneous (b) heterogeneous dataset

TID X-1 X-2 X-3 TID X-1 X-2 X-3 TID X-1 X-2 TID X-1 X-3 TID X-1 X-4
1 1.1 2.2 3.1 8 1.5 2.1 3.1 1 1.1 2.2 1 1.5 3.1 1 1.5 4.1
2 1.1 2.2 3.1 9 1.6 2.2 3.2 2 1.1 2.2 2 1.6 3.1 2 1.6 4.2
3 1.3 2.3 3.3 10 1.3 2.1 3.3 3 1.3 2.3 3 1.3 3.3 3 1.3 4.1
4 1.2 2.5 3.2 11 1.4 2.4 3.4 4 1.2 2.5 4 1.4 3.2 4 1.4 4.4
5 1.7 2.5 3.3 12 1.5 2.4 3.5 5 1.7 2.5 5 1.5 3.3 5 1.5 4.4
6 1.6 2.6 3.6 13 1.6 2.6 3.6 6 1.6 2.6 6 1.6 3.6 6 1.6 4.5
7 1.7 2.7 3.7 14 1.7 2.7 3.7 7 1.7 2.7 7 1.7 3.7 7 1.7 4.5
Site A Site B Site A Site B Site C

(a) (b)

hierarchical taxonomy. If a different taxonomy level exists
in datasets of different sites, as shown in Figure 3, it
becomes very difficult to maintain the accuracy of global
models.
ises to generate association rules with minimal communi-
cation cost, but it also utilizes the resources distributed
among different sites efficiently. However, the accept-
ability of DARM depends to a great extent on the issues
discussed in this article.

FUTURE TRENDS
The DARM algorithms often consider the datasets of
various sites as a single virtual table. On the other hand,
such assumptions become incorrect when DARM uses
different datasets that are not from the same domain.
Enumerating rules using DARM algorithms on such
datasets may cause a discrepancy, if we assume that
semantic meanings of those datasets are the same. The
future DARM algorithms will investigate how such
datasets can be used to find meaningful rules without
increasing the communication cost.
CONCLUSION
REFERENCES
Agrawal, R., Imielinsky, T., & Sawmi, A.N. (1993). Mining
association rules between sets of items in large databases.
Proceedings of the ACM SIGMOD International Confer-
ence on Management of Data, Washington D.C.
Agrawal, R., & Shafer, J.C. (1996). Parallel mining of
association rules. IEEE Transactions on Knowledge
and Data Engineering, 8(6), 962-969.
Agrawal, R., & Srikant, R. (1994). Fast algorithms for
mining association rules in large database. Proceedings
of the International Conference on Very Large Data-
bases, Santiago de Chile, Chile.

The widespread use of computers and the advances in
database technology have provided a large volume of
data distributed among various sites. The explosive
growth of data in databases has generated an urgent need
for efficient DARM to discover useful information and
knowledge. Therefore, DARM becomes one of the ac-
tive subareas of data-mining research. It not only prom-
Figure 3. A generalization scenario
Ashrafi, M.Z., Taniar, D., & Smith, K.A. (2003). To-
wards privacy preserving distributed association rule
mining. Proceedings of the Distributed Computing,
Lecture Notes in Computer Science, IWDC03, Calcutta,
India.
Ashrafi, M.Z., Taniar, D., & Smith, K.A. (2004). Reduc-
ing communication cost in privacy preserving distrib-
uted association rule mining. Proceedings of the Data-
base Systems for Advanced Applications, DASFAA04,
Jeju Island, Korea.

Beverage Level 3
Ashrafi, M.Z., Taniar, D., & Smith, K.A. (2004). ODAM:
An optimized distributed association rule mining algo-
rithm. IEEE Distributed Systems Online, IEEE.
Coffee Tea Soft drink Level 2 Assaf, S., & Ron, W. (2002). Communication-efficient
distributed mining of association rules. Proceedings of
the ACM SIGMOD International Conference on Man-
Ice Tea Coke Pepsi Level 1 agement of Data, California.

406

TEAM LinG
Distributed Association Rule Mining
Cheung, D.W., Ng, V.T., Fu, A.W., & Fu, Y. (1996a).
Efficient mining of association rules in distributed data-
bases. IEEE Transactions on Knowledge and Data En-
gineering, 8(6), 911-922.
Cheung, D.W., Ng, V.T., Fu, A.W., & Fu, Y. (1996b). A fast
distributed algorithm for mining association rules. Pro-
ceedings of the International Conference on Parallel
and Distributed Information Systems, Florida.
Evfimievski, A., Srikant, R., Agrawal, R., & Gehrke, J.
(2002). Privacy preserving mining association rules.
Proceedings of the ACM SIGKDD International Con-
ference on Knowledge Discovery and Data Mining,
Edmonton, Canada.
Jiawei, H., Jian, P., & Yiwen, Y. (2000). Mining fre-
quent patterns without candidate generation. Proceed-
ings of the ACM SIGMOD International Conference
on Management of Data, Dallas, Texas.
Kantercioglu, M., & Clifton, C. (2002). Privacy pre-
serving distributed mining of association rules on hori-
zontal partitioned data. Proceedings of the ACM
SIGMOD Workshop of Research Issues in Data Min-
ing and Knowledge Discovery DMKD, Edmonton, Canada.
Rizvi, S.J., & Haritsa, J.R. (2002). Maintaining data privacy
in association rule mining. Proceedings of the Interna-
tional Conference on Very Large Databases, Hong Kong,
China.
Vaidya, J., & Clifton, C. (2002). Privacy preserving asso-
ciation rule mining in vertically partitioned data. Proceed-
ings of the ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, Edmonton,
Canada.
Zaki, M.J. (1999). Parallel and distributed association
mining: A survey. IEEE Concurrency, 7(4), 14-25. ,
Zaki, M.J. (2000). Scalable algorithms for association
mining. IEEE Transactions on Knowledge and Data
Engineering, 12(2), 372-390.
Zaki, M.J., & Ya, P. (2002). Introduction: Recent develop-
ments in parallel and distributed data mining. Journal of
Distributed and Parallel Databases, 11(2), 123-127.
KEY TERMS
DARM: Distributed Association Rule Mining.
Data Center: A centralized repository for the stor-
age and management of information, organized for a
particular area or body of knowledge.
Frequent Itemset: A set of itemsets that have the
user specified support threshold.
Network Intrusion Detection: A system that de-
tects inappropriate, incorrect, or anomalous activity in
the private network.
SMC: SMC computes a function f (x1, x2 , x3 xn)
that holds inputs from several parties, and, at the end, all
parties know about the result of the function f (x1, x2 , x3
xn) and nothing else.
Taxonomy: A classification based on a pre-deter-
mined system that is used to provide a conceptual frame-
work for discussion, analysis, or information retrieval.
407
TEAM LinG
408
Distributed Data Management of Daily Car
Pooling Problems
Roberto Wolfler Calvo
Universit de Technologie de Troyes, France
Fabio de Luigi
University of Ferrara, Italy
Palle Haastrup
European Commission, Italy
Vittorio Maniezzo
University of Bologna, Italy
INTRODUCTION
The increased human mobility, combined with high use of
private cars, increases the load on the environment and
raises issues about the quality of life. The use of private
cars lends to high levels of air pollution in cities, parking
problems, noise pollution, congestion, and the resulting
low transfer velocity (and, thus, inefficiency in the use of
public resources). Public transportation service is often
incapable of effectively servicing non-urban areas, where
cost-effective transportation systems cannot be set up.
Based on investigations during the last years, problems
related to traffic have been among those most commonly
mentioned as distressing, while public transportation
systems inherently are incapable of facing the different
transportation needs arising in modern societies. A solu-
tion to the problem of the increased passenger and freight
transportation demand could be obtained by increasing
both the efficiency and the quality of public transporta-
tion systems, and by the development of systems that
could provide alternative solutions in terms of flexibility
and costs between the public and private ones. This is the
rationale behind so-called Innovative Transport Systems
(ITS) (Colorni et al., 1999), like car pooling, car sharing,
dial-a-ride, park-and-ride, card car, park pricing, and road
pricing, which are characterized by the exploitation of
innovative organizational elements and by a large flexibil-
ity in their management (e.g., traffic restrictions and fares
can vary according with the time of day).
Specifically, car pooling is a collective transportation
system based on the idea that sets of car owners having
the same travel destination can share their vehicles (pri-
vate cars), with the objective of reducing the number of
cars on the road. Until now, these systems have had a
limited use due to lack of efficient information processing
and communication support. This study presents an
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
integrated system for the organization of a car-pooling
service and reports about a real-world case study.
BACKGROUND
Car pooling can be operated in two main ways: Daily Car
Pooling Problem (DCPP) or Long-Term Car Pooling Prob-
lem (LCPP). In the case of DCPP (Baldacci et al., 2004;
Mingozzi et al., 2004), each day a number of users (here-
after called servers) declare their availability for picking
up and later bringing back colleagues (hereafter clients)
on that particular day.
The problem is to assign clients to servers and to
identify the routes to be driven by the servers in order to
minimize service costs and a penalty due to unassigned
clients, subject to user time window and car capacity
constraints. In the case of LCPP, each user is available
both as a server and as a client, and the objective is to
define crews or user pools, where each user, in turn, on
different days, will pick up the remaining pool members
(Hildmann, 2001; Maniezzo et al., 2004). The objective
here becomes that of maximizing pool sizes and minimizing
the total distance traveled by all users when acting as
servers, again subject to car capacity and time window
constraints.
Car pooling is most effective for daily commuters. It is
an old idea but scarcely applied, at least in Europe, for
many reasons. First of all, it is necessary to collect a lot of
information about service users and to make them avail-
able to the service planner; it can be difficult to cluster
people in such a way that everybody feels satisfied.
Ultimately, it is important that all participants perceive a
clear personal gain in order to continue using the service.
Moreover, past experience has shown already that the
effectiveness and competitiveness of public transporta-
TEAM LinG
Distributed Data Management of Daily Car Pooling Problems
tion, compared to private transportation, can be improved
through an integrated process, based on data manage-
ment facilities. Usually, potential customers of a new
transportation system face serious problems of knowl-
edge retrieval, which is the reason for devoting substan-
tial efforts to providing clients with an easy and powerful
tool to find information. Current information systems give
the possibility of real-time data management and include
the possibility to react to unscheduled events, as they can
be reached from almost anywhere.
Architectures for information browsing, like the World
Wide Web (WWW), are an easy and powerful means
through which to deploy databases and, thus, represent
obvious options for setting up services, such as the one
advocated in this work. The WWW is useful for providing
access to central data storage, for collecting remote data,
and for allowing GIS and optimization software to interact.
All these elements have an important impact on the
implementation of new transport services (Ridematching,
2004). Based on the way they use the WWW and on the
services they provide, car-pooling systems range be-
tween two extremes (Carpoolmatch, 2004; Carpooltool,
2004; Ridepro, 2004; SAC, 2004, etc.), one where there is
a WWW site collecting information about trips, which are
open to every registered user, and another one where the
users of the system are a restricted group and are granted
more functionalities.
A main issue for the first type is to guarantee the
reliability of the information. Their interface often is
designed mainly to help set service-related geographical
information (customer delivery points, customers pick up
points, paths). Such systems only rarely will suggest a
matching between clients and servers but operate as a
post-it wall, where users can consult or leave information
about travel routes.
As for the latter type, the idea is normally to set up a
car-pooling service among the users, usually employees
of the same organization. This system is more structured.
Moreover, the spontaneous user matching often is sub-
stituted by a solution found by means of an algorithmic
approach. An example of this type of system is reported
in Dailey, et al. (1999) or in Lexington-Fayette County
(2004). These systems use the WWW and react to unex-
pected events with e-mail messages.
MAIN THRUST
This article describes an integrated ICT system that
supports the management of a car-pooling service in a
real-world prototypical setting. An approach similar to
Dailey, et al. (1999) is suggested, and a complete system
for supporting the operation of a car-pooling case as a
prototype for a real-life application is described. The
service is supported by a database of potential users (e.g.,
employees of a company) that daily commute from their ,
houses to their workplace. A subset of them offers seats
in their cars. Moreover, they specify the departure time
(when they leave their house) and the mandatory arrival
time at the office. The employees that offer seats in their
cars are called servers. The employees asking for a lift are
called clients. The set of servers and the set of clients need
to be redefined once a day. The effectiveness of the
proposed system is related strictly to the architecture and
the techniques used to manage information.
The objective of this research is to prove that, at least
in a particular site (the Joint Research Center [JRC] of the
European Commission located in Ispra, northern Italy), it
could be possible to reduce the number of transport
vehicles without significantly changing either the number
of commuters or their comfort-level. The users of the
system are commuters normally using their own cars for
traveling between home and a workplace poorly served by
public transportation.
System Architecture
The architecture of the system developed for this problem
is shown in Figure 1 (interested readers can find a com-
plete description of the whole system in Wolfler et al.,
2004). The system consists of the following five main
modules:
OPT: An optimization module that generates a
feasible solution using an algorithm that defines the
paths for the servers. The algorithm makes use of a
heuristic approach and is used to assign the clients
to the servers and to define the path for each server.
Each path minimizes the travel time, maximizes the
number of requests picked up, and satisfies the time
and capacity constraints.
{M-, S-, W-}CAR: Three modules that permit to
receive, decrypt, and send SMS (SCAR), EMAIL
(MCAR) and Web pages (WCAR) to the users,
respectively. The module Sms Car Pooling (SCAR)
allows the server to send and receive Sms messages.
The module Mail Car Pooling (MCAR) supports e-
mail communication and uses POP3 and SMTP as
protocols. The module (WCAR) is the gateway for
the Web interaction. All modules filter the customers
access, allowing the entitled user to insert new data,
to query, and to modify the database (e.g., the
departure and the arrival time desired) and to access
service data.
GUI: A graphical user interface based on ESRI
ArcView. It generates a view (a digital map) of the
current problem instance and provides all relevant
data management.
409
TEAM LinG
 Distributed Data Management of Daily Car Pooling Problems
Figure 1. The architecture of the system
SMS SC AR
Requests
E-mail
Email MCAR
Offers
W EB
W CAR
Employees
GUI Solutions
OPT
Geography
The system collects and uses data of two different
types: geographic and alphanumeric. The geographic da-
tabase contains the maps of the region of interest with a
detailed road network, the geocoded client pickup sites,
and all the geographic information needed to generate a
GIS output on the Web. The alphanumeric data repository,
maintained by a relational database, contains information
about the employees and a representation of the road
network of the area where the car-pooling service is active.
Other data, input by the users, are related strictly to the
daily situationdetailed information about the service,
whether a user is a server or a client, the departure time from
home and the maximal acceptable arrival time at work, the
number of available seats in the cars offered by the servers,
and the maximum accepted delay, which is the parameter
used to specify how far out of the shortest way a server is
willing to drive in order to pick up colleagues. The users
are permitted to consult, modify, or delete their own entries
in the database at any time.
The road network and all user-related data are pro-
cessed by the optimization module in order to define user
pools and car paths. The optimization algorithm can be
activated without immediate presence of anybody on some
regular schedule, searching a local database and present-
ing the results on the Web. All these actions are performed
on a periodic basis on the evening before each work day.
The system is designed to support two types of users:
the system administrator and the employee. The system
administrator must update the static databases (users,
maps) and guarantee all the functionalities. All other data
are entered, edited and deleted by the users through a
distributed GUI.
Communication Subsystems
The system supports three different communication chan-
nels: Web, SMS, and e-mails. The Web is the main interface
to connect a user to the system. The user, by means of a
standard GUI, is allowed to insert transportation requests
410
and offers; the system then generates Web pages con-
taining text and maps presenting the results of the opti-
mization algorithm. Both e-mail and SMS also can be used
for submitting requests and for receiving results or
variations of previously proposed results (following
real-time notifications). The clients know whether their
requests could be matched and the detail of their pickup,
while the servers simply receive an SMS notifying the
pool formation; travel details are separately sent via e-
mail. SMS also is used by the servers (via a service relay)
to inform clients of delays.
Operationally, a user sending an e-mail to the system
must insert the message string in the subject field, which
will be processed by a mail receiver agent and finally sent
to the parser. Economic and privacy considerations sug-
gested sending all SMS to an SMS engine, which is
interfaced with the system and which transfers the string
sent to the parser. Each time the system receives a
syntactically correct e-mail or SMS, it automatically sends
an acknowledgment SMS message.
While the system generates messages to be sent
directly to users, those that users generate, either e-mail
or SMS, must follow a rigidly structured format in order
to allow automatic processing. We implemented a single
parser for all these messages. The Web interface is
obviously more user-friendly. Each employee can specify
through an ASP page the set of days (possibly empty)
when he or she is willing to drive and the set of days when
he or she asks to be picked up.
The system then computes a matching of servers as
clients. The result is a set of routes starting from the
server houses, arriving at the workplace, and passing
through the set of client houses without violating the
time windows constraints and the car capacity con-
straints.
These routes are made available both to clients and
to server. A well-known GIS module, ArcView, loads the
route information given in alphanumeric format, after the
optimization. Then, it transforms this information in a set
of GIS views through a set of queries to its database. The
last step transforms the views into bitmaps displayed by
the Web server. Moreover, the system alerts clients and
servers in real time by means of SMS when any variation
of the scheduling happens.
Optimization Algorithms and
Implementation Details
One of the most interesting features of the approach
described in this article, with respect to the other systems
currently in use, is the set of algorithms used to obtain
a matching of clients and servers. Due to the NP-hard-
ness of the underlying problem (Varrentrapp et al., 2002)
and to the size of the instances to solve, a heuristic
TEAM LinG
Distributed Data Management of Daily Car Pooling Problems
approach was in order. In designing it, the efficiency was
the main parameters, ensuring relatively fast response
time also for larger instances. The matching obtained
minimizes the total travel length of all servers going to the
workplace and maximizes the number of clients serviced,
while meeting the operational constraints. In addition,
real-time services are supported; for example, the system
sends warnings to employees when delays occur.
The system was implemented at the Joint Research
Center of the European Commission, using standard com-
mercial software and developing extra modules in C++
using Microsoft Visual Studio C++ compiler. The data-
base used is Microsoft Access. The geographical data are
stored in shape files and used by an ArcView application
(release 3.2). The system administrator can access data
directly through MSAccess and ArcView, which are both
packages available and running in the same machine
where the car-pooling application resides. Through
Arcview, the system manager can start the optimization
module.
The Joint Research Center of the European Commis-
sion (JRC) is situated in the northwest of Italy not far from
Milan. It covers a big area of 2 km2, and its mission is to
carry out research useful to the European Commission.
The number of employees is about 2,000 people, divided
into three main classes: administrative, staff, and re-
searchers. They come from all around Europe and live in
the area surrounding the center. The wide geographical
area covered by the commuters is about 100 Km2. Since
this is a sparsely populated area, public transportation
(i.e., trains, buses, etc.) is definitely insufficient; thus,
private cars are necessary and used.
Module OPT, in particular, was tested on a set of real-
world problem instances derived from data provided by
the JRC. The instances used are the same as those de-
scribed in (Baldacci et al., 2004); they are derived from the
real-world instance defined by over 600 employees by
randomly selecting the desired number of clients and
servers.
Computational results show that the CPU time used by
our heuristic increases approximately linearly with the
problem dimension, and the number of unserviced re-
quests is a constant proportion of the total number of
employees. Moreover, comparing two instances with the
same dimension, but with different percentage of servers,
one can see that the CPU time increases when fewer
servers are available, as expected, while the total travel
time decreases, since fewer paths are driven.
FUTURE TRENDS
Car-pooling services as well as most other alternative
public transportation services are likely to become more
and more common in the near future. Already, several
software companies are marketing tailored solutions. ,
Therefore, it is easy to envisage that systems featuring
the services included in the prototype described in this
work will define their own market niche. This, however,
can happen only parallel to an increasing sensibility of
local governments, which alone can define incentive
policies for drivers who share their cars. These policies
will reflect features of the implemented systems. As shown
in this article, all needed technological infrastructure is
already available.
CONCLUSION
This article presents the essential elements of a prototypi-
cal deployment of ITC support to a car-pooling service for
a large-sized organization. The essential features that can
make its actual use possible are in the technological
infrastructure, essentially in the distributed data manage-
ment and accompanying optimization, which helps to
provide real-time response to user needs. The current
possibility of using any data access means as a system
interface (we used e-mails, SMS, and Web, but new smart
phones and more in general UMTS can provide other
access points) can ease system acceptance and end-user
interest. However, the ease of usage alone would be of
little interest, if the solutions proposed, in terms of driver
paths, were not of good quality. Current development of
optimization research, both in the areas of heuristic and
exact approaches, provide firm basis for designing sup-
port system that also can deal with the dimension of the
instances induced by large-sized organizations.
REFERENCES
Baldacci, R., Maniezzo, V., & Mingozzi, A. (2004). An exact
method for the car pooling problem based on Lagrangean
column generation. Operations Research, 52(3), 422-439.
Carpoolmatch. (2004). http://www.carpoolmatchnw.org/
Carpooltool. (2004). http://www.carpooltool.com/en/my/
Colorni, A., Cordone, R., Laniado, E., & Wolfler Calvo, R.
(1999). Innovation in transports: Planning and manage-
ment [in Italian]. In S. Pallottino, & A. Sciomachen (Eds.),
Scienze delle decisioni per i trasporti. Franco Angeli.
Cordeau, J.-F., & Laporte, G. (2003). The dial-a-ride prob-
lem (DARP): Variants, modeling issues and algorithms.
Quarterly Journal of the Belgian, French and Italian
Operations Research Societies, 1, 89-101.
411
TEAM LinG
 Distributed Data Management of Daily Car Pooling Problems
Dailey, D.J., Loseff, D., & Meyers, D. (1999). Seattle smart
traveler: Dynamic ridematching on the World Wide Web.
Transportation Research Part C, 7, 17-32.
Hildmann, H. (2001). An ants metaheuristic to solve car
pooling problems [masters thesis]. University of
Amsterdam, The Netherlands.
Lexington-Fayette County. (2004). Ride matching ser-
vices. Retrieved from http://www.lfucg.com/mobility/
rideshare.asp
Maniezzo, V. (2002). Decision support for location prob-
lems. Encyclopedia of Microcomputers, 8, 31-52. NY:
Marcel Dekker.
Maniezzo, V., Carbonaro, A., & Hidmann, H. (2004). An
ANTS heuristic for the long-term car pooling problem. In
G.C. Onwuboulu, & B.V. Babu (Eds.), New optimization
techniques in engineering (pp. 411-430).Heidelber, Ger-
many: Springer-Verlag.
Mattarelli, M., Maniezzo, V., & Haastrup, P. (1998). A
decision support system distributed on the Internet.
Journal of Decision Systems, 6(4), 353-368.
Ridematching. (2004). University of South Florida,
ridematching systems. Retrieved from http://
www.nctr.usf.edu/clearinghouse/ridematching.htm
Ridepro. (2004). http://www.ridepro.net/index.asp
SAC. (2004). San Antonio Colleges carpool matching
service. Retrieved from http://www.accd.edu/sac/
carpool/
Varrentrapp, K., Maniezzo, V., & Sttzle, T. (2002). The
long term car pooling problem: On the soundness of the
problem formulation and proof of NP-completeness. Tech-
nical Report AIDA-02-03. Darmstadt, Germany: Technical
University of Darmstadt.
Wolfler Calvo, R., de Luigi, F., Haastrup, P., & Maniezzo,
V.(2004). A distributed geographic information system for
the daily car pooling problem. Computers & Operations
Research, 31, 2263-2278.
412
KEY TERMS
Car Pooling: A collective transportation system based
on the shared use of private cars (vehicles) with the
objective of reducing the number of cars on the road.
Computational Problem: A relation between input
and output data, where input data are known (and corre-
spond to all possible different problem instances), and
output data are to be identified, but predicates or asser-
tions they must verify are given.
Data Mining: The application of analytical methods
and tools to data for the purpose of identifying patterns
and relationships, such as classification, prediction, es-
timation, or affinity grouping.
GIS: Geographic Information Systems; tools used to
gather, transform, manipulate, analyze, and produce in-
formation related to the surface of the Earth.
Heuristic Algorithms: Optimization algorithms that
do not guarantee to identify the optimal solution of the
problem they are applied to, but which usually provide
good quality solutions in an acceptable time.
NP-Hard Problems: Optimization problems for which
a solution can be verified in polynomial time, but no
polynomial solution algorithm is known, even though no
one so far has been able to demonstrate that none exists.
Optimization Problem: A computational problem for
which an objective function associates a merit figure with
each problem solution, and it is asked to identify a feasible
solution that minimizes or maximizes the objective function.
ENDNOTE
1
This article is an abridged and updated version of
the paper, A Distributed Geographic Information
System for the Daily Car Pooling Problem, pub-
lished as Wolfler et al., 2004.
TEAM LinG

Drawing Representative Samples from Large
Databases
Wen-Chi Hou
Southern Illinois University, USA
Hong Guo
Southern Illinois University, USA
Feng Yan
Williams Power, USA
Qiang Zhu
University of Michigan, USA
INTRODUCTION
Sampling has been used in areas like selectivity estima-
tion (Hou & Ozsoyoglu, 1991; Haas & Swami, 1992,
Jermaine, 2003; Lipton, Naughton & Schnerder, 1990; Wu,
Agrawal, & Abbadi, 2001), OLAP (Acharya, Gibbons, &
Poosala, 2000), clustering (Agrawal, Gehrke, Gunopulos,
& Raghavan, 1998; Palmer & Faloutsos, 2000), and spatial
data mining (Xu, Ester, Kriegel, & Sander, 1998). Due to its
importance, sampling has been incorporated into modern
database systems.
The uniform random sampling has been used in vari-
ous applications. However, it has also been criticized for
its uniform treatment of objects that have non-uniform
probability distributions. Consider the Gallup poll for a
Federal election as an example. The sample is constructed
by randomly selecting residences telephone numbers.
Unfortunately, the sample selected is not truly represen-
tative of the actual voters on the election. A major reason
is that statistics have shown that most voters between
ages 18 and 24 do not cast their ballots, while most senior
citizens go to the poll-booths on Election Day. Since
Gallups sample does not take this into account, the
survey could deviate substantially from the actual elec-
tion results.
Finding representative samples is also important for
many data mining tasks. For example, a carmaker may like
to add desirable features in its new luxury car model. Since
not all people are equally likely to buy the cars, only from
a representative sample of potential luxury car buyers can
most attractive features be revealed. Consider another
example in deriving association rules from market basket
data, recalling that the goal was to place items often
purchased together in near locations. While serving ordi-
nary customers, the store would like to pay some special
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
413
,
tribute to customers who are handicapped, pregnant,
elderly, and etcetera. A uniform sampling may not be able
to include enough such under-populated people. How-
ever, by giving higher inclusion probabilities to (the
transaction records of) these under-populated customers
in sampling, the special care can be reflected in the
association rules.
To find representative samples for populations with
non-uniform probability distributions, some remedies,
such as the density biased sampling (Palmer & Faloutsos,
2000) and the Acceptance/Rejection (AR) sampling (Olken,
1993), have been proposed. The density-biased sampling
is specifically designed for applications where the prob-
ability of a group of objects is inversely proportional to
its size. The AR sampling, based on the acceptance/
rejection approach (Rubinstein, 1981), aims for all prob-
ability distributions and is probably the most general
approach discussed in the database literature.
We are interested in finding a general, efficient, and
accurate sampling method applicable to all probability
distributions. In this research, we develop a Metropolis
sampling method, based on the Metropolis algorithm
(Metropolis, Rosenbluth, Rosenbluth, Teller, & Teller,
1953), to draw representative samples. As it will be clear,
the sample generated by this method is bona fide repre-
sentative.
BACKGROUND
Being a representative sample, it must satisfy some crite-
ria. First, the sample mean and variance must be good
estimates of the population mean and variance, respec-
tively, and converge to the latter when the sample size
increases. In addition, a selected sample must have a
TEAM LinG
 Drawing Representative Samples from Large Databases

similar distribution to that of the underlying population. k

In the following, we briefly describe the population mean 2 = (ri Nwi ) 2 /( Nwi ) , (4)
and variance and the Chi-square (Spiegel, 1991) test used i =1
to examine the similarity of distributions.
where ri is the number of sample objects drawn from the
Mean and Variance Estimation
wi is the probability
k
ith bin, i =1 ri = N is the sample size,
v
Let x be a d-dimensional vector representing a set of d
attributes that characterizes an object in a population, of the i th bin of the population, and Nwi is the expected
and the quantity of interest of the object, denoted as number of sample objects from that bin. A bin here refers
v to a designated group or range of values. The larger
(x ) . Our task is to calculate the mean and variance of
v v the 2 value, the greater is the discrepancy between the
(x ) of the population (relation). Let w( x ) 0 be the
v sample and the population distributions. Usually, a level
probability or weigh function of x . The population
v of significance is specified as the uncertainty of the test.
mean of (x ) , denoted by , is given by 2
If the value of 2 is less than 1 , we are about 1-
v v confident that the sample and population have similar
= r ( x ) w( x ) (1) 2
all x distributions: customarily, = 0.05 . The value of 1 is
also determined by the degree of freedom involved (i.e.,
k - 1 ).
The probability distribution is required to satisfy

v
rw( x ) = 1 . (2) MAIN THRUST
all x

Metropolis algorithm (Metropolis, Rosenbluth,

v Rosenbluth, Teller, & Teller, 1953) has been known as
For example, let x be a registered voter. Assuming the most successful and influential Monte Carlo Method.
there are only two candidates: a Republican candidate and Unlike its use in numerical calculations, we shall use it
v
a Democratic candidate, then we can let (x ) = 1 if the to construct representative samples. In addition, we will
v also incorporate techniques for finding the best start
voter x will cast a vote for the Republican candidate; and sampling point in the algorithm, which can greatly im-
v v
(x ) = 1 , otherwise. w(x ) is the weight of the registered prove the efficiency of the process.
r
voter x . If is positive, the Republican candidate is
Probability Distribution
predicted to win the election. Otherwise, the Democratic
candidate wins the election. v
The probability distribution w(x ) plays an important role
Another useful quantity is the population variance,
which is defined as in the Metropolis algorithm. Unfortunately, such informa-
tion is usually unknown or difficult to obtain due to
v v incompleteness or size of a population. However, the
2 = r( ( x ) ) 2 w( x ) . (3) relative probability distribution or non-normalized prob-
all x v
ability distribution, denoted by W (x ) , can often be ob-
tained from, for example, preliminary analysis, knowledge,
v statistics, and etcetera. Take the Gallup poll for example.
The variance specifies the variability of the (x )
While it may be difficult or impossible to assign a weight
values relative to . v
(i.e., w(x ) ) to each individual voter, it can be easily
known, for example, from Federal Election Commission,
Chi-Square Test that the relative probabilities for people to vote on the
v
Election Day (i.e., W (x ) ) are 18.5%, 38.7%, 56.5%, and
To compare the distributions of a sample and its popula- 61.5% for groups whose ages fall in 18-24, 25-44, 45-
tion, we perform the Chi-square test (Press, Teukolsky, 65, and 65+, respectively. Fortunately, the relative prob-
Vetterling, & Flannery, 1994) by calculating
414

TEAM LinG
Drawing Representative Samples from Large Databases

v ment is that the random selection method must provide

ability distribution W (x ) suffices to perform the calcula-
tions and construct a sample (Kalos & Whilock, 1986).
Sampling Procedure
Similar to simple random sampling, objects are selected
randomly from the population one after another. The
Metropolis sampling guarantees that the sample selected
has a similar distribution to that of the population.
Selecting the Starting Point
Objects with higher weight are more important than others.
If a method does not start with those objects, we could miss
them in the process or take long time to incorporate them
into the sample, which is formidable in cost and detrimental
to accuracy. In the following, we propose a general ap-
proach to selecting a starting point.
We begin with searching for an object with the maxi-
v
mum W (x ) . If there are several objects with the same
maximum value, we could just pick any of them as the best
v After each Monte Carlo step, we add one more object
starting host x1 . In many applications, such as the Gallup
v until a predefined sample size N is reached. It is expected
example, finding the maximal value of W (x ) is straightfor-
ward. For others, there are several useful approaches for
searching for the maximum of functions, such as Golden
Section search, Downhill Simplex method, Conjugate Gra-
dient method, and etcetera. A good review of these meth-
ods can be found in the reference (Press, Teukolsky,
Vetterling, & Flannery, 1994).
a chance for every element in the entire data space to be ,
picked.
v
We now decide if the trial object y should be incor-
porated into the sample. We calculate the ratio
v v v
= W ( y ) / W ( xi ) . If 1 , we accept y into the sample
v v v
and let xi +1 = y . That is, y becomes the last object
added to the sample. If < 1 , we generate a random
number R, which has an equal probability to lie between
v
0 and 1. If R , the trial objecty is accepted into the
v v v
sample and we let xi +1 = y . Otherwise, the trial object y
v v
is rejected and we let xi +1 = xi . It is noted that in the
latter situation, we incorporate the just selected object
v v v
xi into the sample again (i.e. xi +1 = xi ). Therefore, an
object with a high probability may appear more than once
in our sample. The above step is called a Monte Carlo
step.
into the sample. The above Monte Carlo step is repeated
that the sample average converges very fast as N in-
creases and the fluctuation decreases in the order of
1 / N (Metropolis, Rosenbluth, Rosenbluth, Teller, &
Teller, 1953). The complete sampling procedure, which
includes selecting the starting point and the Metropolis
algorithm, is summarized in Figure 1.

Incorporating Objects into Sample Properties of a Metropolis Sample

v Since the selection starts with an object having the
Let the object last added to the sample be xi . Now, we pick
v
v
a trial object y from the population randomly. In general, maximum weight W (x ) , it ensures that the sample
v always includes the most important objects. In addition,
there is no restriction on how to select y . The only require-

Figure 1. The Metropolis Sampling

v
(1) locate an object with the maximum weight W (xv ) as the first object x1 of the sample.
(2) for i from 1 to N-1 do
v
(3) randomly select an object y ;
(4) compute = W ( yv) / W ( xvi ) ;
v v
(5) if 1 then xi +1 = y ;
(6) else generate a random number R;
v v
(7) if R then xi +1 = y ;
v v
(8) else xi +1 = xi ;
(9) end if;
(10) end if;
(11) end for.

415

TEAM LinG

the sample has a distribution close to that of the popula-
tion when the sample size is large enough. Let r l be a
random variable denoting the number of occurrences
for object xl (l = 1, 2, , k) in the sample, then
E (r1 ) : E (r2 ) : ... : E (rk ) W ( x1 ) : W ( x2 ) : ... : W ( xk )
, Quality of the Samples
when the sample size is large enough, where E(rl) is the
expected number of occurrences for object rl in the sample.
We also have E (r1 ) / N w( x1 ), E ( r2 ) / N w( x2 ), ,
E ( rk ) / N w( xk ) .
Finally, it should be pointed out that when all objects
have an equal weight, the Metropolis sampling degener-
ates to the simple random sampling.
Experimental Results
Now, we report the results of our empirical evaluations of
the Metropolis sampling and the AR sampling. We com-
pare the efficiency of the methods and the quality of the
samples yielded.
To compare the methods quantitatively, we select a
model for which we know the analytic values. Here, we
have chosen the Gaussian model
v v
(1 / ) d / 2 e ( x ) , where d is the dimension and ( x ) = x 2 .
2
The mean, the second moment, and variance are:
v v v
= x 2 w( x ) dx = d / 2 ,
v 2 v v
= ( x 2 ) w( x ) dx = ( d 2 + 2d ) / 4 ,
2 (6)
2 = 2 ( ) 2 = d / 2 .
Figure 2 is a 2-D Gaussian distribution. It has a high
peak at the center. Only a small region near the center
makes significant contributions to the integrations of
Equations (5) and (6), and the vast remaining region has
vanishing contributions. As the dimension increases, the
peak becomes higher and narrower, and the distribution
becomes more skewed. In our experiments, we let d =
1, 3, 10, and 20.
Sampling Efficiency
First, let us compare the cost of sampling. As shown in
Figure 3, the AR method roughly needs 5, 10, 75, and 1,750
trials to accept just one object into the sample in 1, 3, 10,
and 20-dimensional cases, respectively. It is noted that
the higher the dimension, the more skewed the Gaussian
416
Drawing Representative Samples from Large Databases
distribution and the greater the Wmax / Wavg value. This
explains why the AR sampling becomes less efficient as
the dimension increases. In comparison, our Metropolis
sampling method accepts one object in every trial.
2
Instead of showing the variance of estimation < > ,
here we show the second moment of , < 2 > . The
second moment tells how different the estimates are from
the actual values, especially when the estimate is biased.
As shown in Figures 4 and 5, both methods yield pretty
accurate estimates of the population mean (= 0.5) and
second moment (= 0.75) for one-dimensional case. Hence,
from Equation (7), they also give good estimates of the
variance.
As shown in Figures 6 and 7 for d=3, the AR sampling
yields estimates around 1.0 and 1.7 for the mean and
second moment, respectively, which are below their re-
spective exact values 3/2 and 15/4. On the other hand, our
Metropolis sampling yields quite accurate estimates. As
the sample size gets larger, our estimates get closer to the
v analytic values, but ARs do not.
w(x ) =
Similar results are also observed for higher dimen-
v
sional cases. As shown in Figures 8 and 9, when d=20, the
AR sampling yields estimates around 5.1 and 29 for the
mean and second moment, respectively, which are well
below the exact values 10 and 110. As for our method, a
(5) sample of size as small as of 10,000 objects already gives
< > 10 and < 2 > 110.
The underestimation of the AR sampling was attrib-
uted to two factors: the acceptance/rejection criterion
v
(7) W ( x ) / Wmax of the AR sampling and the random number
generator. In the Gaussian model described earlier, W max
v v
appears at the center (i.e., x = 0 ), and the weight W (x )
diminishes quickly as we move away from the center.
Indeed, the majority of the points have very small weights
and thus small W ( xv ) / Wmax = e ( x ) ratios, especially when
v2
v
the dimension is high. The low W ( x ) / Wmax values could
make the remote points not selectable when compared
with the random numbers generated in the process.
The random number generators on most computers
are based on the linear congruent method, which first
generates a sequence of integers by the recurrence rela-
tion I j +1 = aI j + c (mod m), where m is the modulus, and
a and c are positive integers (Press, Teukolsky, Vetterling,
& Flannery, 1994). The random numbers generated are I1/
m, I 2/m, I3/m, ..., and the smallest numbers generated is 1/
m. As a result, a trial point in the AR sampling, whose
TEAM LinG
Drawing Representative Samples from Large Databases

Figure 2. A Gaussian distribution: d=2 Figure 3. Cost of AR sampling

,
2.0M

d=20
1.6M

1.2M
d=10

Trial
s
800.0k

d=1
400.0k
d=3
0.0
0 2k 4k 6k 8k 10k 12k 14k 16k 18k 2k
Sample Size

Figure 4. Sample means for d=1. < > = 0.5 Figure 5. Sample means of second moment. < 2 > = 0.75
0.9
0.8
MC
0.7 AR
0.6
0.5
<>

0.4
0.3
0.2
0.1
0.0
0 2k 4k 6k 8k 10k 12k 14k 16k 18k 20k
Sample Size

Figure 6. Sample means for d=3. < > = 1.5 Figure 7. Sample means of second moment < 2 > = 15/
4
2.5 7

MC 6 MC
2.0 AR AR
5

1.5 4
< 2 >
<>

3
1.0

2
0.5
1

0.0 0
0 2k 4k 6k 8k 10k 12k 14k 16k 18k 20k 0 2k 4k 6k 8k 10k 12k 14k 16k 18k 20k
Sample Size Sample Size

417

TEAM LinG
 Drawing Representative Samples from Large Databases
Figure 8. Sample means for d=20. < > = 10
12
10
8
<>
6 80
4
MC
2 AR
0 0
0 2k 4k 6k 8k 10k 12k 14k 16k 18k 20k
Sample Size
v stabilizes as N gets larger. We have also performed tests
W ( x ) / Wmax is smaller than 1/m, cannot be accepted. Since
these remote points have larger values than the points
v v
near the center, recalling that ( x ) = x 2 , underestima-
tion sets in. The higher the dimension, the more skewed
the distribution, and the more serious the underestima-
tion. Increasing sample size would not help because those
points simply will not be accepted.
On the other hand, our Metropolis sampling uses the
weight ratio of the trial and the last accepted objects. As
points, away from the center, are accepted, the chances of
accepting remote points increase. Therefore, it is immune
from the difficulty associated with the AR sampling method.
Based on our experiments with the Gaussian distribu-
tions, a sufficient minimum size should be in the neighbor-
hood of 500d to 1,000d, where d is the number of dimen-
sions for the population. For smoother distributions, the
results can be even better.
Distribution
To examine whether a sample has a similar distribution to
the underlying population, generally a Chi-square test is
performed. Since the Chi-square test is designed for
discrete data, we divide each dimension into 7 bins of
equal length to compute the 2 values.
Figure 10 shows the 2 -test results of ARs and our
samples. We have chosen the commonly used signifi-
cance level =0.05 for the test. It is observed that when
sample size N > 200, our 2 is below 12 = 02.95 = 12.6 FUTURE TRENDS
with = 6 (=7-1). This indicates the agreement of the
2 ful, many databases in social, economical, engineering,
sample with the population distributions. The value
418
Figure 9. Sample means of second moment. < 2 > = 110
160
140 MC
AR
120
< 2 >
100
60
40
20
0 2k 4k 6k 8k 10k 12k 14k 16k 18k 20k
Sampel Size
on other bin values and the results are similar, all verifying
the agreement of the sample distribution with the popu-
lation distribution. As for AR sampling, it also performs
very well, showing a strong agreement of the sample
distribution with the underlying population distribution.
Extending this test to the three-dimensional Gaussian
model, we concentrate on the cubic space divided it into
7 3 = 343 bins. The results are plotted in Figure 11. As
observed, our 2 values are always less than 02.95 ( 386
with = 342 ), indicating a good agreement with the
population distribution. As for the AR sampling, it is
observed that for any sample of reasonable size, its 2
well exceeds 02.95 386 for = 342 : moreover, its 2
increases with the sample size. This indicates that the
samples have different distributions from the population.
As explained earlier, this is because the AR sampling
could not include points with very low weights and thus
as more points are included (or as the sample size in-
creases) the more different the sample distribution is from
the population distribution. The results are consistent
with the evaluation of means and variance discussed in
the previous sections. The AR sampling may work well
when the probability distribution is not very skewed, but
our approach works for all distributions. In addition, our
approach is also more efficient.
While modern computers become more and more power-
scientific, and statistical applications may still be too
TEAM LinG
Drawing Representative Samples from Large Databases
Figure 10. Chi-square test on 1-d samples
with 2
0.95 = 12.6 and = 6
25
MC
20
AR
15
2
10
5
0
0 500 1000 1500 2000
Sam p le S iz e N
large to handle. Sampling, therefore, becomes a necessity
for analyses, surveys, and numerical calculations in these
applications. In addition, for many modern applications in
OLAP and data mining, where fast responses are required,
sampling also becomes a viable approach for construct-
ing an in-core representation of the data. A general
sampling algorithm that applies to all distributions is
needed more than ever.
CONCLUSION
The Metropolis sampling presented in this paper can be
a useful and powerful tool for studying large databases of
any distributions. We propose to start the sampling by
taking an object from where the probability distribution
has its maximum. This guarantees that the sample always
includes the most important objects and improves the
efficiency of the process. Our experiments also indicate a
strong agreement between the selected sample and the
population distributions. The selected sample is bona
fide representative, better than the samples produced by
other existing methods.
REFERENCES
Acharya, S., Gibbons, P., & Poosala, V. (2000).
Cogressional samples for approximate answering of group-
by queries. In Proceedings of ACM SIGMOD Conference
(pp. 487-498).
Agrawal, R., Gehrke, J., Gunopulos, D., & Raghavan, P.
(1998). Automatic subspace clustering of high dimen-
Figure 11. Chi-square test on 3-d samples with
02.95 = 386 and = 342 ,
1000
MC
AR
800
600
2
400
200
2500 3000
0
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Sample Size N
sional data for data mining application. In Proceedings of
the ACM SIGMOD Conference (pp. 94-105).
Haas, P., & Swami, A. (1992). Sequential sampling proce-
dures for query size estimation. In Proceedings of the
ACM SIGMOD Conference (pp. 341-350).
Hou, W.-C., & Ozsoyoglu, G. (1991). Statistical estimators
for aggregate relational algebra queries. ACM Transac-
tions on Database Systems, 16(4), 600-654.
Jermaine, C. (2003). Robust estimation with sampling and
approximate pre-aggregation. In Proceedings of VLDB
Conferences (pp. 886-897).
Kalos, M., & Whilock, P. (1986). Monte Carlo methods:
basic. New York: John Wiley & Sons.
Lipton, R., Naughton, J., & Schnerder, D. (1990). Practical
selectivity estimation through adaptive sampling. In Pro-
ceedings of the ACM SIGMOD Conference (pp. 1-11).
Metropolis, N., Rosenbluth, A., Rosenbluth, M., Teller
A., & Teller, E. (1953). Equation of state calculations by
fast computing machines. Journal of Chemical Phys-
ics, 21(6), 1087-1092.
Olken, F. (1993). Random sampling from databases.
Ph.D. dissertation. University of California.
Palmer, C., & Faloutsos, C. (2000). Density biased
sampling: An improved method for data mining and
clustering Proceedings of the ACM SIGMOD Confer-
ence, 29 (2), 82-92.
Press, W., Teukolsky, S., Vetterling, W., & Flannery, B.
(1994). Numerical recipes in C. Cambridge: Cambridge
University Press.
419
TEAM LinG
 Drawing Representative Samples from Large Databases
Rubinstein, R. (1981). Simulation and the Monte Carlo
Method. New York: John Wiley & Sons.
Spiegel, M. (1991). Probability and statistics. McGraw-
Hill, Inc.
Wu, Y., Agrawal, D., & Abbadi, A. (2001). Using the
golden rule of sampling for query estimation. In Proceed-
ings of ACM SIGMOD Conference (pp. 279-290).
Xu, X., Ester, M., Kriegel, H., & Sander, J. (1998). A
distribution-based clustering algorithm for mining in large
spatial databases. In Proceedings of the IEEE ICDE
Conference (pp. 324-331).
KEY TERMS
Metropolis Algorithm: Was proposed in 1953 by
Metropolis et al. for studying statistical physics. Since
then it has become a powerful tool for investigating
420
thermodynamics, solid state physics, biological systems,
and etcetera. The algorithm is known as the most success-
ful and influential Monte Carlo method.
Monte Carlo Method: The heart of this method is a
random number generator. The term Monte Carlo Method
now stands for any sort of stochastic modeling.
OLAP: Online Analytical Processing.
Representative Sample: A sample whose distribution
is the same as that of the underlying population.
Sample: A set of elements drawn from a population.
Selectivity: The ratio of the number of output tuples
of a query to the total number of tuples in the relation.
Uniform Sampling: All objects or clusters of objects
are drawn with equal probability.
TEAM LinG
 421

Efficient Computation of Data Cubes and -

Aggregate Views
Leonardo Tininini
CNR - Istituto di Analisi dei Sistemi e Informatica Antonio Ruberti, Italy

INTRODUCTION precisely to data groups, each containing a subset of the

data and homogeneous with respect to a given set of
This paper reviews the main techniques for the efficient attributes. For example, the data Average duration of
calculation of aggregate multidimensional views and data calls in 2003 by region and call plan is obtained from the
cubes, possibly using specifically designed indexing so-called fact table, which is usually the product of
structures. The efficient evaluation of aggregate multidi- complex source integration activities (Lenzerini, 2002) on
mensional queries is obviously one of the most important the raw data corresponding to each phone call in that year.
aspects in data warehouses (OLAP systems). In particu- Several groups are defined, each consisting of calls made
lar, a fundamental requirement of such systems is the in the same region and with the same call plan, and finally
ability to perform multidimensional analyses in online applying the average aggregation function on the dura-
response times. As multidimensional queries usually in- tion attribute of the data in each group (see Figure 1). The
volve a huge amount of data to be aggregated, the only triple of values (region, call plan, year) is used to identify
way to achieve this is by pre-computing some queries, each group and is associated with the corresponding
storing the answers permanently in the database and average duration value. In multidimensional databases,
reusing these almost exclusively when evaluating queries the attributes used to group data define the dimensions,
in the multidimensional database. These pre-computed whereas the aggregate values define the measures.
queries are commonly referred to as materialized views The term multidimensional data comes from the well-
and carry several related issues, particularly how to effi- known metaphor of the data cube (Gray, Bosworth, Lay-
ciently compute them (the focus of this paper), but also man, & Pirahesh, 1996). For each of n attributes, used to
which views to materialize and how to maintain them. identify a single measure, a dimension of an n-dimen-
sional space is considered. The possible values of the
identifying attributes are mapped to points on the
BACKGROUND dimensions axis, and each point of this n-dimensional
space is thus mapped to a single combination of the
Multidimensional data are obtained by applying aggrega- identifying attribute values and hence to a single aggre-
tions and statistical functions to elementary data, or more gate value. The collection of all these points, along with

Figure 1. From facts to data cubes and drill-down on the time dimension
C P2 C P3 n Italy
Norther
C P1 Italy
C entral
Phone calls
rn Italy
in 2003 So uthe

Average on the
duration attribute

Time
Italy Italy
CP3 hern CP3 hern
No rt aly No rt tral It
aly
CP2 tral It Italy CP2 C en Italy
C en thern thern
CP1 Sou CP1 Sou
2003 (1. quat.)
2003 2003 (2. quat.)
2003 (3. quat.)
Drill-down on the 2003 (4. quat.)
time dimension
2002

2001

plan R eg
C all ion

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
 Efficient Computation of Data Cubes and Aggregate Views
all possible projections in lower dimensional spaces,
constitutes the so-called data cube. In most cases, dimen-
sions are structured in hierarchies, representing several
granularity levels of the corresponding measures
(Jagadish, Lakshmanan, & Srivastava, 1999). Hence a time
dimension can be organized into days, months, quarters
and years; a territorial dimension into towns, regions and
countries; a product dimension into brands, families and
types. When querying multidimensional data, the user
specifies the measures of interest and the level of detail
required by indicating the desired hierarchy level for each
dimension. In a multidimensional environment querying
is often an exploratory process, where the user moves
along the dimension hierarchies by increasing or reducing
the granularity of displayed data. The drill-down opera-
tion corresponds to an increase in detail, for example, by
requesting the number of calls by region and quarter,
starting from data on the number of calls by region or by
region and year. Conversely, roll-up allows the user to
view data at a coarser level of granularity.
Multidimensional querying systems are commonly
known as OLAP (Online Analytical Processing) Systems,
in contrast to conventional OLTP (Online Transactional
Processing) Systems. The two types have several con-
trasting features, although they share the same require-
ment of fast online response times:
Number of records involved: One of the key differ-
ences between OLTP and multidimensional queries
is the number of records required to calculate the
answer. OLTP queries typically involve a rather
limited number of records, accessed through pri-
mary key or other specific indexes, which need to be
processed for short, isolated transactions or to be
issued on a user interface. In contrast, multidimen-
sional queries usually require the classification and
aggregation of a huge amount of data.
Indexing techniques: Transaction processing is
mainly based on the access of a few records through
primary key or other indexes on highly selective
attribute combinations. Efficient access is easily
achieved by well-known and established indexes,
particularly B+-tree indexes. In contrast, multidi-
mensional queries require a more articulated ap-
proach, as different techniques are required and
each index performs well only for some categories of
queries/aggregation functions (Jrgens & Lenz, 1999).
Current state vs. historical DBs: OLTP operations
require up-to-date data. Simultaneous information
access/update is a critical issue and the database
usually represents only the current state of the
system. In OLAP systems, the data does not need
to be the most recent available and should in fact be
time-stamped, thus enabling the user to perform
422
historical analyses with trend forecasts. However,
the presence of this temporal dimension may cause
problems in query formulation and processing, as
schemes may evolve over time and conventional
query languages are not adequate to cope with them
(Vaisman & Mendelzon, 2001).
Target users: Typical OLTP system users are clerks,
and the types of query are rather limited and predict-
able. In contrast, multidimensional databases are
usually the core of decision support systems, tar-
geted at management level. Query types are only
partly predictable and often require highly expres-
sive (and complex) query language. However, the
user usually has little experience even in easy
query languages like basic SQL: the typical interac-
tion paradigm is a spreadsheet-like environment
based on iconic interfaces and the graphical meta-
phor of the multidimensional cube (Cabibbo &
Torlone, 1998).
MAIN THRUST
In this section we briefly analyze the techniques pro-
posed to compute data cubes and, more generally, mate-
rialized views containing aggregates. The focus here is on
the exact calculation of the views from scratch. In particu-
lar, we do not consider (a) the problems of aggregate view
maintenance in the presence of insertions, deletions and
updates (Kotidis & Roussopoulos, 1999; Riedewald,
Agrawal, & El Abbadi, 2003) and (b) the approximated
calculation of data cube views (Wu, Agrawal, & El Abbadi,
2000; Chaudhuri, Das, & Narasayya, 2001), as they are
beyond the scope of this paper.
(Efficiently) Computing Data Cubes and
Materialized Views
A typical multidimensional query consists of an aggre-
gate group by query applied to the join of the fact table
with two or more dimension tables. In consequence, it has
the form of an aggregate conjunctive query, for example:
SELECT D1.dim1, D2.dim2, AGG(F.measure)
FROM fact_table F, dim_table1 D1, dim_table2 D2
WHERE F.dimKey1 = D1.dimKey1
AND F.dimKey2 = D2.dimKey2
GROUP BY D1.dim1, D2.dim2 (Q1)
where AGG is an aggregation function, such as SUM,
MIN, AVG, etc.
For example, in the above-mentioned phone call data
warehouse the fact table Phone_calls may have the (sim-
TEAM LinG
Efficient Computation of Data Cubes and Aggregate Views
plified) schema (call_id, territ_id, call_plan_id, duration),
and the dimension tables Terr and Call_pl the simplified
schemas (territ_id, town, region) and (call_plan_id,
call_plan_name) respectively. The aggregation illustrated
in Figure 1 would be performed by the following query:
SELECT D1.region, D2.call_plan_name, AVG(F.duration)
FROM Phone_calls F, Terr D1, Call_pl D2
WHERE F.territ_id = D1.territ_id
AND F.call_plan_id = D2.call_plan_id
GROUP BY D1.region, D2.call_plan_name (Q2)
Traditional query processing systems first perform all
joins expressed in the FROM and WHERE clause, and only
afterwards perform the grouping on the result of the join
and aggregation on each group. The algorithms producing
the groups can be broadly classified as techniques based
on (a) sorting on the GROUP BY attributes and (b) hashing
tables [see Graefe (1993)]. However, there are many com-
mon cases where an early evaluation of the GROUP BY is
possible. This can significantly reduce calculation time, as
it (i) reduces the input size of the join (usually very large
in the context of multidimensional databases) and (ii)
enables the query processing engine to use indexes to
perform the (early) GROUP BY on the base tables.
In Chaudhuri & Shim (1995, 1996), some techniques and
applicability conditions are proposed for transforming
execution plans into equivalent (more efficient) ones. As
is typical in query optimization, the technique is based on
pull-up transformations, which delay the execution of a
costly operation (e.g., a group by on a large dataset) by
moving it towards the root of the query tree, and on push-
down transformations, used for example to anticipate an
aggregation, thus decreasing the size of a join.
Several transformations for multidimensional queries
are also proposed in Gupta, Harinarayan, & Quass (1995),
based on the concept of generalized projection (GP). Trans-
formations enable the optimizer to (i) push a GP down the
query tree; (ii) pull a GP up the query tree; (iii) coalesce two
GPs into one, or conversely split one GP into two. Query
tree transformations are also used in the rewriting process.
In Agarwal et al. (1996) some algorithms are proposed
and compared to extend the traditional techniques for the
GROUP BY query evaluation to the CUBE operator pro-
cessing. These are applicable in the case of distributive
aggregate functions and are based on the property that
higher-level aggregates can be calculated from lower lev-
els in this case.
In MOLAP systems, however, the above methods for
CUBE calculation are inadequate, as they are substantially
based on sorting and hashing techniques, which can not
be applied to multidimensional arrays. In Zhao, Deshpande,
& Naughton (1997) an algorithm is proposed to calculate
CUBEs in a MOLAP environment. It is shown that this can
be made significantly more efficient (and even more so
than ROLAP-based calculations) by exploiting the inher- -
ently compressed data representation of MOLAP sys-
tems. The algorithm particularly benefits from the com-
pactness of multidimensional arrays, enabling the query
processor to transfer larger chunks of data to the main
memory and efficiently process them.
In many practical cases GROUP BYs at the finest
granularity level correspond to sparse data cubes, that
is, cubes where a high percentage of points correspond
to null values. Consider for instance the CUBE corre-
sponding to the query Number of calls by customer, day
and antenna: it is evident that a considerable number of
combinations correspond to zero. Fast techniques to
compute sparse data cubes are proposed in Ross &
Srivastava (1997). They are based on (i) decomposing the
fact table into fragments that can be stored in main
memory; (ii) computing the data cube in the main memory
for each fragment and finally (iii) combining the partial
results obtained in (i) and (ii).
Using Indexes to Improve Efficiency
The most common indexes used in traditional DBMSs are
probably B+-trees, a particular form labeled with index
keyvalues and having a list of record identifiers on the
leaf level; that is, a list of elements specifying the actual
position of each record on the disk. Index keyvalues may
consist of one or more columns of the indexed table.
OLTP queries usually retrieve a very limited number of
tuples (or even a single tuple accessed through the
primary key index) and in these cases B+-trees have been
demonstrated as particularly efficient.
In contrast, OLAP queries typically involve aggrega-
tion of large tuples groups, requiring specifically de-
signed indexing structures. In contrast to the OLTP
context, there is no universally good index for multidi-
mensional queries, but rather a variety of techniques,
each of which may perform well for specific data types
and query forms but be inappropriate for others.
Let us again consider the typical multidimensional
query expressed by the SQL query (Q1). The core opera-
tions related to its evaluation are: (1) the joins of the fact
table with two or more dimension tables, (2) tuple group-
ing by various dimensional values, and (3) application of
an aggregation function to each tuple group. An interest-
ing index type which can be used to efficiently perform
operation (1) is the join index; while conventional in-
dexes map column values to records in one table, join
indexes map them to records in two (or more) joined
tables, thus constituting a particular form of materialized
view. Join indexes in their original version can not be
used directly for efficient evaluation of OLAP queries,
but can be very effective in combination with other
423
TEAM LinG
 Efficient Computation of Data Cubes and Aggregate Views
indexing techniques, such as bitmaps and partitioning.
Bitmap indexes (Chan & Ioannidis, 1998) are useful for
tuple grouping and performing some aggregation forms.
In practice, these indexes use a bitmap representation for
the list of record identifiers in the trees leaf level: if table
t contains n records, then each leaf of the bitmap index
(corresponding to a specific value c of the indexed column
C) contains a sequence of n bits, where the i-th bit is set
to 1 if ti.C=c, and otherwise to zero. Bitmap representa-
tions are indicated when the number of distinct keyvalues
is low and several predicates of the form (Column = value)
are to be combined in AND/OR, as the operation can be
efficiently performed by AND-/OR-ing the corresponding
bitmap representation bit to bit. This operation can be
performed in parallel and very efficiently on modern
processors. Finally, bitmap indexes can be used for fast
count query evaluation, as counting can be performed
directly on the bitmap representation without even ac-
cessing the selected records.
In projection indexes (ONeil & Quass, 1997) the tree
access structure is coupled with a sort of materialized
view, representing the projection of the table on the
indexed column. The technique has some analogies with
vertically partitioned tables and is indicated when the
aggregate operations need to be performed on one or more
indexed columns.
Bit-sliced indexes (ONeil & Quass, 1997) can be con-
sidered as a combination of the two previous techniques.
Values of the projected column are encoded and a bitmap
associated with each resulting bit component. The tech-
nique has some analogies with bit transposed files (Wong,
Li, Olken, Rotem, & Wong, 1986), which were proposed for
query evaluation in very large scientific and statistical
databases. These indexes work best for SUM and AVG
aggregations, but are not well suited for aggregations
involving more than one column.
In Chan & Ioannidis (1998, 1999) some variations on
the general idea of bitmap indexes are presented. Encod-
ing schemes, time-optimal and space-optimal indexes, and
trade-off solutions are studied. A comparison of the use
of STR-tree based indexes (a particular form of spatial
index) and some variations of bitmap indexes in the context
of OLAP range queries can be found in Jrgens & Lenz (1999).
FUTURE TRENDS
The efficiency of many evaluation techniques is strictly
related to the adopted query language and storage tech-
nique (e.g., MOLAP and ROLAP). This stresses the im-
portance of a standardization process: there is indeed a
general consensus on data warehouse key concepts, but
a common unified framework for multidimensional data
424
querying is still lacking, particularly a standardized query
language independent from the specific storage tech-
nique.
Although research results on the exact computation
of data cubes in the last few years have been mainly
incremental, several interesting techniques have been
proposed for the approximated computation, particularly
some based on wavelets, which certainly require further
investigations.
In contrast to the OLTP context, we have shown that
there is no universally good index for multidimensional
queries, but rather a variety of techniques, each of which
may perform well for specific data types and query forms
but be inappropriate for others. Hence, an algorithm of
index selection for data warehouses should determine not
only the several sets of attributes to be indexed, but also
the index type(s) to be used. The definition of an indexing
structure enabling a good trade-off in the several cases of
interest is also an interesting issue for future research.
CONCLUSION
In this paper we have discussed the main issues related
to the computation of data cubes and aggregate material-
ized views in a data warehouse environment. First of all,
the main features of OLAP queries with respect to conven-
tional OLTP queries have been summarized, particularly
the number of records involved, the temporal aspects, the
specific indexes. Various techniques for the exact compu-
tation of materialized views and data cubes from scratch
in both ROLAP and MOLAP environments have been
discussed. Finally, the main indexing techniques for OLAP
queries and their applicability have been illustrated.
REFERENCES
Agarwal, S., Agrawal, R., Deshpande, P., Gupta, A.,
Naughton, J.F., Ramakrishnan, R., & Sarawagi, S. (1996).
On the computation of multidimensional aggregates. In
International Conference on Very Large Data Bases
(VLDB96) (pp. 506-521).
Cabibbo, L., & Torlone, R. (1998). From a procedural to a
visual query language for OLAP. In International Con-
ference on Scientific and Statistical Database Manage-
ment (SSDBM98) (pp. 74-83).
Chan, C.Y., & Ioannidis, Y.E. (1998). Bitmap index design
and evaluation. In ACM International Conference on
Management of Data (SIGMOD98) (pp. 355-366).
Chan, C.Y., & Ioannidis, Y.E. (1999). An efficient bitmap
encoding scheme for selection queries. In ACM Interna-
TEAM LinG
Efficient Computation of Data Cubes and Aggregate Views
tional Conference on Management of Data (SIGMOD99)
(pp. 215-226).
Chaudhuri, S., Das, G., & Narasayya, V. (2001). A robust,
optimization-based approach for approximate answering
of aggregate queries. In ACM International Conference
on Management of Data (SIGMOD01) (pp. 295-306).
Chaudhuri, S., & Shim, K. (1995). An overview of cost-
based optimization of queries with aggregates. Data En-
gineering Bulletin, 18(3), 3-9.
Chaudhuri, S., & Shim, K. (1996). Optimizing queries with
aggregate views. In International Conference on Ex-
tending Database Technology (EDBT96) (pp. 167-182).
Graefe, G. (1993). Query evaluation techniques for large
databases. ACM Computing Surveys, 25(2), 73-170.
Gray, J., Bosworth, A., Layman, A., & Pirahesh, H. (1996).
Data cube: A relational aggregation operator generalizing
group-by, cross-tab, and sub-total. In International Con-
ference on Data Engineering (ICDE96) (pp. 152-159).
Gupta, A., Harinarayan, V., & Quass, D. (1995). Aggre-
gate-query processing in data warehousing environments.
In International Conference on Very Large Data Bases
(VLDB95) (pp. 358-369).
Jagadish, H.V., Lakshmanan, L.V.S., & Srivastava, D.
(1999). What can hierarchies do for data warehouses?
International Conference on Very Large Data Bases
(VLDB99) (pp. 530-541).
Jrgens, M., & Lenz, H.J. (1999). Tree based indexes vs.
bitmap indexes: A performance study. In International
Workshop on Design and Management of Data Ware-
houses (DMDW99). Retrieved from <http://
sunsite.informatik.rwth-aachen.de/Publications/CEUR-
WS/Vol-19/paper1.pdf>
Kotidis, Y., & Roussopoulos, N. (1999). DynaMat: A
dynamic view management system for data warehouses.
In ACM International Conference on Management of
Data (SIGMOD99) (pp. 371-382).
Lenzerini, M. (2002). Data integration: A theoretical per-
spective. In ACM Symposium on Principles of Database
Systems (PODS02) (pp. 233-246).
ONeil, P.E., & Quass, D. (1997). Improved query perfor-
mance with variant indexes. In ACM International Con-
ference on Management of Data (SIGMOD97) (pp. 38-
49).
Riedewald, M., Agrawal, D., & El Abbadi, A. (2003).
Dynamic multidimensional data cubes. In M. Rafanelli
(Ed.), Multidimensional Databases: Problems and Solu-
tions (pp. 200-221). Hershey, PA: Idea Group Publishing.
Ross, K.A., & Srivastava, D. (1997). Fast computation of
sparse datacubes. In International Conference on Very -
Large Data Bases (VLDB97) (pp. 116-125).
Vaisman A.A., & Mendelzon A.O. (2001). A temporal
query language for OLAP: Implementation and a case
study. In 8th International Workshop on Database Pro-
gramming Languages (DBPL 2001) (pp. 78-96).
Wong, H.K.T., Li, J., Olken, F., Rotem, D., & Wong, L.
(1986). Bit transposition for very large scientific and
statistical databases. Algorithmica, 1(3), 289-309.
Wu, Y., Agrawal, D., & El Abbadi, A. (2000). Using
wavelet decomposition to support progressive and ap-
proximate range-sum queries over data cubes. In Confer-
ence on Information and Knowledge Management
(CIKM00) (pp. 414-421).
Zhao, Y., Deshpande, P., & Naughton, J.F. (1997). An
array-based algorithm for simultaneous multidimensional
aggregates. In ACM International Conference on Man-
agement of Data (SIGMOD97) (pp. 159-170).
KEY TERMS
Aggregate Materialized View: A materialized view
(see below) in which the results of a query containing
aggregations (like count, sum, average, etc.) are stored.
B+-Tree: A particular form of search tree in which the
keys used to access data are stored in the leaves. Particu-
larly efficient for key-access to data stored in slow memory
devices (e.g., disks).
Data Cube: A collection of aggregate values classified
according to several properties of interest (dimensions).
Combinations of dimension values are used to identify the
single aggregate values in the data cube.
Dimension: A property of the data used to classify it
and navigate the corresponding data cube. In multidimen-
sional databases dimensions are often organized into
several hierarchical levels, for example, a time dimension
may be organized into days, months and years.
Drill-Down (Roll-Up): Typical OLAP operation, by
which aggregate data are visualized at a finer (coarser)
level of detail along one or more analysis dimensions.
Fact (Multidimensional Datum): A single elementary
datum in an OLAP system, the properties of which corre-
spond to dimensions and measures.
Fact Table: A table of (integrated) elementary data
grouped and aggregated in the multidimensional query-
ing process.
425
TEAM LinG
 Efficient Computation of Data Cubes and Aggregate Views
Materialized View: A particular form of query whose
answer is stored in the database to accelerate the evalu-
ation of further queries.
Measure: A numeric value obtained by applying an
aggregate function (such as count, sum, min, max or
average) to groups of data in a fact table.
426
Multidimensional Query: A query on a collection of
multidimensional data, which produces a collection of
measures classified according to some specified dimen-
sions.
OLAP System: A particular form of information sys-
tem specifically designed for processing, managing and
reporting multidimensional data.
TEAM LinG

Embedding Bayesian Networks in Sensor Grids
Juan E. Vargas
University of South Carolina, USA
INTRODUCTION
In their simplest form, sensors are transducers that con-
vert physical phenomena into electrical signals. By com-
bining recent innovations in wireless technology, distrib-
uted computing, and transducer design, grids of sensors
equipped with wireless communication can monitor large
geographical areas. However, just getting the data is not
enough. In order to react intelligently to the dynamics of
the physical world, advances at the lower end of the
computing spectrum are needed to endow sensor grids
with some degree of intelligence at the sensor and the
network levels. Integrating sensory data into representa-
tions conducive to intelligent decision making requires
significant effort. By discovering relationships between
seemingly unrelated data, efficient knowledge represen-
tations, known as Bayesian networks, can be constructed
to endow sensor grids with the needed intelligence to
support decision making under conditions of uncertainty.
Because sensors have limited computational capabilities,
methods are needed to reduce the complexity involved in
Bayesian network inference. This paper discusses meth-
ods that simplify the calculation of probabilities in Baye-
sian networks and perform probabilistic inference with
such a small footprint that the algorithms can be encoded
in small computing devices, such as those used in wireless
sensors and in personal digital assistants (PDAs).
BACKGROUND
Recent innovations in wireless development, distributed
computing, and sensing design have resulted in energy-
efficient sensor architectures with some computing capa-
bilities. By spreading a number of smart sensors across a
geographical area, wireless ad-hoc sensor grids can be
configured as complex monitoring systems that can react
intelligently to changes in the physical world. Wireless
sensor architectures such as the University of California
Berkeleys Motes (Pister, Kahn, & Boser, 1999) are being
increasingly used in a variety of fields to
1. Monitor information about enemy movements, ex-
plosions, and other phenomena of interest (Vargas
& Wu, 2003)
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
427
-
2. Monitor chemical, biological, radiological, nuclear,
and explosive (CBRNE) attacks and materials
3. Monitor environmental changes in forests, oceans,
and so forth
4. Monitor vehicle traffic
5. Provide security in shopping malls, parking ga-
rages, and other facilities
6. Monitor parking lots to determine which spots are
occupied and which are free
Typically, sensor networks produce vast amounts of
data, which may arrive at any time and may contain noise,
missing values, or other types of uncertainty. Therefore,
a theoretically sound framework is needed to construct
virtual worlds populated by decision-making agents that
may receive data from sensing agents or other decision-
making entities. Specific aspects of the real world could be
naturally distributed and mapped to devices acting as
data collectors, data integrators, data analysts, effectors,
and so forth. The data collectors would be devices
equipped with sensors (optical, acoustical, radars, etc.) to
monitor the environment or to provide domain-specific
variables relevant to the overall decision-making process.
The data analysts would be engaged in low-level data
filtering or in high-level decision making. In all cases, a
single principle should integrate these activities. Baye-
sian theory is the preferred methodology, because it
offers a good balance between the need for separation at
the data source level and the integrative needs at the
analysis level (Stone, Lawrence, Barlow, & Corwin, 1999).
Another major advantage of Bayesian theory is that data
from different measurement spaces can be fused into a
rich, unified, representation that supports inference un-
der conditions of uncertainty and incompleteness. Baye-
sian theory offers the following additional advantages:
Robust operational behavior: Multisensor data fu-
sion has an increased robustness when compared
to single-sensor data fusion. When one sensor
becomes unavailable or is inoperative, other sen-
sors can provide information about the environ-
ment.
Extended spatial and temporal coverage: Some parts
of the environment may not be accessible to some
sensors due to range limitations. This occurs espe-
cially when the environment being monitored is
TEAM LinG

vast. In such scenarios, multiple sensors that are
mounted at different locations can maximize the
regions of scanning. Multisensor data fusion pro-
vides increased temporal coverage, as some sen-
sors can provide information when others cannot.
Increased confidence: Single target location can be
confirmed by more than one sensor, which increases
the confidence in target detection.
Reduced ambiguity: Joint information from multiple
sensors can reduce the set of beliefs about data.
Decreased costs: Multiple, inexpensive sensors
can replace expensive single-sensor architectures
at a significant reduction of cost.
Improved detection: Integrating measurements from
multiple sensors can reduce the signal-to-noise
ratio, which ensures improved detection.
Bayesian-based or entropy-based algorithms can the
used to construct efficient data structuresknown as
Bayesian networksto represent the relations and the
uncertainties in the domain (Pearl, 1988). After the Baye-
sian networks are created, they can act as hyperdimensional
knowledge representations that can be used for probabi-
listic inference. In situations when data is not as rich, the
knowledge representations can still be created from state-
ments of causality and independence formulated by ex-
pert opinions. Under this framework, activities such as
vehicle control, maneuvering, and scheduling could be
planned, and the effectiveness of those plans could be
evaluated online as the actions of the plans are executed.
To illustrate these ideas, consider a domain composed
of threats, assets, and grids of sensors. Although un-
manned vehicles loaded with sensors might be able to
detect potential targets and provide data to guide the
distribution of assets, integrating and transforming those
data into meaningful information that is amenable for
intelligent decisions is very demanding. The data must be
filtered, the relationships between seemingly unrelated
data sets must be determined, and knowledge representa-
tions must be created to support wise and timely deci-
sions, because conditions of uncertain and incomplete
information are the norm, not the exception. Therefore, a
solution is to endow sensors with embedded local and
global intelligence, as shown in Figure 1. In the figure,
friendly airplanes and tanks, in red, use Bayesian net-
works (BNs) to make decisions. The BNs are illustrated as
red boxes containing graphs. Each node in the graphs
corresponds to a variable in the domain. The data for each
variable may come from sensors spread in the battlefield.
The red nodes are variables related to the friendly re-
sources, and the blue variables to the enemy resources.
The dotted red arrows connecting the BNs represent
wireless communication between the BNs.
428
Embedding Bayesian Networks in Sensor Grids
Figure 1. A domain scenario
The goal is to recognize situations locally and glo-
bally, identify the available options, and make global and
local decisions quickly in order to reduce or eliminate the
threats and optimize the use of assets. The task is difficult
due to the dynamics and uncertainties in the domain.
Threats may change in many ways, targets may move,
enemy forces may identify the sensing capabilities and
eliminate them, and so forth. In most cases, sensing
information will contain noise and most likely will be
inaccurate, unreliable, and uncertain. These constraints
suggest a distributed, bottom-up approach to match the
natural dynamics and uncertainties of the problem. Thus,
at the core of this problem, a theoretical framework that
effectively balances local and global conditions is needed.
Distributed Bayesian networks offer that balance (Xiang,
2002; Valtorta, Kim, & Vomlel, 2002).
MAIN THRUST
Embedding Bayesian Networks in
Sensor Grids
Recent advances in the theory of Bayesian network infer-
ence (Darwiche, 2003; Castillo, Gutierrez, & Hadi, 1996;
Utete, 1998) have resulted in algorithms that can perform
probabilistic inference on very small-scale computing
devices that are comparable to commercially available
PDAs. The algorithms can encode, in real-time, families of
polynomial equations representing queries of the type
p(e|h) involving sets of variables local to the device and
its neighbors.
Using the knowledge representations locally encoded
into these devices, larger, distributed systems can be
interconnected. The devices can assess their local condi-
tions given local observations and engage with other
devices in the system to gain better understanding of the
global situation, to obtain more assets, or to convey
TEAM LinG
Embedding Bayesian Networks in Sensor Grids

information needed by other devices engaged in larger
scale tactical decisions. The devices can maintain models
of the local parameters surrounding them, making them
situationally aware and enabling them to participate as
cells of a larger decision-making process. Sensory infor-
mation might be redirected towards a device with no
access to that sensors information. For example, an air-
plane might have full access to information about the
topology of a certain area, but another vehicle may have
access to this information only if it can communicate with
the airplane. Global information known by groups of de-
vices can be sent to the relevant devices such that when-
ever a device gets new input data, the findings are used to
determine locally the most probable states for each of the
variables within the devices and propagate those determi-
nations to the neighboring devices.
The situation is illustrated in Figure 2, in which the
variables x3, x4, and x6 are shared by more than one
device. The direction of the dotted red arrows indicate the
direction of the flow of evidence.
Symbolic Propagation of Evidence
Methods for exact and approximate propagation of evi-
dence in Bayesian network models are discussed in other
sections of this encyclopedia. A common requirement for
both types of methods is that all the parameters of the joint
probability distribution (JPD) must be known prior to
propagation. However, complete specifications might not
always be available. This may happen when the number of
observations in some combinations of variables is not
sufficient to support exact quantifications of conditional
probabilities, or in cases when domain experts may only
know ranges but may not know the exact values of the
parameters. In such cases, symbolic propagation can be
used to obtain the values of the unknown parameters to -
propagate and compute the probabilities of all the rel-
evant variables in a query (Zhaoyu, DAmbrosio, 1994;
Castillo, Gutierrez, & Hadi, 1995; Castillo et al., 1996).
Symbolic propagation of evidence consists of asserting
polynomials involving the known and missing param-
eters and the available evidence. After those polynomi-
als are found, customized code generators can produce
code to solve the expressions in real-time, obtain the
exact values of the parameters, and complete the propa-
gation of evidence.
In general, a node X i having a conditional probability
p ( xi | i ) can be expressed as a parametric family of the
form
ij = p( X i = j | i = ), j {0,..., ri } (1)
where i refers to the node number, j refers to the state of
the node, and is any instantiation i of the parents of
X i . Assume a JPD given on the binary variables
{x1 , x2 , x3 , x4 } , as
p( x1 , x2 , x3 , x4 ) = p( x1 ) p( x2 | x1 ) p( x3 | x1 ) p( x4 | x2 , x3 ) (2)
The complete set of parameters for the binary vari-
ables for a Bayesian network compatible with the JPD is
given in Table 1.

Table 1. Complete set of parameters for the binary

variables of a Bayesian network compatible with the
JPD of equation 2
Figure 2. A collection of three Bayesian networks

X1 10 = p ( x1 ) 11 = 1 10
X1 X5
X2 200 = p ( x2 | x1 ) 210 = 1 200
X4
X4 201 = p ( x2 | x1 ) 211 = 1 201
X2

X3
X6 X3 300 = p( x3 | x1 ) 310 = 1 300
301 = p ( x3 | x1 ) 311 = 1 300

X4 4000 = p ( x4 | x2 , x3 ) 4100 = 1 4000

X3 X6
4001 = p ( x4 | x2 , x3 ) 4101 = 1 4001
X7 4010 = p ( x4 | x2 , x3 ) 4110 = 1 4010
4011 = p ( x4 | x2 , x3 ) 4111 = 1 4011

429

TEAM LinG

In general, because
ri
ijs = 1
j =0
the parameters for categorical variables can be obtained
from
ri
iks = 1 ijs
j =0..rj \ k
Knowing the JPD in Equation 2 and the parameters in
Table 1, the probability of any set of nodes can be
calculated by extracting the marginal probabilities out of
the JPD. Keeping with the example, I obtain
p ( x2 = 0 | x3 = 1) as follows:
p( x1,1,0, x 4)
p( x2 = 1 | x3 = 0) =
x1, x 4
=
p( x1, x2,0, x 4)
x1, x 2, x 4
10300 10 200 300 + 301 10301 201301 + 10 201 301 dowed with Bayesian intelligence. Symbolic propagation
= (4)
10 300 + 301 10 301
A you can see in Equation 4, if all the parameters are
know, then the expression can be solved by computing
nine multiplications, three additions, four subtractions,
and one division. These operations can be performed in
real-time by using simple computing devices, such as
those available to PDAs or most modern wireless sensor
architectures. If, on the other hand, some of the param-
eters are not available but the topology of the graph is
known, then the use of clustering algorithms for exact
inference can produce additional polynomial expressions
that solve for the unknown parameters. In this case, the
entire inference process consists of the following steps:
1. From the topology of the graph, obtain a junction
tree, using clustering methods for exact propaga-
tion (Lauritzen & Spiegelhalter, 1988; Jensen, 2001;
Olesen, Lauritzen, & Jensen, 1992).
2. Use the junction tree to generate the polynomial
expressions that solve for the unknown parameters
of conditional independence.
3. If the inference process involves goal-oriented
queries, obtain expressions for each parameter, as in
Equations 3 and 4, involving the parameters and the
query. At this point, all the parameters are known,
either numerically or symbolically.
4. Use a code generator to configure the equations
resulting from steps 2 and 3.
430
Embedding Bayesian Networks in Sensor Grids
5. Compute the probabilities by executing the gener-
ated expressions.
6. Complete the process by propagating the results to
the rest of the network.
Castillo et al. (1995, 1996) present methods that take
advantage of the polynomial structure of conditional
probabilities. Their procedure consists of asserting poly-
nomials involving known and unknown parameters as
sets of parametric equations, which are solved symboli-
cally by using computer packages such as Mathematica
or Maple. Having such symbolic packages available to
embedded systems is not a reasonable expectation. None-
theless, two important results can be derived from that
work: (a) Any parameter can be expressed as a first-degree
polynomial and (b) the combined probability of any
instantiation of a set of variables in a network can be
expressed as a polynomial of degree less than or equal to
the number of variables.
Taking advantage of the parametric structure of the
marginal and conditional probabilities is very valuable;
(3) the parametric functions can reduce the complexity in the
calculation and propagation of probabilities and in turn
makes it possible to have grids of sensing devices en-
provides the theoretical framework for a distributed archi-
tecture that can be reconfigured in real-time as a grid of
smart sensors, each performing calculations on variables
only relevant to the query and the local reasoner.
FUTURE TRENDS
Conservative estimates compare the impact of wireless
sensors in the next decade to the microprocessor revolu-
tion of the 1980s. Several million efforts are already on
their way: Wal-Mart announced in 2003 that it would
require its main suppliers to use RFID tags on all pallets
and cartons of goods, with an estimated savings of over
$8 billion a year (Goode, 2003). It is expected that the future
of the automobile and aircraft industries will heavily
depend on sensors (Goode, 2004). Innovations in trans-
ducers and architectures are making worldwide news
everyday, as industrial giants such as Intel, Microsoft
(Microsoft launches, 2002), and IBM enter the compe-
tition with their own products (Goode, 2004). The capabil-
ity of sensors will continue to increase, and their ability
to process information locally will improve. However,
despite the progress at the lower level, the fundamental
issues related to the fusion of data and its analysis at the
higher levels are not likely to undergo dramatic improve-
ments. Bayesian theory, and in particular Bayesian net-
works, still offer the best framework for data fusion,
knowledge discovery, and probabilistic inference. Until
TEAM LinG
Embedding Bayesian Networks in Sensor Grids
now, the main barriers to the use of Bayesian networks in
sensor grids have been the computational complexity of
the inference process the lack of efficient methods for
learning the graph of a network, and adapting to the
dynamics of the domain. The latter two, however, are
problems that permeate the field of Bayesian networks
and constitute fertile ground for future research.
CONCLUSION
This paper outlines recent work aimed at endowing sensor
grids with local and global intelligence. I describe the
advantages of knowledge representations known as Baye-
sian networks and argue that Bayesian networks offer an
excellent theoretical foundation to accomplish this goal.
The power of Bayesian networks lends them to decom-
posing a problem into a set of smaller, distributed problem
solvers. Each variable in a network could be associated to
a different sensing agent represented at the local level as
a Bayesian network. The sensing agents could decide if
and when to send observations to the other agents. As
information flows between the agents, they in turn could
decide to send their own observations, their local infer-
ences, or request local or remote agents for more informa-
tion, given the available evidence. Another advantage of
this approach is that it is scalable on demand; more agents
can be added as the problem gets bigger. Recovery from
loss of parts of the system is possible by introducing a set
of redundant observations, each of which can be part of
a local solver agent.
REFERENCES
Castillo, E. Gutierrez, J. M., & Hadi, A. S. (1995). Symbolic
propagation in discrete and continuous Bayesian net-
works. In V. Keranen & P. Mitic (Eds.), Proceedings of the
First International Mathematica Symposium: Vol. Math-
ematics with vision (pp. 77-84). Southhampton, UK: Com-
putational Mechanics Publications, Southerhampton, UK.
Castillo, E., Gutierrez, J. M., & Hadi, A. S. (1996). A new
method for efficient symbolic propagation in discrete
Bayesian networks. Networks, 28, 31-43.
Darwiche, A. (2003). Revisiting the problem of belief
revision with uncertain evidence. Proceedings of the 18th
International Joint Conference on Artificial Intelligence,
Acapulco, Mexico.
Goode, B. (2003, September). Having wonder full time.
Sensors Magazine.
Goode, B. (2004, August). Keys to keynotes. Sensors
Magazine. -
Jensen, F. V. (2001). Bayesian networks and decision
graphs. New York: Springer.
Lauritzen, S.L., & Spiegelhalter, D.J. (1988). Local compu-
tations with probabilities on graphical structures and
their application to expert systems. Journal of the Royal
Statistical Society, Series B, 50, 157-224.
Microsoft launches smart personal object technology
initiative. (2002, November 17). Retrieved from http://
www.microsoft.com/presspass/features/2002/nov02/11-
17SPOT.asp
Olesen, K. G. , Lauritzen, & Jensen, F. V. (1992). Hugin: A
system creating adaptive causal probabilistic networks.
Proceedings of the Eighth Conference on Uncertainty in
Artificial Intelligence (pp. 223-229), USA.
Pearl, J. (1988). Probabilistic reasoning in intelligent
systems: Networks of plausible inference. San Mateo,
CA: Morgan Kaufmann.
Pister, K. S. J., Kahn, J. M., & Boser, B. E.(1999). Smart
dust: Wireless networks of millimeter-scale sensor nodes.
Highlight Article in Electronics Research Laboratory
Research Summary. Retrieved from http://
www.xbow.com/Products/Wireless_Sensor_Networks.
htm
Sensors Magazine. (Ed.). (2004, August). Best of sensors
expo awards [Special issue]. Sensors Magazine.
Stone, C. A., Lawrence, D., Barlow, C. A., & Corwin, T. L.
(1999). Bayesian multiple target tracking. Boston: Artech
House.
Utete, S. W. (1998). Local information processing for
decision making in decentralised sensing networks. Pro-
ceedings of the 11th International Conference on Indus-
trial and Engineering Applications of Artificial Intelli-
gence and Expert Systems, IEA/AIE-667-676, Castellon,
Spain.
Valtorta, M., Kim, Y. G., & Vomlel, J. (2002). Soft evidential
update for probabilistic multiagent systems. Interna-
tional Journal of Approximate Reasoning, 29(1), 71-106.
Vargas, J. E., Tvarlapati, K., & Wu, Z. (2003). Target
tracking with Bayesian estimation. In V. Lesser, C. Ortiz,
& M. Tambe (Eds.), Distributed sensor networks. Kluwer
Academic Press.
Vargas, J. E., & Wu, Z. (2003). Real-time multiple-target
tracking using networked wireless sensors. Proceedings
of the Second Conference on Autonomous Intelligent
Networks and Systems, Palo Alto, CA.
431
TEAM LinG

Xiang, Y. (2002). Probabilistic reasoning in multiagent
systems: A graphical models approach. Cambridge, MA:
Cambridge University Press.
Zhaoyu, L., & DAmbrosio, B. (1994). Efficient inference
in Bayes networks as a combinatorial optimization prob-
lem. International Journal of Approximate Reasoning,
11, 55-81.
KEY TERMS
Bayesian Network: A directed acyclic graph (DAG)
that encodes the probabilistic dependencies between the
variables within a domain and is consistent with a joint
probability distribution (JPD) for that domain. For ex-
ample, a domain with variables {A,B,C,D}, in which the
variables B and C depend on A and the variable D depends
on C and B, would have the following JPD: P(A,B,C,D) =
p(A)p(B|A)p(C|A)p(D|C,D) and the graph
A
B C
D
Joint Probability Distribution: A function that en-
codes the probabilistic dependencies among a set of
variables in a domain.
Junction Tree: A representation that captures the
joint probability distribution of a set of variables in a very
432
Embedding Bayesian Networks in Sensor Grids
efficient data structure. A junction tree contains cliques,
each of which is a set of variables from the domain. The
junction tree is configured to maintain the probabilistic
dependencies of the domain variables and provides a data
structure over queries of the type What is the most
probable value of variable D given that the values of
variables A, B, etc., are known?
Knowledge Discovery: The process by which new
pieces of information can be revealed from a set of data.
For example, given a set of data for variables {A,B,C,D},
a knowledge discovery process could discover unknown
probabilistic dependencies among variables in the do-
main, using measures such as Kullbacks mutual informa-
tion, which, for the discrete case, is given by the formula
p (ai , b j )
I ( A : B) = p(a , b ) log P(a ) P(b )
i j
i j
i j
Smart Sensors: Transducers that convert some physi-
cal parameters into an electrical signal and are equipped
with some level of computing power for signal processing.
Symbolic Propagation in Bayesian Networks: A
method that reduces the number of operations required to
compute a query in a Bayesian network by factorizing the
order of operations in the joint probability distribution.
Wireless Ad-Hoc Sensor Network: A number of sen-
sors spread across a geographical area. Each sensor has
wireless communication capability and some computing
capabilities for signal processing and data networking.
TEAM LinG

Employing Neural Networks in Data Mining
Mohamed Salah Hamdi
UAE University, UAE
INTRODUCTION
Data-mining technology delivers two key benefits: (i) a
descriptive function, enabling enterprises, regardless of
industry or size, in the context of defined business objec-
tives, to automatically explore, visualize, and understand
their data and to identify patterns, relationships, and
dependencies that impact business outcomes (i.e., rev-
enue growth, profit improvement, cost containment, and
risk management); (ii) a predictive function, enabling
relationships uncovered and identified through the data-
mining process to be expressed as business rules or
predictive models. These outputs can be communicated
in traditional reporting formats (i.e., presentations, briefs,
electronic information sharing) to guide business plan-
ning and strategy. Also, these outputs, expressed as
programming code, can be deployed or hard wired into
business-operating systems to generate predictions of
future outcomes, based on newly generated data, with
higher accuracy and certainty.
However, there also are barriers to effective large-
scale data mining. Barriers related to the technical aspects
of data mining concern issues such as large datasets,
highly complex data, high algorithmic complexity, heavy
data management demands, and qualification of results
(Musick, Fidelis & Slezak, 1997). Barriers related to atti-
tudes, policies, and resources concern potential dangers
such as privacy concerns (Kobsa, 2002).
The number of research projects and publications
reporting experiences with data mining has been growing
steadily. Researchers in many different fields, including
database systems, knowledge-base systems, artificial
intelligence, machine learning, knowledge acquisition,
statistics, spatial databases, data visualization, and
Internet computing, have shown great interest in data
mining. In this contribution, the focus is on the relation-
ship between artificial neural networks and data mining.
We review some of the related literature and report on our
own experience in this context.
BACKGROUND
Artificial Neural Networks
Neural networks are patterned after the biological ganglia
and synapses of the nervous system. The essential ele-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
433
-
ment of the neural network is the neuron. A typical neuron
j receives a set of input signals from the other connected
neurons, each of which is multiplied by a synaptic weight
of wij (weight for connection between neurons i and j). The
resulting activation weights are then summed to produce
the activation level for the neuron j. Learning is carried out
by adjusting the weights in a neural network. Neurons
that contribute to the correct answer have their weights
strengthened, while other neurons have their weights
reduced. Several architectures and error correction
algorithms have been developed for neural networks
(Haykin, 1999).
In general, neural networks can help where an algo-
rithmic solution cannot be formulated, where lots of
examples of the required behavior are available, and where
picking out the structure from existing data is needed.
Neural networks work by feeding in some input variables
and producing some output variables. Therefore, they
can be used where some known information is available
and some unknown information should be inferred.
Data Mining
The data-mining revolution started in the mid-1990s. It
was characterized by the incorporation of existing and
already well-established tools and algorithms such as
machine learning. In 1995, the International Conference
on Knowledge Discovery and Data Mining became the
most important annual event for data mining. The frame-
work of data mining also was outlined in many books, such
as Advances in Knowledge Discovery and Data Mining
(Fayyad et al., 1996). Data-mining conferences like ACM
SIGKDD, SPIE, PKDD, and SIAM, and journals like Data
Mining and Knowledge Discovery Journal (1997), Jour-
nal of Knowledge and Information Systems (1999),
and IEEE Transactions on Knowledge and Data En-
gineering (1989) have become an integral part of the
data-mining field.
The trends in data mining over the last few years
include OLAP (Online Analytical Processing), data ware-
housing, association rules, high performance data-min-
ing systems, visualization techniques, and applications
of data mining. Recently, new trends have emerged that
have great potential to benefit the data-mining field, like
XML (eXtensible Markup Language) and XML-related
technologies, database products that incorporate data-
mining tools, and new developments in the design and
TEAM LinG

implementation of the data-mining process. Another im-
portant data-mining issue is concerned with the relation-
ship between theoretical data-mining research and data-
mining applications. Data mining is an exponentially grow-
ing field with a strong emphasis on applications.
A further issue of great importance is the research in
data-mining algorithms and the discussion of issues of
scale (Hearst, 1997). The commonly used tools may not
scale up to huge volumes of data. Scalable data-mining
tools are characterized by the linear increase of their
runtime with the increase of the number of data points
within a fixed amount of available memory. An overview
of scalable data-mining tools is given in Ganti, Gehrke, and
Ramakrishnan (1999). In addition to scalability, robust
techniques to model noisy data sets containing an un-
known number of overlapping categories are of great
importance (Krishnapuram et al., 2001).
MAIN THRUST
Exploiting Neural Networks in Data
Mining
How is data mining able to tell you important things that
you didnt know or tell you what is going to happen next?
The technique that is used to perform these feats is called
modeling. Modeling is simply the act of building a model
based on data from situations where the answer is known,
and then applying the model to other situations where the
answers arent known. Modeling techniques have been
around for centuries, but it is only recently that data
storage and communication capabilities required to col-
lect and to store huge amounts of data and the computa-
tional power to automate modeling techniques to work
directly on the data have been available. Modeling tech-
niques used for data mining include decision trees, rule
induction, genetic algorithms, nearest neighbor, artificial
neural networks, and many other techniques (Chen, Han
& Yu, 1996; Cios, Pedrycz & Swiniarski, 1998; Hand,
Mannila & Smyth, 2000).
Exploiting artificial neural networks as a modeling
technique for data mining is considered to be an important
direction of research. Neural networks can be applied to
a number of data-mining problems, including classifica-
tion, regression, and clustering, and there are quite a few
interesting developments and tools that are being devel-
oped in this field. Lu, Setiono, and Liu (1996) applied
neural networks to mine symbolic classification rules from
large databases. They report that neural networks were
able to deliver a lower error rate and are more robust
against noise than decision trees. Ainslie and Drze
434
Employing Neural Networks in Data Mining
(1996) show how effective data mining can be achieved by
combining the power of neural networks with the rigor of
more traditional statistical tools. They argue that this
alliance can generate important synergies. Craven and
Shavlik (1997) describe neural network learning algo-
rithms for data mining that are able to produce comprehen-
sible models and that do not require excessive training
times. They argue that neural network methods deserve a
place in the toolboxes of data-mining specialists. Mitra,
Pal, and Mitra (2002) provide a survey of the available
literature on data mining using soft computing method-
ologies, including neural networks. They came to the
conclusion that neural networks are suitable in data-rich
environments and are typically used for extracting embed-
ded knowledge in the form of rules, quantitative evalua-
tion of these rules, clustering, self-organization, classifi-
cation, and regression. Vesely (2003) argues that from the
methods of data mining based on neural networks, the
Kohonens self-organizing maps are the most promising,
because, by using a self-organizing map, one can more
easily visualize high-dimensional data. Self-organizing
maps also outperform other conventional methods such
as the popular Principal Component Analysis (PCA)
method for screening analysis of high-dimensional data.
Although highly successful in typical cases, PCA suffers
from the drawback of being a linear method. Furthermore,
real-world data manifolds, besides being nonlinear, often
are corrupted by noise and embed into high-dimensional
spaces. Self-organizing maps are more robust against
noise and are often used to provide representations that
can be analyzed successfully using conventional meth-
ods like PCA.
In spite of their excellent performance in concept
discovery, neural networks do suffer from some short-
comings. They are sensitive to the net topology, initial
weights, and the selection of attributes. If the number of
layers is not selected suitably, the learning efficiency will
be affected. Too many irrelevant nodes can cause unnec-
essary computational expense and overfit (i.e., the net-
work creates meaningless concepts); randomly selected
initial weights sometimes can trap the nets in so-called
pitfalls; that is, neural nets stabilize around local minima
instead of the global minimum. Background knowledge
remains unused in neural nets. The knowledge discov-
ered by nets is not transparent to users. This is perhaps
the main failing of neural networks, as they are unintelli-
gible black boxes.
Our own work is focused on mining educational data
to assist e-learning in a variety of ways. In the following
section, we report on the experience with MASACAD
(Multi-Agent System for ACademic ADvising), a data-
mining, multi-agent system that advises students using
neural networks.
TEAM LinG
Employing Neural Networks in Data Mining
Case Study: Academic Advising of
Students
At the UAE University, there is enormous interest in the
area of online education. Rigorous steps are being taken
toward the creation of the technological infrastructure and
the academic infrastructure for the improvement of teach-
ing and learning. MASACAD, the academic advising sys-
tem described in the following, is to be understood as a tool
that mines educational data to support learning.
The general goal of academic advising is to assist
students in developing educational plans that are consis-
tent with academic, career, and life goals, and to provide
students with information and skills needed to pursue
those goals. In order to improve the advising process and
make it easier, an intelligent assistant in the form of a
computer program will be of great interest. The goal of
academic advising, as stated previously, is too general,
because many experts are involved and a huge amount of
expertise is needed. This makes the realization of such an
assistant too difficult, if not impossible. Therefore, in the
implemented system, the scope of academic advising was
restricted. It was understood as just being intended to
provide the student with an opportunity to plan programs
of study, select appropriate required and elective classes,
and schedule classes in a way that provides the greatest
potential for academic success.
Data Mined by the Advising System
MASACAD
To extract the academic advice (i.e., to provide the student
with a set of appropriate courses he or she should register
for in the coming term), MASACAD has to mine a huge
amount of educational data available in different formats.
The data contain the student profile, which includes the
courses already attended, the corresponding grades, the
desires of the student concerning the courses to be at-
tended, and much other information. The part of the profile
consisting of the courses already attended, the corre-
sponding grades, and so forth, is maintained by the univer-
sity administration in appropriate databases. The part of
the profile consisting of the desires of the student con-
cerning the courses to be attended should be asked for
from the student before advising is performed. The data to
be mined also include the courses that are offered in the
semester for which advising is needed. This information is
maintained by the university administration in appropriate
Web sites. Finally, a very important component of the data
that the system has to mine is expertise. For the problem
of academic advising, expertise consists partly of the
university laws concerning academic advising. These con-
sist of all the details and regulations concerning courses,
programs, and curricula. This kind of information is
published in Web pages, in booklets, and in many other -
forms such as printouts and announcements. The sources
of knowledge are many; however, the primary source will
be a human expert, who should posses more complex
knowledge than can be found in documented sources.
The Advising System MASACAD
MASACAD is a multi-agent system that offers aca-
demic advice to students by mining the educational data
described in the previous section. It consists of a user
system, a grading system, a course announcement sys-
tem, and a mediation agent. The mediation agent pro-
vides the information retrieving service. It moves from
the site of an application to another, where it interacts
with the agent wrappers. The agent wrappers manage the
states of the applications they are wrapped around,
invoking them when necessary. The application grading
system is a database application for answering queries
about the students and the courses they have already
taken. The application course announcement system is
a Web application for answering queries about the
courses that are expected to be offered in the semester
for which advising is needed. The application user
system is the heart of the advising system, and it is here
where intelligence resides. The application gives stu-
dents the opportunity to express their desires concern-
ing the courses to be attended by choosing among the
courses that are offered, initiating a query to obtain
advice, and, finally, seeing the results returned by the
advising system. The system also alerts the user auto-
matically via e-mail when something changes in the
offered courses or in the student profile. The advising
procedure suggests courses according to university
laws, in a way that provides the greatest potential for
academic success, as seen by a human academic advi-
sor. Taking into account the adequacy of the machine-
learning approach for data mining, added to the avail-
ability of experience with advising students, made the
adoption of a paradigm of supervised learning from
examples using artificial neural networks interesting.
For academic advising, the known information (input
variables) consists of the profile of the student and of
the offered courses. The unknown information (output
variables) consists of the advice expected by the stu-
dent. In order for the network to be able to infer the
unknown information, prior training is needed. Train-
ing will integrate the expertise in academic advising
into the network. The back-propagation algorithm was
used for training the neural network.
Information (i.e., training examples) is gained from
information about students and courses they really took
in previous semesters. The selection of these courses
435
TEAM LinG

was made, based on the advice of human experts special-
izing in academic advising. About 250 computer science
students in different stages of study were available for the
learning procedures. Each one of the 250 examples con-
sisted of a pair of input-output vectors. The input vector
summarized all the information needed for advising a
particular student (85 real-valued components; each com-
ponent encodes the information about one of the 85
courses of the curriculum). The output vector encodes the
final decision concerning the courses in which the stu-
dent actually enrolled, based on the advice of the human
academic advisor (85 integer-valued components; each
component represents a priority value for one of the 85
courses of the curriculum, and a higher priority value
indicates a more appropriate course for the student). The
aim of the learning phase was to determine the most
suitable values for the learning rate, the size of the net-
work (number of neurons, number of hidden layers was set
to 2), and the number of training cycles that are needed for
the convergence of the network. Many experiments were
conducted to obtain these parameters and to test the
system. With a network topology of 85-100-100-85 and
systematically selected network parameters (50 different
experiments chosen carefully were performed to obtain
these parameters), the layered, fully connected back-
propagation network was able to deliver a considerable
performance. Fifty students participated in the evaluation
of the system. In 92% of the cases, the network was able
to produce very appropriate advice according to human
experts in academic advising. In the remaining 8% of the
cases (4 cases), some unsatisfactory course suggestions
were produced by the network (Hamdi, 2004).
FUTURE TRENDS
The rapid growth of business, industrial, and educational
data sources has overwhelmed the traditional, interactive
approaches to data analysis and created a need for a new
generation of tools for intelligent and automated discov-
ery in data. Neural networks are well suited for data-
mining tasks, due to their ability to model complex, multi-
dimensional data. As data availability has magnified, so
has the dimensionality of problems to be solved, thus
limiting many traditional techniques, such as manual
examination of the data and some statistical methods.
Although there are many techniques and algorithms that
can be used for data mining, some of which can be used
effectively in combination, neural networks offer many
desirable qualities, such as the automatic search of all
possible interrelationships among key factors, the auto-
matic modeling of complex problems without prior knowl-
edge of the level of complexity, and the ability to extract
436
Employing Neural Networks in Data Mining
key findings much faster than many other tools. As
computer systems become faster, the value of neural
networks as a data-mining tool only will increase.
CONCLUSION
The amount of raw data stored in databases and the Web
is exploding. Raw data by itself, however, does not pro-
vide much information. One benefits when meaningful
trends and patterns are extracted from the data. Data-
mining techniques help to recognize significant facts,
relationships, trends, patterns, exceptions, and anoma-
lies that might otherwise go unnoticed. In this contribu-
tion, we have seen an example of how neural networks can
be used to help mine data about students and courses with
the aim of developing educational plans that are consis-
tent with academic, career, and life goals, and providing
students with information and skills needed to pursue
those goals. The neural network paradigm seems interest-
ing and viable enough to be used as a data-mining tool.
REFERENCES
Ainslie, A., & Drze, X. (1996). Data mining: Using neural
networks as a benchmark for model building. Decision
Marketing, 7, 77-86.
Chen, M.S., Han, J., & Yu, P.S. (1996). Data mining: An
overview from database perspective. IEEE Transactions
on Knowledge and Data Engineering, 8(6), 866-883.
Cios, K.J., Pedrycz, W., & Swiniarski, R.W. (1998). Data
mining methods for knowledge discovery. Norwell, MA:
Kluwer Academic Publishers.
Craven, M.W., & Shavlik, J.W. (1997). Using neural net-
works for data mining. Future Generation Computer
Systems, 13(2-3), 211-229.
Fayyad, U.M., Piatesky-Shapiro, G., Smyth, P., &
Uthurusamy, R. (1996). Advances in knowledge discov-
ery and data mining. Menlo Park, CA: MIT Press.
Ganti, V., Gehrke, J., & Ramakrishnan, R. (1999). Mining
very large databases. IEEE Computer, 32(8), 38-45.
Hamdi, M.S. (2004). MASACAD: A learning multi-agent
system that mines the Web to advise students. Proceed-
ings of the International Conference on Internet Com-
puting (IC04), Las Vegas, Nevada.
Hand, D.J., Mannila, H., & Smyth, P. (2000). Principles of
data mining. Cambridge, MA: MIT Press.
TEAM LinG
Employing Neural Networks in Data Mining
Haykin, S. (1999). Neural networks: A comprehensive
foundation. Upper Saddle River, NJ: Prentice Hall.
Hearst, M. (1997). Distinguishing between web data min-
ing and information access. The Internet <http://
www.sims.berkeley.edu/~hearst/talks/data-mining-
panel/index.htm>
Kobsa, A. (2002). Personalized hypermedia and interna-
tional privacy. Communications of the ACM, 45(5), 64-67.
Krishnapuram, R., Joshi, A., Nasraoui, O., & Yi, L. (2001).
Low complexity fuzzy relational clustering algorithms for Web
mining. IEEE Transactions on Fuzzy Systems, 9(4), 596-607.
Lu, H., Setiono, R., & Liu, H. (1996). Effective data mining
using neural networks. IEEE Transactions on Knowledge
and Data Engineering, 8(6), 957-961.
Mitra, S., Pal, S.K., & Mitra, P. (2002). Data mining in soft
computing framework: A survey. IEEE Transactions On
Neural Networks, 13(1), 3-14.
Musick, R., Fidelis, K., & Slezak, T. (1997). Large-scale
data mining pilot project in human genome. Retrieved
from http://home.comcast.net/~crmusick/papers/
RDOFIS.html
Vesely, A. (2003). Neural networks in data mining.
AGRIC.ECON.-CZECH, 49(9), 427-431.
KEY TERMS
Artificial Neural Networks: Non-linear predictive
models that learn through training and resemble biologi-
cal neural networks in structure.
Classification: The process of dividing a dataset into
mutually exclusive groups such that the members of each -
group are as close as possible to one another, and differ-
ent groups are as far as possible from one another, where
distance is measured with respect to specific variable(s)
one is trying to predict. For example, a typical classifica-
tion problem is to divide a database of companies into
groups that are as homogeneous as possible with respect
to a creditworthiness variable with values good and bad.
Supervised classification is when we know the class
labels and the number of classes.
Clustering: The process of dividing a dataset as in
classification, but the distance is now measured with
respect to all available variables. Unsupervised classifi-
cation is when we do not know the class labels and may
not know the number of classes.
Data Mining: The extraction of hidden predictive
information from large databases.
Decision Tree: A tree-shaped structure that repre-
sents a set of decisions. These decisions generate rules
for the classification of a dataset.
Linear Regression: A classic statistical problem is to
try to determine the relationship between two random
variables X and Y. For example, we might consider height
and weight of a sample of adults. Linear regression at-
tempts to explain this relationship with a straight line fit
to the data.
OLAP: Online analytical processing. Refers to array-
oriented database applications that allow users to view,
navigate through, manipulate, and analyze multidimen-
sional databases.
437
TEAM LinG
438
Enhancing Web Search through Query Log
Mining
Ji-Rong Wen
Microsoft Research Asia, China
INTRODUCTION
Web query log is a type of file keeping track of the
activities of the users who are utilizing a search engine.
Compared to traditional information retrieval setting in
which documents are the only information source avail-
able, query logs are an additional information source in
the Web search setting. Based on query logs, a set of Web
mining techniques, such as log-based query clustering,
log-based query expansion, collaborative filtering and
personalized search, could be employed to improve the
performance of Web search.
BACKGROUND
Web usage mining is an application of data mining tech-
niques to discovering interesting usage patterns from
Web data, in order to understand and better serve the
needs of Web-based applications. Since the majority of
usage data is stored in Web logs, usage mining is usually
also referred to as log mining. Web logs can be divided
into three categories based on the location of data collect-
ing: server log, client log, and proxy log. Server log
provides an aggregate picture of the usage of a service by
all users, while client log provides a complete picture of
usage of all services by a particular client, with the proxy
log being somewhere in the middle (Srivastava, Cooley,
Deshpande, & Tan, 2000).
Query log mining could be viewed as a special kind of
Web usage mining. While there is a lot of work about
mining Website navigation logs for site monitoring, site
adaptation, performance improvement, personalization
and business intelligence, there is relatively little work of
mining search engines query logs for improving Web
search performance. In early years, researchers have
proved that relevance feedback can significantly improve
retrieval performance if users provide sufficient and cor-
rect relevance judgments for queries (Xu & Croft, 2000).
However, in real search scenarios, users are usually
reluctant to explicitly give their relevance feedback. A
large amount of users past query sessions have been
accumulated in the query logs of search engines. Each
query session records a user query and the correspond-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
ing pages the user has selected to browse. Therefore, a
query log can be viewed as a valuable source containing
a large amount of users implicit relevance judgments.
Obviously, these relevance judgments can be used to
more accurately detect users query intentions and im-
prove the ranking of search results.
One important assumption behind query log mining is
that the clicked pages are relevant to the query. Al-
though the clicking information is not as accurate as
explicit relevance judgment in traditional relevance feed-
back, the users choice does suggest a certain degree of
relevance. In the long run with a large amount of log data,
query logs can be treated as a reliable resource containing
abundant implicit relevance judgments from a statistical
point of view.
MAIN THRUST
Web Query Log Preprocessing
Typically, each record in a Web query log includes the IP
address of the client computer, timestamp, the URL of the
requested item, the type of Web browser, protocol, etc.
The Web log of a search engine records various kinds of
user activities, such as submitting queries, clicking URLs
in the result list, getting HTML pages and skipping to
another result list. Although all these activities reflect,
more or less, a users intention, the query terms and the
Web pages the user visited are the most important data for
mining tasks. Therefore, a query session, the basic unit of
mining tasks, is defined as a query submitted to a search
engine together with the Web pages the user visits in
response to the query.
Since the HTTP protocol requires a separate connec-
tion for every client-server interaction, the activities of
multiple users usually interleave with each other. There
are no clear boundaries among user query sessions in the
logs, which makes it a difficult task to extract individual
query sessions from Web query logs (Cooley, Mobasher,
& Srivastava, 1999). There are mainly two steps to extract
query sessions from query logs: user identification and
session identification. User identification is the process
of isolating from the logs the activities associated with an
TEAM LinG
Enhancing Web Search through Query Log Mining
individual user. Activities of the same user could be
grouped by their IP addresses, agent types, site topolo-
gies, cookies, user IDs, etc. The goal of session identifi-
cation is to divide the queries and page accesses of each
user into individual sessions. Finding the beginning of a
query session is trivial: a query session begins when a
user submits a query to a search engine. However, it is
difficult to determine when a search session ends. The
simplest method of achieving this is through a timeout,
where if the time between page requests exceeds a certain
limit, it is assumed that the user is starting a new session.
Log-Based Query Clustering
Query clustering is a technique aiming at grouping users
semantically (not syntactically) related queries in Web
query logs. Query clustering could be applied to FAQ
detecting, index-term selection and query reformulation,
which are effective ways to improve Web search. First of
all, FAQ detecting means to detect Frequently Asked
Questions (FAQs), which can be achieved by clustering
similar queries in the query logs. A cluster being made up
of many queries can be considered as a FAQ. Some search
engines (e.g. Askjeeves) prepare and check the correct
answers for FAQs by human editors, and a significant
majority of users queries can be answered precisely in
this way. Second, inconsistency between term usages in
queries and those in documents is a well-known problem
in information retrieval, and the traditional way of directly
extracting index terms from documents will not be effec-
tive when the user submits queries containing terms
different from those in the documents. Query clustering
is a promising technique to provide a solution to the word
mismatching problem. If similar queries can be recognized
and clustered together, the resulting query clusters will be
very good sources for selecting additional index terms for
documents. For example, if queries such as atomic bomb,
Manhattan Project, Hiroshima bomb and nuclear logs, some very good terms, such as personal com-
weapon are put into a query cluster, this cluster, not the
individual terms, can be used as a whole to index docu-
ments related to atomic bomb. In this way, any queries
contained in the cluster can be linked to these documents.
Third, most words in the natural language have inherent
ambiguity, which makes it quite difficult for user to formu-
late queries with appropriate words. Obviously, query
clustering could be used to suggest a list of alternative
terms for users to reformulate queries and thus better
represent their information needs.
The key problem underlying query clustering is to
determine an adequate similarity function so that truly
similar queries can be grouped together. There are mainly
two categories of methods to calculate the similarity
between queries: one is based on query content, and the
other on query session. Since queries with the same or
similar search intentions may be represented with differ- -
ent words and the average length of Web queries is very
short, content-based query clustering usually does not
perform well.
Using query sessions mined from query logs to cluster
queries is proved to be a more promising method (Wen,
Nie, & Zhang, 2002). Through query sessions, query
clustering is extended to query session clustering.
The basic assumption here is that the activities following
a query are relevant to the query and represent, to some
extent, the semantic features of the query. The query text
and the activities in a query session as a whole can
represent the search intention of the user more precisely.
Moreover, the ambiguity of some query terms is elimi-
nated in query sessions. For instance, if a user visited a
few tourism Websites after submitting a query Java, it
is reasonable to deduce that the user was searching for
information about Java Island, not Java programming
language or Java coffee. Moreover, query clustering
and document clustering can be combined and reinforced
with each other (Beeferman & Berger, 2000).
Log-Based Query Expansion
Query expansion involves supplementing the original
query with additional words and phrases, which is an
effective way to overcome the term-mismatching problem
and to improve search performance. Log-based query
expansion is a new query expansion method based on
query log mining. Taking query sessions in query logs as
a bridge between user queries and Web pages, probabi-
listic correlations between terms in queries and those in
pages can then be established. With these term-term
correlations, relevant expansion terms can be selected
from the documents for a query. For example, a recent work
by Cui, Wen, Nie, and Ma (2003) shows that, from query
puter, Apple Computer, CEO, Macintosh and
graphical user interface, can be detected to be tightly
correlated to the query Steve Jobs, and using these
terms to expand the original query can lead to more
relevant pages.
Experiments by Cui, Wen, Nie, and Ma (2003) show
that mining user logs is extremely useful for improving
retrieval effectiveness, especially for very short queries
on the Web. The log-based query expansion overcomes
several difficulties of traditional query expansion meth-
ods because a large number of user judgments can be
extracted from user logs, while eliminating the step of
collecting feedbacks from users for ad-hoc queries. Log-
based query expansion methods have three other impor-
tant properties. First, the term correlations are pre-
439
TEAM LinG

computed offline and thus the performance is better than
traditional local analysis methods which need to calculate
term correlations on the fly. Second, since user logs
contain query sessions from different users, the term
correlations can reflect the preference of the majority of
the users. Third, the term correlations may evolve along
with the accumulation of user logs. Hence, the query
expansion process can reflect updated user interests at a
specific time.
Collaborative Filtering and Personalized
Web Search
Collaborative filtering and personalized Web search are
two most successful examples of personalization on the
Web. Both of them heavily rely on mining query logs to
detect users preferences or intentions.
Collaborative Filtering
Collaborative filtering is a method of making automatic
predictions (filtering) about the interests of a user by
collecting taste information from many users (collaborat-
ing). The basic assumption of collaborative filtering is that
users having similar tastes on some items may also have
similar preferences on other items. From Web logs, a
collection of items (books, music CDs, movies, etc.) with
users searching, browsing and ranking information could
be extracted to train prediction and recommendation mod-
els. For a new user with a few items he/she likes or dislikes,
other items meeting the users taste could be selected
based on the trained models and recommended to him/her
(Shardanand & Maes, 1995; Konstan, Miller, Maltz,
Herlocker, & Gordon, L. R., et al., 1997). Generally, vector
space models and probabilistic models are two major mod-
els for collaborative filtering (Breese, Heckerman, & Kadie,
1998). Collaborative filtering systems have been imple-
mented by several e-commerce sites including Amazon.
Collaborative filtering has two main advantages over
content-based recommendation systems. First, content-
based annotations of some data types are often not avail-
able (such as video and audio) or the available annotations
usually do not meet the tastes of different users. Second,
through collaborative filtering, multiple users knowledge
and experiences could be remembered, analyzed and shared,
which is a characteristic especially useful for improving
new users information seeking experiences.
The original form of collaborative filtering does not use
the actual content of the items for recommendation, which
may suffers from the scalability, sparsity and synonymy
problems. Most importantly, it is incapable of overcoming
the so-called first-rater problem. The first-rater problem
440
Enhancing Web Search through Query Log Mining
means that objects newly introduced into the system
have not been rated by any users and can therefore not
be recommended. Due to the absence of recommenda-
tions, users tend to not be interested in these new
objects. This in turn has the consequence that the newly
added objects remain in their state of not being recom-
mendable. Combining collaborative and content-based
filtering is therefore a promising approach to solving the
above problems (Baudisch, 1999).
Personalized Web Search
While most modern search engines return identical re-
sults to the same query submitted by different users,
personalized search targets to return results related to
users preferences, which is a promising way to alleviate
the increasing information overload problem on the Web.
The core task of personalization is to obtain the prefer-
ence of each individual user, which is called user profile.
User profiles could be explicitly provided by users or
implicitly learned from users past experiences (Hirsh,
Basu, & Davison, 2000; Mobasher, Cooley, & Srivastava,
2000). For example, Googles personalized search re-
quires users to explicitly enter their preferences. In the
Web context, query log provides a very good source for
learning user profiles since it records the detailed infor-
mation about users past search behaviors. The typical
method of mining a users profile from query logs is to
first collect all of this users query sessions from logs,
and then learn the users preferences on various topics
based on the queries submitted and the pages viewed by
the user (Liu, Yu, & Meng, 2004).
User preferences could be incorporated into search
engines to personalize both their relevance ranking and
their importance ranking. A general way for personaliz-
ing relevance ranking is through query reformulation,
such as query expansion and query modification, based
on user profiles. A main feature of modern Web search
engines is that they assign importance scores to pages
through mining the link structures and the importance
scores will significantly affect the final ranking of search
results. The most famous link analysis algorithms are
PageRank (Page, Brin, Motwani, & Winograd, 1998) and
HITS (Kleinberg, 1998). One main shortcoming of
PageRank is that it assigns a static importance score to
a page, no matter who is searching and what query is
being used. Recently, a topic-sensitive PageRank algo-
rithm (Haveliwala, 2002) and a personalized PageRank
algorithm (Jeh & Widom, 2003) are proposed to calculate
different PageRank scores based on different users
preferences and interests. A search engine using topic-
sensitive PageRank and Personalized PageRank is ex-
pected to retrieve Web pages closer to user preferences.
TEAM LinG
Enhancing Web Search through Query Log Mining
FUTURE TRENDS
Although query log mining is promising to improve the
information seeking process on the Web, the number of
publications in this field is relatively small. The main
reason might lie in the difficulty of obtaining a large
amount of query log data. Due to lack of data, especially
lack of a standard data set, it is difficult to repeat and
compare existing algorithms. This may block knowledge
accumulation in the field. On the other hand, search
engine companies are usually reluctant to make pubic of
their log data because of the costs and/or legal issues
involved. However, to create such a kind of standard log
dataset will greatly benefit both the research community
and the search engine industry, and thus may become a
major challenge of this field in the future.
Nearly all of the current published experiments were
conducted on snapshots of small to medium scale query
logs. In practice, query logs are usually continuously
generated and with huge sizes. To develop scalable and
incremental query log mining algorithms capable of han-
dling the huge and growing dataset is another challenge.
We also foresee that client-side query log mining and
personalized search will attract more and more attentions
from both academy and industry. Personalized informa-
tion filtering and customization could be effective ways to
ease the deteriorated information overload problem.
CONCLUSION
Web query log mining is an application of data mining
techniques to discovering interesting knowledge from
Web query logs. In recent years, some research work and
industrial applications have demonstrated that Web log
mining is an effective method to improve the performance
of Web search. Web search engines have become the
main entries for people to exploit the Web and find needed
information. Therefore, to keep improving the perfor-
mance of search engines and to make them more user-
friendly are important and challenging tasks for research-
ers and developers. Query log mining is expected to play
a more and more important role in enhancing Web search.
REFERENCES
Baudisch, P. (1999). Joining collaborative and content-
based filtering. Proceedings of the Conference on Human
Factors in Computing Systems (CHI99).
Beeferman, D., & Berger. A. (2000). Agglomerative clus-
tering of a search engine query log. Proceedings of the 6th
ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (pp. 407-416). -
Breese, J., Heckerman, D., & Kadie, C. (1998). Empirical
analysis of predictive algorithms for collaborative filter-
ing. Proceedings of the Fourteenth Conference on Un-
certainty in Artificial Intelligence (pp. 43-52).
Cooley, R., Mobasher, B., & Srivastava, J. (1999). Data
preparation for mining World Wide Web browsing pat-
terns. Journal of Knowledge and Information Systems, 1(1).
Cui, H., Wen, J.-R., Nie, J.-Y., & Ma, W.-Y. (2003). Query
expansion by mining user logs. IEEE Transaction on
Knowledge and Data Engineering, 15(4), 829-839.
Haveliala, T. H. (2002). Topic-sensitive PageRank. Pro-
ceeding of the Eleventh World Wide Web conference
(WWW 2002).
Hirsh, H., Basu, C., & Davison, B. D. (2000). Learning to
personalize. Communications of the ACM, 43(8), 102-106.
Jeh, G., & Widom, J. (2003). Scaling personalized Web
search. Proceedings of the Twelfth International World
Wide Web Conference (WWW 2003).
Kleinberg, J. (1998). Authoritative sources in a hyperlinked
environment. Proceedings of the 9th ACM SIAM Interna-
tional Symposium on Discrete Algorithms (pp. 668-677).
Konstan, J. A., Miller, B. N., Maltz, D., Herlocker, J. L.,
Gordon, L. R., & Riedl, J. (1997). GroupLens: applying
collaborative filtering to Usenet news. Communications
of the ACM, 40(3), 77-87.
Liu, F., Yu, C., & Meng, W. (2004). Personalized Web
search for improving retrieval effectiveness. IEEE Transac-
tions on Knowledge and Data Engineering, 16(1), 28-40.
Mobasher, B., Cooley, R., & Srivastava, J. (2000). Auto-
matic personalization based on Web usage mining. Com-
munications of the ACM, 43(8), 142-151.
Page, L., Brin, S., Motwani, R., & Winograd, T. (1998). The
PageRank citation ranking: Bringing order to the Web.
Technical report of Stanford University.
Shardanand, U., & Maes, P. (1995). Social information
filtering: Algorithms for automating word of mouth. Pro-
ceedings of the Conference on Human Factors in Com-
puting Systems.
Srivastava, J., Cooley, R., Deshpande, M., & Tan, P.
(2000). Web usage mining: Discovery and applications of
usage patterns from Web data. SIGKDD Explorations,
1(2), 12-23.
441
TEAM LinG

Wen, J.-R., Nie, J.-Y., & Zhang, H.-J. (2002). Query clus-
tering using user logs. ACM Transactions on Information
Systems (ACM TOIS), 20(1), 59-81.
Xu, J., & Croft, W.B. (2000). Improving the effectiveness
of information retrieval with local context analysis. ACM
Transactions on Information Systems, 18(1), 79-112.
KEY TERMS
Collaborative Filtering: A method of making auto-
matic predictions (filtering) about the interests of a user
by collecting taste information from many users (collabo-
rating).
Log-Based Query Clustering: A technique aiming at
grouping users semantically related queries collected in
Web query logs.
Log-Based Query Expansion: A new query expansion
method based on query log mining. Probabilistic correla-
442
Enhancing Web Search through Query Log Mining
tions between terms in the user queries and those in the
documents can then be established through user logs.
With these term-term correlations, relevant expansion
terms can be selected from the documents for a query.
Log-Based Personalized Search: Personalized search
targets to return results related to users preferences. The
core task of personalization is to obtain the preference of
each individual user, which could be learned from query logs.
Query Log: A type of file keeping track of the activi-
ties of the users who are utilizing a search engine.
Query Log Mining: An application of data mining
techniques to discover interesting knowledge from Web
query logs. The mined knowledge is usually used to
enhance Web search.
Query Session: a query submitted to a search engine
together with the Web pages the user visits in response
to the query. Query session is the basic unit of many query
log mining tasks.
TEAM LinG

Enhancing Web Search through Web
Structure Mining
Ji-Rong Wen
Microsoft Research Asia, China
INTRODUCTION
The Web is an open and free environment for people to
publish and get information. Everyone on the Web can
be either an author, a reader, or both. The language of the
Web, HTML (Hypertext Markup Language), is mainly
designed for information display, not for semantic rep-
resentation. Therefore, current Web search engines
usually treat Web pages as unstructured documents, and
traditional information retrieval (IR) technologies are
employed for Web page parsing, indexing, and search-
ing. The unstructured essence of Web pages seriously
blocks more accurate search and advanced applications
on the Web. For example, many sites contain structured
information about various products. Extracting and inte-
grating product information from multiple Web sites
could lead to powerful search functions, such as com-
parison shopping and business intelligence. However,
these structured data are embedded in Web pages, and
there are no proper traditional methods to extract and
integrate them. Another example is the link structure of
the Web. If used properly, information hidden in the
links could be taken advantage of to effectively improve
search performance and make Web search go beyond
traditional information retrieval (Page, Brin, Motwani,
& Winograd, 1998, Kleinberg, 1998).
Although XML (Extensible Markup Language) is an
effort to structuralize Web data by introducing seman-
tics into tags, it is unlikely that common users are
willing to compose Web pages using XML due to its
complication and the lack of standard schema defini-
tions. Even if XML is extensively adopted, a huge amount
of pages are still written in the HTML format and remain
unstructured. Web structure mining is the class of meth-
ods to automatically discover structured data and infor-
mation from the Web. Because the Web is dynamic,
massive and heterogeneous, automated Web structure
mining calls for novel technologies and tools that may
take advantage of state-of-the-art technologies from various
areas, including machine learning, data mining, information
retrieval, and databases and natural language processing.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
443
-
BACKGROUND
Web structure mining can be further divided into three
categories based on the kind of structured data used.
Web graph mining: Compared to a traditional
document set in which documents are indepen-
dent, the Web provides additional information about
how different documents are connected to each
other via hyperlinks. The Web can be viewed as a
(directed) graph whose nodes are the Web pages
and whose edges are the hyperlinks between them.
There has been a significant body of work on ana-
lyzing the properties of the Web graph and mining
useful structures from it (Page et al., 1998; Kleinberg,
1998; Bharat & Henzinger, 1998; Gibson, Kleinberg,
& Raghavan, 1998). Because the Web graph struc-
ture is across multiple Web pages, it is also called
interpage structure.
Web information extraction (Web IE): In addition,
although the documents in a traditional information
retrieval setting are treated as plain texts with no or
few structures, the content within a Web page does
have inherent structures based on the various HTML
and XML tags within the page. While Web content
mining pays more attention to the content of Web
pages, Web information extraction has focused on
automatically extracting structures with various
accuracy and granularity out of Web pages. Web
content structure is a kind of structure embedded in
a single Web page and is also called intrapage
structure.
Deep Web mining: Besides Web pages that are
accessible or crawlable by following the
hyperlinks, the Web also contains a vast amount of
noncrawlable content. This hidden part of the Web,
referred to as the deep Web or the hidden Web
(Florescu, Levy, & Mendelzon, 1998), comprises
a large number of online Web databases. Compared
to the static surface Web, the deep Web contains a
much larger amount of high-quality structured in-
TEAM LinG

formation (Chang, He, Li, & Zhang, 2003). Auto-
matically discovering the structures of Web data-
bases and matching semantically related attributes
between them is critical to understanding the struc-
tures and semantics of the deep Web sites and to
facilitating advanced search and other applications.
MAIN THRUST
Web Graph Mining
Mining the Web graph has attracted a lot of attention in
the last decade. Some important algorithms have been
proposed and have shown great potential in improving
the performance of Web search. Most of these mining
algorithms are based on two assumptions. (a) Hyperlinks
convey human endorsement. If there exists a link from
page A to page B, and these two pages are authored by
different people, then the first author found the second
page valuable. Thus the importance of a page can be
propagated to those pages it links to. (b) Pages that are
co-cited by a certain page are likely related to the same
topic. Therefore, the popularity or importance of a page
is correlated to the number of incoming links to some
extendt, and related pages tend to be clustered together
through dense linkages among them.
Hub and Authority
In the Web graph, a hub is defined as a page containing
pointers to many other pages, and an authority is de-
fined as a page pointed to by many other pages. An
authority is usually viewed as a good page containing
useful information about one topic, and a hub is usually
a good source to locate information related to one
topic. Moreover, a good hub should contain pointers to
many good authorities, and a good authority should be
pointed to by many good hubs. Such a mutual reinforce-
ment relationship between hubs and authorities is taken
advantage of by an iterative algorithm called HITS
(Kleinberg, 1998). HITS computes authority scores and
hub scores for Web pages in a subgraph of the Web,
which is obtained from the (subset of) search results of
a query with some predecessor and successor pages.
Bharat and Henzinger (1998) addressed three prob-
lems in the original HITS algorithm: mutually rein-
forced relationships between hosts (where certain docu-
ments conspire to dominate the computation), auto-
matically generated links (where no humans opinion is
expressed by the link), and irrelevant documents (where
the graph contains documents irrelevant to the query
topic). They assign each edge of the graph an authority
444
Enhancing Web Search through Web Structure Mining
weight and a hub weight to solve the first problem and
combine connectivity and content analysis to solve the
latter two. Chakrabarti, Joshi, and Tawde (2001) ad-
dressed another problem with HITS: regarding the whole
page as a hub is not suitable, because a page always
contains multiple regions in which the hyperlinks point
to different topics. They proposed to disaggregate hubs
into coherent regions by segmenting the DOM (docu-
ment object model) tree of an HTML page.
PageRank
The main drawback of the HITS algorithm is that the hubs
and authority score must be computed iteratively from
the query result on the fly, which does not meet the real-
time constraints of an online search engine. To over-
come this difficulty, Page et al. (1998) suggested using
a random surfing model to describe the probability that
a page is visited and taking the probability as the impor-
tance measurement of the page. They approximated this
probability with the famous PageRank algorithm, which
computes the probability scores in an iterative manner.
The main advantage of the PageRank algorithm over the
HITS algorithm is that the importance values of all pages
are computed off-line and can be directly incorporated
into ranking functions of search engines.
Noisy link and topic drifting are two main problems
in the classic Web graph mining algorithms. Some links,
such as banners, navigation panels, and advertisements,
can be viewed as noise with respect to the query topic
and do not carry human editorial endorsement. Also,
hubs may be mixed, which means that only a portion of
the hub content may be relevant to the query. Most link
analysis algorithms treat each Web page as an atomic,
indivisible unit with no internal structure. This leads to
false reinforcements of hub/authority and importance
calculation. Cai, He, Wen, and Ma (2004) used a vision-
based page segmentation algorithm to partition each
Web page into blocks. By extracting the page-to-block,
block-to-page relationships from the link structure and
page layout analysis, a semantic graph over the Web can
be constructed such that each node exactly represents a
single semantic topic. This graph can better describe the
semantic structure of the Web. Based on block-level
link analysis, they proposed two new algorithms, Block
Level PageRank and Block Level HITS, whose perfor-
mances are shown to exceed the classic PageRank and
HITS algorithms.
Community Mining
Many communities, either in an explicit or implicit
form, exist in the Web today, and their number is grow-
TEAM LinG
Enhancing Web Search through Web Structure Mining
ing at a very fast speed. Discovering communities from a
network environment such as the Web has recently be-
come an interesting research problem. The Web can be
abstracted into directional or nondirectional graphs with
nodes and links. It is usually rather difficult to understand
a networks nature directly from its graph structure, par-
ticularly when it is a large scale complex graph. Data mining
is a method to discover the hidden patterns and knowledge
from a huge network. The mined knowledge could provide
a higher logical view and more precise insight of the nature
of a network and will also dramatically decrease the dimen-
sionality when trying to analyze the structure and evolu-
tion of the network.
Quite a lot of work has been done in mining the
implicit communities of users, Web pages, or scientific
literature from the Web or document citation database
using content or link analysis. Several different defini-
tions of community were also raised in the literature. In
Gibson et al. (1998), a Web community is a number of
representative authority Web pages linked by important
hub pages that share a common topic. Kumar, Raghavan,
Rajagopalan, and Tomkins (1999) define a Web commu-
nity as a highly linked bipartite subgraph with at least one
core containing complete bipartite subgraph. In Flake,
Lawrence, and Lee Giles (2000), a set of Web pages that
linked more pages in the community than those outside
of the community could be defined as a Web community.
Also, a research community could be based on a single-
most-cited paper and could contain all papers that cite it
(Popescul, Flake, Lawrence, Ungar, & Lee Giles, 2000).
Web Information Extraction
Web IE has the goal of pulling out information from a
collection of Web pages and converting it to a homoge-
neous form that is more readily digested and analyzed for
both humans and machines. The results of IE could be
used to improve the indexing process, because IE re-
moves irrelevant information in Web pages and facili-
tates other advanced search functions due to the struc-
tured nature of data. The structuralization degrees of
Web pages are diverse. Some pages can be just taken as
plain text documents. Some pages contain a little loosely
structured data, such as a product list in a shopping page
or a price table in a hotel page. Some pages are organized
with more rigorous structures, such as the home pages of
the professors in a university. Other pages have very strict
structures, such as the book description pages of Amazon,
which are usually generated by a uniform template.
Therefore, basically two kinds of Web IE techniques
exist: IE from unstructured pages and IE from
semistructured pages. IE tools for unstructured pages
are similar to those classical IE tools that typically use
natural language processing techniques such as syntactic
analysis, semantic analysis, and discourse analysis. IE
tools for semistructured pages are different from the -
classical ones, as IE utilizes available structured informa-
tion, such as HTML tags and page layouts, to infer the
data formats or pages. Such a kind of methods is also
called wrapper induction (Kushmerick, Weld, &
Doorenbos, 1997; Cohen, Hurst, & Jensen, 2002). In
contrast to classic IE approaches, wrapper induction
operates less dependently of the specific contents of
Web pages and mainly focuses on page structure and
layout. Existing approaches for Web IE mainly include
the manual approach, supervised learning, and unsuper-
vised learning. Although some manually built wrappers
exist, supervised learning and unsupervised learning
are viewed as more promising ways to learn robust and
scalable wrappers, because building IE tools manually
is not feasible and scalable for the dynamic, massive and
diverse Web contents. Moreover, because supervised
learning still relies on manually labeled sample pages and
thus also requires substantial human effort, unsuper-
vised learning is the most suitable method for Web IE.
There have been several successful fully automatic IE
tools using unsupervised learning (Arasu & Garcia-
Molina, 2003; Liu, Grossman, & Zhai, 2003).
Deep Web Mining
In the deep Web, it is usually difficult or even impos-
sible to directly obtain the structures (i.e. schemas) of the
Web sites backend databases without cooperation from
the sites. Instead, the sites present two other distin-
guishing structures, interface schema and result schema,
to users. The interface schema is the schema of the query
interface, which exposes attributes that can be queried in
the backend database. The result schema is the schema
of the query results, which exposes attributes that are
shown to users.
The interface schema is useful for applications, such
as a mediator that queries multiple Web databases, be-
cause the mediator needs complete knowledge about the
search interface of each database. The result schema is
critical for applications, such as data extraction, where
instances in the query results are extracted. In addition
to the importance of the interface schema and result
schema, attribute matching across different schemas is
also important. First, matching between different inter-
face schemas and matching between different results
schemas (intersite schema matching) are critical for
metasearching and data-integration among related Web
databases. Second, matching between the interface
schema and the result schema of a single Web database
(intrasite schema matching) enables automatic data an-
notation and database content crawling.
445
TEAM LinG

Most existing schema-matching approaches for Web
databases primarily focus on matching query interfaces
(He & Chang, 2003; He, Meng, Yu, & Wu, 2003; Raghavan
& Garcia-Molina, 2001). They usually adopt a label-
based strategy to identify attribute labels from the de-
scriptive text surrounding interface elements and then
find synonymous relationships between the identified
labels. The performance of these approaches may be
affected when no attribute description can be identified
or when the identified description is not informative. In
Wang, Wen, Lochovsky, and Ma (2004), an instance-
based schema-matching approach was proposed to iden-
tify both the interface and result schemas of Web data-
bases. Instance-based approaches depend on the content
overlap or statistical properties, such as data ranges and
patterns, to determine the similarity of two attributes.
Thus, they could effectively deal with the cases where
attribute names or labels are missing or not available,
which are common for Web databases.
FUTURE TRENDS
It is foreseen that the biggest challenge in the next
several decades is how to effectively and efficiently dig
out a machine-understandable information and knowl-
edge layer from unorganized and unstructured Web data.
However, Web structure mining techniques are still in
their youth today. For example, the accuracy of Web
information extraction tools, especially those auto-
matically learned tools, is still not satisfactory to meet
the requirements of some rigid applications. Also, deep
Web mining is a new area, and researchers have many
challenges and opportunities to further explore, such as
data extraction, data integration, schema learning and
matching, and so forth. Moreover, besides Web pages,
various other types of structured data exist on the Web,
such as e-mail, newsgroup, blog, wiki, and so forth.
Applying Web mining techniques to extract structures
from these data types is also a very important future
research direction.
CONCLUSION
Despite the efforts of XML and semantic Web, which
target to bring structures and semantics to the Web,
Web structure mining is considered a more promising
way to structuralize the Web due to its characteristics of
automation, scalability, generality, and robustness. As a
result, there has been a rapid growth of technologies for
automatically discovering structures from the Web, namely
Web graph mining, Web information extraction, and deep
Web mining. The mined information and knowledge will
446
Enhancing Web Search through Web Structure Mining
greatly improve the effectiveness of current Web search
and will enable much more sophisticated Web information
retrieval technologies in the future.
REFERENCES
Arasu, A., & Garcia-Molina, H. (2003). Extracting struc-
tured data from Web pages. Proceedings of the ACM
SIGMOD International Conference on Management
of Data.
Bharat, K., & Henzinger, M. R. (1998). Improved algo-
rithms for topic distillation in a hyperlinked environ-
ment. Proceedings of the 21st Annual International
ACM SIGIR Conference.
Cai, D., He, X., Wen J.-R., & Ma, W.-Y. (2004). Block-
level link analysis. Proceedings of the 27th Annual
International ACM SIGIR Conference.
Chakrabarti, S., Joshi, M., & Tawde, V. (2001). En-
hanced topic distillation using text, markup tags, and
hyperlinks. Proceedings of the 24th Annual Interna-
tional ACM SIGIR Conference (pp. 208-216).
Chang, C. H., He, B., Li, C., & Zhang, Z. (2003). Structured
databases on the Web: Observations and implications
discovery (Tech. Rep. No. UIUCCDCS-R-2003-2321). Ur-
bana-Champaign, IL: University of Illinois, Department of
Computer Science.
Cohen, W., Hurst, M., & Jensen, L. (2002). A flexible
learning system for wrapping tables and lists in HTML
documents. Proceedings of the 11th World Wide Web
Conference.
Flake, G. W., Lawrence, S., & Lee Giles, C. (2000).
Efficient identification of Web communities. Proceed-
ings of the Sixth International Conference on Knowl-
edge Discovery and Data Mining.
Florescu, D., Levy, A. Y., & Mendelzon, A. O. (1998).
Database techniques for the World Wide Web: A sur-
vey. SIGMOD Record, 27(3), 59-74.
Gibson, D., Kleinberg, J., & Raghavan, P. (1998). Inferring
Web communities from link topology. Proceedings of the
Ninth ACM Conference on Hypertext and Hypermedia.
He, B., & Chang, C. C. (2003). Statistical schema matching
across Web query interfaces. Proceedings of the ACM
SIGMOD International Conference on Management of
Data.
He, H., Meng, W., Yu, C., & Wu, Z. (2003). WISE-Integra-
tor: An automatic integrator of Web search interfaces for
TEAM LinG
Enhancing Web Search through Web Structure Mining
e-commerce. Proceedings of the 29th International Con-
ference on Very Large Data Bases.
Kleinberg, J. (1998). Authoritative sources in a hyperlinked
environment. Proceedings of the Ninth ACM SIAM Inter-
national Symposium on Discrete Algorithms (pp. 668-
677).
Kumar, R., Raghavan, P., Rajagopalan, S., & Tomkins, A.
(1999). Trawling the Web for emerging cyber-commu-
nities. Proceedings of the Eighth International World
Wide Web Conference.
Kushmerick, N., Weld, D., & Doorenbos, R. (1997).
Wrapper induction for information extraction. Pro-
ceedings of the International Joint Conference on
Artificial Intelligence.
Liu, B., Grossman, R., & Zhai, Y. (2003). Mining data
records in Web pages. Proceedings of the ACM SIGKDD
International Conference on Knowledge Discovery &
Data Mining.
Page, L., Brin, S., Motwani, R., & Winograd, T. (1998).
The PageRank citation ranking: Bringing order to the
Web (Tech. Rep.). Stanford University.
Popescul, A., Flake, G. W., Lawrence, S., Ungar, L. H.,
& Lee Giles, C. (2000). Clustering and identifying
temporal trends in document databases. Proceedings of
the IEEE Conference on Advances in Digital Librar-
ies.
Raghavan, S., & Garcia-Molina, H. (2001). Crawling the
hidden Web. Proceedings of the 27th International
Conference on Very Large Data Bases.
Wang, J., Wen, J.-R., Lochovsky, F., & Ma, W.-Y. (2004).
Instance-based schema matching for Web databases by -
domain-specific query probing. Proceedings of the 30th
International Conference on Very Large Data Bases.
KEY TERMS
Community Mining: A Web graph mining algorithm
to discover communities from the Web graph in order to
provide a higher logical view and more precise insight of
the nature of the Web.
Deep Web Mining: Automatically discovering the
structures of Web databases hidden in the deep Web and
matching semantically related attributes between them.
HITS: A Web graph mining algorithm to compute
authority scores and hub scores for Web pages.
PageRank: A Web graph mining algorithm that uses
the probability that a page is visited by a random surfer
on the Web as a key factor for ranking search results.
Web Graph Mining: The mining techniques used to
discover knowledge from the Web graph.
Web Information Extraction: The class of mining
methods to pull out information from a collection of
Web pages and converting it to a homogeneous form
that is more readily digested and analyzed for both
humans and machines.
Web Structure Mining: The class of methods used
to automatically discover structured data and informa-
tion from the Web.
447
TEAM LinG
448
Ensemble Data Mining Methods
Nikunj C. Oza
NASA Ames Research Center, USA
INTRODUCTION
Ensemble data mining methods, also known as committee
methods or model combiners, are machine learning meth-
ods that leverage the power of multiple models to achieve
better prediction accuracy than any of the individual
models could on their own. The basic goal when design-
ing an ensemble is the same as when establishing a
committee of people: Each member of the committee should
be as competent as possible, but the members should
complement one another. If the members are not comple-
mentary, that is, if they always agree, then the committee
is unnecessary any one member is sufficient. If the
members are complementary, then when one or a few
members make an error, the probability is high that the
remaining members can correct this error. Research in
ensemble methods has largely revolved around designing
ensembles consisting of competent yet complementary
models.
BACKGROUND
A supervised machine learning task involves construct-
ing a mapping from input data (normally described by
several features) to the appropriate outputs. In a classi-
fication learning task, each output is one or more classes
to which the input belongs. The goal of classification
learning is to develop a model that separates the data into
the different classes, with the aim of classifying new
examples in the future. For example, a credit card company
may develop a model that separates people who defaulted
on their credit cards from those who did not, based on
other known information such as annual income. The goal
would be to predict whether a new credit card applicant is
likely to default on his or her credit card and thereby
decide whether to approve or deny this applicant a new
card. In a regression learning task, each output is a
continuous value to be predicted (e.g., the average bal-
ance that a credit card holder carries over to the next
month).
Many traditional machine learning algorithms gener-
ate a single model (e.g., a decision tree or neural network).
Ensemble learning methods instead generate multiple
models. Given a new example, the ensemble passes it to
each of its multiple base models, obtains their predictions,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
and then combines them in some appropriate manner (e.g.,
averaging or voting). As mentioned earlier, it is important
to have base models that are competent but also comple-
mentary. To further motivate this point, consider Figure
1. This figure depicts a classification problem in which the
goal is to separate the points marked with plus signs from
points marked with minus signs. None of the three indi-
vidual linear classifiers (marked A, B, and C) is able to
separate the two classes of points. However, a majority
vote over all three linear classifiers yields the piecewise-
linear classifier shown as a thick line. This classifier is able
to separate the two classes perfectly. For example, the
plusses at the top of the figure are correctly classified by
A and B, but are misclassified by C. The majority vote over
these classifiers correctly identifies them as plusses. This
happens because A and B are very different from C. If our
ensemble instead consisted of three copies of C, then all
three classifiers would misclassify the plusses at the top
of the figure, and so would a majority vote over these
classifiers.
MAIN THRUST
We now discuss the key elements of an ensemble-learn-
ing method and ensemble model and, in the process,
discuss several ensemble methods that have been devel-
oped.
Ensemble Methods
The example shown in Figure 1 is an artificial example. We
cannot normally expect to obtain base models that
misclassify examples in completely separate parts of the
input space and ensembles that classify all the examples
correctly. However, many algorithms attempt to generate
a set of base models that make errors that are as uncorrelated
as possible. Methods such as bagging (Breiman, 1994)
and boosting (Freund & Schapire, 1996) promote diver-
sity by presenting each base model with a different subset
of training examples or different weight distributions over
the examples. For example, in Figure 1, if the plusses in the
top part of the figure were temporarily removed from the
training set, then a linear classifier learning algorithm
trained on the remaining examples would probably yield
a classifier similar to C. On the other hand, removing the
TEAM LinG
Ensemble Data Mining Methods
Figure 1. An ensemble of linear classifiers
plusses in the bottom part of the figure would probably
yield classifier B, or something similar. In this way, run-
ning the same learning algorithm on different subsets of
training examples can yield very different classifiers,
which can be combined to yield an effective ensemble.
Input decimation ensembles (IDE) (Oza & Tumer, 2001;
Tumer & Oza, 2003) and stochastic attribute selection
committees (SASC) (Zheng & Webb, 1998) instead pro-
mote diversity by training each base model with the same
training examples but different subsets of the input fea-
tures. SASC trains each base model with a random subset
of input features. IDE selects, for each class, a subset of
features that has the highest correlation with the presence
of that class. Each feature subset is used to train one base
model. However, in both SASC and IDE, all the training
patterns are used with equal weight to train all the base
models.
So far we have distinguished ensemble methods by
the way they train their base models. We can also distin-
guish methods by the way they combine their base mod-
els predictions. Majority or plurality voting is frequently
used for classification problems and is used in bagging.
If the classifiers provide probability values, simple aver-
aging is commonly used and is very effective (Tumer &
Ghosh, 1996). Weighted averaging has also been used,
and different methods for weighting the base models have
been examined. Two particularly interesting methods for
weighted averaging include mixtures of experts (Jordan &
Jacobs, 1994) and Merzs use of principal components
analysis (PCA) to combine models (Merz, 1999). In the
mixtures of experts method, the weights in the weighted
average combination are determined by a gating network,
which is a model that takes the same inputs that the base
models take and returns a weight on each of the base
models. The higher the weight for a base model, the more
that base model is trusted to provide the correct answer.
These weights are determined during training by how well
the base models perform on the training examples. The
gating network essentially keeps track of how well each
base model performs in each part of the input space. The -
hope is that each model learns to specialize in different
input regimes and is weighted highly when the input falls
into its specialty. Merzs method uses PCA to lower the
weights of base models that perform well overall but are
redundant and, therefore, effectively give too much weight
to one model. For example, in Figure 1, if an ensemble of
three models instead had two copies of A and one copy of
B, we may prefer to lower the weights of the two copies of
A because, essentially, A is being given too much weight.
Here, the two copies of A would always outvote B, thereby
rendering B useless. Merzs method also increases the
weight on base models that do not perform as well overall
but perform well in parts of the input space, where the
other models perform poorly. In this way, a base models
unique contributions are rewarded.
When designing an ensemble learning method, in
addition to choosing the method by which to bring about
diversity in the base models and choosing the combining
method, one has to choose the type of base model and
base model learning algorithm to use. The combining
method may restrict the types of base models that can be
used. For example, to use average combining in a classi-
fication problem, one must have base models that can
yield probability estimates. This precludes the use of
linear discriminant analysis or support vector machines,
which cannot return probabilities. The vast majority of
ensemble methods use only one base model learning
algorithm but use the methods described earlier to bring
about diversity in the base models. Surprisingly, little
work has been done (e.g., Merz, 1999) on creating en-
sembles with many different types of base models.
Two of the most popular ensemble learning algorithms
are bagging and boosting, which we briefly explain next.
Bagging
Bootstrap aggregating (Bagging) generates multiple
bootstrap training sets from the original training set (by
using sampling with replacement) and uses each of them
to generate a classifier for inclusion in the ensemble. The
algorithms for bagging and sampling with replacement are
given in Figure 2. In these algorithms, T is the original
training set of N examples, M is the number of base models
to be learned, L b is the base model learning algorithm, the
his are the base models, random_integer (a,b) is a func-
tion that returns each of the integers from a to b with equal
probability, and I(A) is the indicator function that returns
1 if A is true and 0 otherwise.
To create a bootstrap training set from an original
training set of size N, we perform N multinomial trials,
where in each trial, we draw one of the N examples. Each
example has the probability 1/N of being drawn in each
449
TEAM LinG

Figure 2. Batch bagging algorithm and sampling with
replacement
Bagging (T , M )
For each m = 1, 2,K , M
Tm = Sample_With_Replacement(T ,| T |)
hm = Lb (Tm )
M If m 1/2 then,
Return h fin ( x) = arg max yY I (h
m =1
m ( x) = y ).
Sample_With_Replacement(T , N )
S = {}
For i = 1, 2,K , N
r = random_integer(1, N )
Add T [r ] to S .
Return S .
trial. The second algorithm shown in Figure 2 does exactly
this N times, the algorithm chooses a number r from 1 to
N and adds the rth training example to the bootstrap
training set S. Clearly, some of the original training ex-
amples will not be selected for inclusion in the bootstrap
training set, and others will be chosen one time or more. In
bagging, we create M such bootstrap training sets and
then generate classifiers using each of them. Bagging
returns a function h fin ( x) that classifies new examples by
returning the class y that gets the maximum number of
votes from the base models h1,h2,,hM. In bagging, the M
bootstrap training sets that are created are likely to have
some differences. If these differences are enough to in-
duce noticeable differences among the M base models
while leaving their performances reasonably good, then
the ensemble will probably perform better than the base
models individually. Breiman (1994) demonstrates that
bagged ensembles tend to improve upon their base models
more if the base model learning algorithms are unstable
differences in their training sets tend to induce significant
differences in the models. He notes that decision trees are
unstable, which explains why bagged decision trees often
outperform individual decision trees; however, decision
stumps (decision trees with only one variable test) are
stable, which explains why bagging with decision stumps
tends to not improve upon individual decision stumps.
450
Ensemble Data Mining Methods
Figure 3. Batch boosting algorithm
AdaBoost({(x1, y1 ),(x 2 , y 2 ),K,(x N , y N )},Lb , M)
Initialize D1 (n ) = 1/N for all n {1,2,K,N }.
For each m = 1,2,K, M :
hm = Lb ({(x1, y1 ),(x 2 , y 2 ),K,(x N , y N )},Dm ).
m = Dm (n ).
n:h m (x n )y n
set M = m 1 and abort this loop.
Update distribution Dm :
1
2 1 if hm (x n ) = y n
( m)
Dm +1 (n) = Dm (n )
1
otherwise.
2m
M 1 m
Return h fin (x) = argmaxy Y I(hm (x) = y)log .
m=1 m
Boosting
We now explain the AdaBoost algorithm, because it is
the most frequently used among all boosting algorithms.
AdaBoost generates a sequence of base models with
different weight distributions over the training set. The
AdaBoost algorithm is shown in Figure 3. Its inputs are
a set of N training examples, a base model learning
algorithm Lb, and the number M of base models that we
wish to combine. AdaBoost was originally designed for
two-class classification problems; therefore, for this ex-
planation, we will assume that two possible classes exist.
However, AdaBoost is regularly used with a larger num-
ber of classes. The first step in AdaBoost is to construct
an initial distribution of weights D1 over the training set.
This distribution assigns equal weight to all N training
examples. We now enter the loop in the algorithm. To
construct the first base model, we call Lb with distribution
D1 over the training set.1 After getting back a model h1, we
calculate its error e1 on the training set itself, which is just
the sum of the weights of the training examples that h1
misclassifies. We require that e1 < 1/2 (this is the weak
learning assumption the error should be less than
what we would achieve through randomly guessing the
class2). If this condition is not satisfied, then we stop and
return the ensemble consisting of the previously gener-
TEAM LinG
Ensemble Data Mining Methods
ated base models. If this condition is satisfied, then we
calculate a new distribution, D2, over the training examples
as follows. Examples that were correctly classified by h1
have their weights multiplied by 1/(2(1-e1). Examples that
were misclassified by h1 have their weights multiplied by
1/(2e1). Note that because of our condition e1 < 1/2,
correctly classified examples have their weights reduced,
and misclassified examples have their weights increased.
Specifically, examples that h1 misclassified have their total
weight increased to 1/2 under D2, and examples that h1
correctly classified have their total weight reduced to 1/
2 under D2. We then go into the next iteration of the loop
to construct base model h2 using the training set and the
new distribution D2. The point is that the next base model
will be generated by a weak learner (i.e., the base model will
have an error less than 1/2); therefore, at least some of the
examples misclassified by the previous base model will
have to be correctly classified by the current base model.
In this way, boosting forces subsequent base models to
correct the mistakes made by earlier models. We construct
M base models in this fashion. The ensemble returned by
AdaBoost is a function that takes a new example as input
and returns the class that gets the maximum weighted vote
over the M base models, where each base models weight
is log((1-em)/em), which is proportional to the base models
accuracy on the weighted training set presented to it.
AdaBoost has performed very well in practice and is
one of the few theoretically motivated algorithms that has
turned into a practical algorithm. However, AdaBoost can
perform poorly when the training data is noisy (Dietterich,
2000); that is, the inputs or outputs have been randomly
contaminated. Noisy examples are normally difficult to
learn. Because of this fact, the weights assigned to noisy
examples often become much higher than for the other
examples, often causing boosting to focus too much on
those noisy examples at the expense of the remaining data.
Some work has been done to mitigate the effect of noisy
examples on boosting (Oza, 2003, 2004; Ratsch, Onoda, &
Muller, 2001).
FUTURE TRENDS
The fields of machine learning and data mining are in-
creasingly moving away from working on small datasets
in the form of flat files that are presumed to describe a
single process. They are changing their focus toward the
types of data increasingly being encountered today: very
large datasets, possibly distributed over different loca-
tions, describing operations with multiple regimes of
operation, time-series data, online applications (the data
is not a time series but nevertheless arrives continually
and must be processed as it arrives), partially labeled data,
and documents. Research in ensemble methods is begin-
ning to explore these new types of data. For example, -
ensemble learning traditionally has required access to the
entire dataset at once; that is, it performs batch learning.
However, this idea is clearly impractical for very large
datasets that cannot be loaded into memory all at once.
Oza and Russell (2001) and Oza (2001) apply ensemble
learning to such large datasets. In particular, this work
develops online bagging and boosting; that is, they learn
in an online manner. Whereas standard bagging and
boosting require at least one scan of the dataset for every
base model created, online bagging and online boosting
require only one scan of the dataset, regardless of the
number of base models. Additionally, as new data arrive,
the ensembles can be updated without reviewing any past
data. However, because of their limited access to the data,
these online algorithms do not perform as well as their
standard counterparts. Other work has also been done to
apply ensemble methods to other types of data, such as
time-series data (Weigend, Mangeas, & Srivastava, 1995).
However, most of this work is experimental. Theoretical
frameworks that can guide us in the development of new
ensemble learning algorithms specifically for modern
datasets have yet to be developed.
CONCLUSION
Ensemble methods began about 10 years ago as a separate
area within machine learning and were motivated by the
idea of wanting to leverage the power of multiple models
and not just trust one model built on a small training set.
Significant theoretical and experimental developments
have occurred over the past 10 years and have led to
several methods, especially bagging and boosting, being
used to solve many real problems. However, ensemble
methods also appear to be applicable to current and
upcoming problems of distributed data mining and online
applications. Therefore, practitioners in data mining should
stay tuned for further developments in the vibrant area of
ensemble methods. An excellent way to do this is to follow
the series of workshops called the International Work-
shop on Multiple Classifier Systems. This series balance
between theory, algorithms, and applications of ensemble
methods gives a comprehensive idea of the work being
done in the field.
REFERENCES
Breiman, L. (1994). Bagging predictors (Tech. Rep. 421).
Berkeley: University of California, Department of Statis-
tics.
451
TEAM LinG

Dietterich, T. (2000). An experimental comparison of three
methods for constructing ensembles of decision trees:
Bagging, boosting, and randomization. Machine Learn-
ing, 40, 139-158.
Freund, Y., & Schapire, R. (1996). Experiments with a new
boosting algorithm. In M. Kaufmann (Ed.), Proceedings
of the 13th International Conference on Machine Learn-
ing (pp. 148-156). Bari, Italy: Morgan Kaufmann Publish-
ers.
Jordan, M. I., & Jacobs, R. A. (1994). Hierarchical mixture
of experts and the EM algorithm. Neural Computation, 6,
181-214.
Merz, C. J. (1999). A principal component approach to
combining regression estimates. Machine Learning, 36,
9-32.
Oza, N. C. (2001). Online ensemble learning. Unpublished
doctoral dissertation, University of California, Berkeley.
Oza, N. C. (2003). Boosting with averaged weight vectors.
In T. Windeatt & F. Roli (Eds.), Proceedings of the Fourth
International Workshop on Multiple Classifier Systems
(pp. 15-24). Guildford, UK: Springer-Verlag.
Oza, N. C. (2004). AveBoost2: Boosting with noisy data.
In F. Roli, J. Kittler, & T. Windeatt (Eds.), Proceedings of
the Fifth International Workshop on Multiple Classifier
Systems (pp. 31-40). Cagliari, Italy: Springer-Verlag.
Oza, N. C., & Russell, S. (2001). Experimental comparisons
of online and batch versions of bagging and boosting.
Proceedings of the Seventh ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining,
USA, (pp. 359-364). ACM Press.
Oza, N. C., & Tumer, K. (2001). Input decimation en-
sembles: Decorrelation through dimensionality reduc-
tion. Proceedings of the Second International Workshop
on Multiple Classifier Systems, Berlin (pp. 238-247).
Springer-Verlag.
Ratsch, G., Onoda, T., & Muller, K. R. (2001). Soft margins
for AdaBoost. Machine Learning, 42, 287-320.
Tumer, K., & Ghosh, J. (1996). Error correlation and error
reduction in ensemble classifiers. Connection Science,
8(3-4), 385-404.
Tumer, K., & Oza, N. C. (2003). Input decimated en-
sembles. Pattern Analysis and Applications, 6(1), 65-77.
Weigend, A. S., Mangeas, M., & Srivastava, A. N. (1995).
Nonlinear gated experts for time-series: Discovering re-
gimes and avoiding overfitting. International Journal of
Neural Systems, 6(4), 373-399.
452
Ensemble Data Mining Methods
Zheng, Z., & Webb, G. (1998). Stochastic attribute selec-
tion committees. Proceedings of the 11th Australian
Joint Conference on Artificial Intelligence (pp. 321-332),
Brisbane, Australia. Springer-Verlag.
KEY TERMS
Batch Learning: Learning by using an algorithm that
views the entire dataset at once and can access any part
of the dataset at any time and as many times as desired.
Decision Tree: A model consisting of nodes that
contain tests on a single attribute and branches repre-
senting the different outcomes of the test. A prediction is
generated for a new example by performing the test de-
scribed at the root node and then proceeding along the
branch that corresponds to the outcome of the test. If the
branch ends in a prediction, then that prediction is re-
turned. If the branch ends in a node, then the test at that
node is performed and the appropriate branch selected.
This continues until a prediction is found and returned.
Ensemble: A function that returns a combination of
the predictions of multiple machine learning models.
Machine Learning: The branch of artificial intelli-
gence devoted to enabling computers to learn.
Neural Network: A nonlinear model derived through
analogy with the human brain. It consists of a collection
of elements that linearly combine their inputs and pass the
result through a nonlinear transfer function.
Online Learning: Learning by using an algorithm that
only examines the dataset once in order. This paradigm is
often used in situations when data arrives continually in
a stream and when predictions must be obtainable at any
time.
Principal Components Analysis (PCA): Given a
dataset, PCA determines the axes of maximum variance.
For example, if the dataset were shaped like an egg, then
the long axis of the egg would be the first principal
component, because the variance is greatest in this direc-
tion. All subsequent principal components are found to
be orthogonal to all previous components.
ENDNOTES
1
If Lb cannot take a weighted training set, then one
can call it with a training set that is generated by
sampling with replacement from the original training
set according to the distribution Dm.
2
This requirement is perhaps too strict when more
than two classes exist. AdaBoost has a multiclass
TEAM LinG
Ensemble Data Mining Methods

version (Freund & Schapire, 1997) that does not

have this requirement. However, the AdaBoost al- -
gorithm presented here is often used even when
more than two classes exist, if the base model learn-
ing algorithm is strong enough to satisfy the re-
quirement.

453

TEAM LinG
454
Ethics of Data Mining
Jack Cook
Rochester Institute of Technology, USA
INTRODUCTION
Decision makers thirst for answers to questions. As more
data is gathered, more questions are posed: Which cus-
tomers are most likely to respond positively to a marketing
campaign, product price change or new product offering?
How will the competition react? Which loan applicants are
most likely or least likely to default? The ability to raise
questions, even those that currently cannot be answered,
is a characteristic of a good decision maker. Decision
makers no longer have the luxury of making decisions
based on gut feeling or intuition. Decisions must be
supported by data; otherwise decision makers can expect
to be questioned by stockholders, reporters, or attorneys
in a court of law. Data mining can support and often direct
decision makers in ways that are often counterintuitive.
Although data mining can provide considerable insight,
there is an inherent risk that what might be inferred may
be private or ethically sensitive (Fule & Roddick, 2004, p. 159).
Extensively used in telecommunications, financial
services, insurance, customer relationship management
(CRM), retail, and utilities, data mining more recently has
been used by educators, government officials, intelli-
gence agencies, and law enforcement. It helps alleviate
data overload by extracting value from volume. However,
data analysis is not data mining. Query-driven data analy-
sis, perhaps guided by an idea or hypothesis, that tries to
deduce a pattern, verify a hypothesis, or generalize infor-
mation in order to predict future behavior is not data
mining (Edelstein, 2003). It may be a first step, but it is not
data mining. Data mining is the process of discovering
and interpreting meaningful, previously hidden patterns
in the data. It is not a set of descriptive statistics. Descrip-
tion is not prediction. Furthermore, the focus of data
mining is on the process, not a particular technique, used
to make reasonably accurate predictions. It is iterative in
nature and generically can be decomposed into the fol-
lowing steps: (1) data acquisition through translating,
cleansing, and transforming data from numerous sources,
(2) goal setting or hypotheses construction, (3) data
mining, and (4) validating or interpreting results.
The process of generating rules through a mining
operation becomes an ethical issue, when the results are
used in decision-making processes that affect people or
when mining customer data unwittingly compromises the
privacy of those customers (Fule & Roddick, 2004). Data
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
miners and decision makers must contemplate ethical
issues before encountering one. Otherwise, they risk not
identifying when a dilemma exists or making poor choices,
since all aspects of the problem have not been identified.
BACKGROUND
Technology has moral properties, just as it has political
properties (Brey 2000; Feenberg, 1999; Sclove, 1995;
Winner, 1980). Winner (1980) argues that technological
artifacts and systems function like laws, serving as frame-
works for public order by constraining individuals be-
haviors. Sclove (1995) argues that technologies possess
the same kinds of structural effects as other elements of
society, such as laws, dominant political and economic
institutions, and systems of cultural beliefs. Data mining,
being a technological artifact, is worthy of study from an
ethical perspective due to its increasing importance in
decision making, both in the private and public sectors.
Computer systems often function less as background
technologies and more as active constituents in shaping
society (Brey, 2000). Data mining is no exception. Higher
integration of data mining capabilities within applications
ensures that this particular technological artifact will
increasingly shape public and private policies.
Data miners and decision makers obviously are obli-
gated to adhere to the law. But ethics are oftentimes more
restrictive than what is called for by law. Ethics are
standards of conduct that are agreed upon by cultures
and organizations. Supreme Court Justice Potter Stewart
defines the difference between ethics and laws as know-
ing the difference between what you have a right to do
(legally, that is) and what is right to do. Sadly, a number
of IS professionals either lack an awareness of what their
company actually does with data and data mining results
or purposely come to the conclusion that it is not their
concern. They are enablers in the sense that they solve
managements problems. What management does with
that data or results is not their concern.
Most laws do not explicitly address data mining,
although court cases are being brought to stop certain
data mining practices. A federal court ruled that using
data mining tools to search Internet sites for competitive
information may be a crime under certain circumstances
(Scott, 2002). In EF Cultural Travel BV vs. Explorica Inc.
TEAM LinG
Ethics of Data Mining
(No. 01-2000 1st Cir. Dec. 17, 2001), the First Circuit Court
of Appeals in Massachusetts held that Explorica, a tour
operator for students, improperly obtained confidential
information about how rival EFs Web site worked and
used that information to write software that gleaned data
about student tour prices from EFs Web site in order to
undercut EFs prices (Scott, 2002). In this case, Explorica
probably violated the federal Computer Fraud and Abuse
Act (18 U.S.C. Sec. 1030). Hence, the source of the data is
important when data mining.
Typically, with applied ethics, a morally controversial
practice, such as how data mining impacts privacy, is
described and analyzed in descriptive terms, and finally
moral principles and judgments are applied to it and moral
deliberation takes place, resulting in a moral evaluation,
and operationally, a set of policy recommendations (Brey,
2000, p. 10). Applied ethics is adopted by most of the
literature on computer ethics (Brey, 2000). Data mining
may appear to be morally neutral, but appearances in this
case are deceiving. This paper takes an applied perspec-
tive to the ethical dilemmas that arise from the application
of data mining in specific circumstances as opposed to
examining the technological artifacts (i.e., the specific
software and how it generates inferences and predictions)
used by data miners.
MAIN THRUST
Computer technology has redefined the boundary be-
tween public and private information, making much more
information public. Privacy is the freedom granted to
individuals to control their exposure to others. A custom-
ary distinction is between relational and informational
privacy. Relational privacy is the control over ones
person and ones personal environment, and concerns
the freedom to be left alone without observation or inter-
ference by others. Informational privacy is ones control
over personal information in the form of text, pictures,
recordings, and so forth (Brey, 2000).
Technology cannot be separated from its uses. It is
the ethical obligation of any information systems (IS)
professional, through whatever means he or she finds out
that the data that he or she has been asked to gather or
mine is going to be used in an unethical way, to act in a
socially and ethically responsible manner. This might
mean nothing more than pointing out why such a use is
unethical. In other cases, more extreme measures may be
warranted. As data mining becomes more commonplace
and as companies push for even greater profits and market
share, ethical dilemmas will be increasingly encountered.
Ten common blunders that a data miner may cause, result-
ing in potential ethical or possibly legal dilemmas, are
(Skalak, 2001):
1. Selecting the wrong problem for data mining.
2. Ignoring what the sponsor thinks data mining is and -
what it can and cannot do.
3. Leaving insufficient time for data preparation.
4. Looking only at aggregated results, never at indi-
vidual records.
5. Being nonchalant about keeping track of the mining
procedure and results.
6. Ignoring suspicious findings in a haste to move on.
7. Running mining algorithms repeatedly without think-
ing hard enough about the next stages of the data
analysis.
8. Believing everything you are told about the data.
9. Believing everything you are told about your own
data mining analyses.
10. Measuring results differently from the way the
sponsor will measure them.
These blunders are hidden ethical dilemmas faced by
those who perform data mining. In the next subsections,
sample ethical dilemmas raised with respect to the appli-
cation of data mining results in the public sector are
examined, followed briefly by those in the private sector.
Ethics of Data Mining in the Public
Sector
Many times, the objective of data mining is to build a
customer profile based on two types of datafactual
(who the customer is) and transactional (what the cus-
tomer does) (Adomavicius & Tuzhilin, 2001). Often, con-
sumers object to transactional analysis. What follows are
two examples; the first (identifying successful students)
creates a profile based primarily on factual data, and the
second (identifying criminals and terrorists) primarily on
transactional.
Identifying Successful Students
Probably the most common and well-developed use of
data mining is the attraction and retention of customers.
At first, this sounds like an ethically neutral application.
Why not apply the concept of students as customers to
the academe? When students enter college, the transition
from high school for many students is overwhelming,
negatively impacting their academic performance. High
school is a highly structured Monday-through-Friday
schedule. College requires students to study at irregular
hours that constantly change from week to week, depend-
ing on the workload at that particular point in the course.
Course materials are covered at a faster pace; the duration
of a single class period is longer; and subjects are often
more difficult. Tackling the changes in a students aca-
455
TEAM LinG

demic environment and living arrangement as well as
developing new interpersonal relationships is daunting
for students. Identifying students prone to difficulties and
intervening early with support services could significantly
improve student success and, ultimately, improve reten-
tion and graduation rates.
Consider the following scenario that realistically could
arise at many institutions of higher education. Admissions
at the institute has been charged with seeking applicants
who are more likely to be successful (i.e., graduate from the
institute within a five-year period). Someone suggests
data mining existing student records to determine the
profile of the most likely successful student applicant.
With little more than this loose definition of success, a
great deal of disparate data is gathered and eventually
mined. The results indicate that the most likely successful
applicant, based on factual data, is an Asian female whose
familys household income is between $75,000 and $125,000
and who graduates in the top 25% of her high school class.
Based on this result, admissions chooses to target market
such high school students. Is there an ethical dilemma?
What about diversity? What percentage of limited market-
ing funds should be allocated to this customer segment?
This scenario highlights the importance of having well-
defined goals before beginning the data mining process.
The results would have been different if the goal were to
find the most diverse student population that achieved a
certain graduation rate after five years. In this case, the
process was flawed fundamentally and ethically from the
beginning.
Identifying Criminals and Terrorists
The key to the prevention, investigation, and prosecution
of criminals and terrorists is information, often based on
transactional data. Hence, government agencies increas-
ingly desire to collect, analyze, and share information
about citizens and aliens. However, according to Rep. Curt
Weldon (R-PA), chairman of the House Subcommittee on
Military Research and Development, there are 33 classified
agency systems in the federal government, but none of
them link their raw data together (Verton, 2002). As Steve
Cooper, CIO of the Office of Homeland Security, said, I
havent seen a federal agency yet whose charter includes
collaboration with other federal agencies (Verton, 2002, p.
5). Weldon lambasted the federal government for failing to
act on critical data mining and integration proposals that
had been authored before the terrorists attacks on Sep-
tember 11, 2001 (Verton, 2002).
Data to be mined is obtained from a number of sources.
Some of these are relatively new and unstructured in
nature, such as help desk tickets, customer service com-
plaints, and complex Web searches. In other circumstances,
data miners must draw from a large number of sources. For
456
Ethics of Data Mining
example, the following databases represent some of those
used by the U.S. Immigration and Naturalization Service
(INS) to capture information on aliens (Verton, 2002).
Employment Authorization Document System
Marriage Fraud Amendment System
Deportable Alien Control System
Reengineered Naturalization Application Casework
System
Refugees, Asylum, and Parole System
Integrated Card Production System
Global Enrollment System
Arrival Departure Information System
Enforcement Case Tracking System
Student and Schools System
General Counsel Electronic Management System
Student Exchange Visitor Information System
Asylum Prescreening System
Computer-Linked Application Information Man-
agement System (two versions)
Non-Immigrant Information System
There are islands of excellence within the public
sector. One such example is the U.S. Armys Land Infor-
mation Warfare Activity (LIWA), which is credited with
having one of the most effective operations for mining
publicly available information in the intelligence commu-
nity (Verton, 2002, p. 5).
Businesses have long used data mining. However,
recently, governmental agencies have shown growing
interest in using data mining in national security initia-
tives (Carlson, 2003, p. 28). Two government data min-
ing projects, the latter renamed by the euphemism fac-
tual data analysis, have been under scrutiny (Carlson,
2003) These projects are the U.S. Transportation Secu-
rity Administrations (TSA) Computer Assisted Passen-
ger Prescreening System II (CAPPS II) and the Defense
Advanced Research Projects Agencys (DARPA) Total
Information Awareness (TIA) research project (Gross,
2003). TSAs CAPPS II will analyze the name, address,
phone number, and birth date of airline passengers in an
effort to detect terrorists (Gross, 2003). James Loy, direc-
tor of the TSA, stated to Congress that, with CAPPS II,
the percentage of airplane travelers going through extra
screening is expected to drop significantly from 15% that
undergo it today (Carlson, 2003). Decreasing the number
of false positive identifications will shorten lines at
airports.
TIA, on the other hand, is a set of tools to assist
agencies such as the FBI with data mining. It is designed
to detect extremely rare patterns. The program will in-
clude terrorism scenarios based on previous attacks,
intelligence analysis, war games in which clever people
imagine ways to attack the United States and its de-
TEAM LinG
Ethics of Data Mining
ployed forces, testified Anthony Tether, director of
DARPA, to Congress (Carlson, 2003, p. 22). When asked
how DARPA will ensure that personal information caught
in TIAs net is correct, Tether stated that were not the
people who collect the data. Were the people who supply
the analytical tools to the people who collect the data
(Gross, 2003, p. 18). Critics of data mining say that while
the technology is guaranteed to invade personal privacy,
it is not certain to enhance national security. Terrorists do
not operate under discernable patterns, critics say, and
therefore the technology will likely be targeted primarily
at innocent people (Carlson, 2003, p. 22). Congress voted
to block funding of TIA. But privacy advocates are
concerned that the TIA architecture, dubbed mass
dataveillance, may be used as a model for other programs
(Carlson, 2003).
Systems such as TIA and CAPPS II raise a number of
ethical concerns, as evidenced by the overwhelming
opposition to these systems. One system, the Multistate
Anti-TeRrorism Information EXchange (MATRIX), rep-
resents how data mining has a bad reputation in the public
sector. MATRIX is self-defined as a pilot effort to
increase and enhance the exchange of sensitive terrorism
and other criminal activity information between local,
state, and federal law enforcement agencies (matrix-
at.org, accessed June 27, 2004). Interestingly, MATRIX
states explicitly on its Web site that it is not a data-mining
application, although the American Civil Liberties Union
(ACLU) openly disagrees. At the very least, the perceived
opportunity for creating ethical dilemmas and ultimately
abuse is something the public is very concerned about, so
much so that the project felt that the disclaimer was
needed. Due to the extensive writings on data mining in
the private sector, the next subsection is brief.
Ethics of Data Mining in the Private
Sector
Businesses discriminate constantly. Customers are clas-
sified, receiving different services or different cost struc-
tures. As long as discrimination is not based on protected
characteristics such as age, race, or gender, discriminat-
ing is legal. Technological advances make it possible to
track in great detail what a person does. Michael Turner,
executive director of the Information Services Executive
Council, states, For instance, detailed consumer infor-
mation lets apparel retailers market their products to
consumers with more precision. But if privacy rules im-
pose restrictions and barriers to data collection, those
limitations could increase the prices consumers pay when
they buy from catalog or online apparel retailers by 3.5%
to 11% (Thibodeau, 2001, p. 36). Obviously, if retailers
cannot target their advertising, then their only option is
to mass advertise, which drives up costs.
With this profile of personal details comes a substan-
tial ethical obligation to safeguard this data. Ignoring any -
legal ramifications, the ethical responsibility is placed
firmly on IS professionals and businesses, whether they
like it or not; otherwise, they risk lawsuits and harming
individuals. The data industry has come under harsh
review. There is a raft of federal and local laws under
consideration to control the collection, sale, and use of
data. American companies have yet to match the tougher
privacy regulations already in place in Europe, while
personal and class-action litigation against businesses
over data privacy issues is increasing (Wilder & Soat,
2001, p. 38).
FUTURE TRENDS
Data mining traditionally was performed by a trained
specialist, using a stand-alone package. This once na-
scent technique is now being integrated into an increas-
ing number of broader business applications and legacy
systems used by those with little formal training, if any,
in statistics and other related disciplines. Only recently
has privacy and data mining been addressed together, as
evidenced by the fact that the first workshop on the
subject was held in 2002 (Clifton & Estivill-Castro, 2002).
The challenge of ensuring that data mining is used in an
ethically and socially responsible manner will increase
dramatically.
CONCLUSION
Several lessons should be learned. First, decision makers
must understand key strategic issues. The data miner
must have an honest and frank dialog with the sponsor
concerning objectives. Second, decision makers must not
come to rely on data mining to make decisions for them.
The best data mining is susceptible to human interpreta-
tion. Third, decision makers must be careful not to explain
away with intuition data mining results that are
counterintuitive. Decision making inherently creates ethi-
cal dilemmas, and data mining is but a tool to assist
management in key decisions.
REFERENCES
Adomavicius, G. & Tuzhilin, A. (2001). Using data mining
methods to build customer profiles. Computer, 34(2), 74-82.
Brey, P. (2000). Disclosive computer ethics. Computers
and Society, 30(4), 10-16.
457
TEAM LinG

Carlson, C. (2003a). Feds look at data mining. eWeek,
20(19), 22.
Carlson, C. (2003b). Lawmakers will drill down into data
mining. eWeek, 20(13), 28.
Clifton, C. & Estivill-Castro, V. (Eds.). (2002). Privacy,
security and data mining. Proceedings of the IEEE Inter-
national Conference on Data Mining Workshop on Pri-
vacy, Security, and Data Mining, Maebashi City, Japan.
Edelstein, H. (2003). Description is not prediction. DM
Review, 13(3), 10.
Feenberg, A. (1999). Questioning technology. London:
Routledge.
Fule, P., & Roddick, J.F. (2004). Detecting privacy and
ethical sensitivity in data mining results. Proceedings of
the 27 th Conference on Australasian Computer Science,
Dunedin, New Zealand.
Gross, G. (2003). U.S. agencies defend data mining plans.
ComputerWorld, 37(19), 18.
Sclove, R. (1995). Democracy and technology. New York:
Guilford Press.
Scott, M.D. (2002). Can data mining be a crime? CIO
Insight, 1(10), 65.
Skalak, D. (2001). Data mining blunders exposed! 10 data
mining mistakes to avoid making today. DB2 Magazine, 6(2),
10-13.
Thibodeau, P. (2001). FTC examines privacy issues raised
by data collectors. ComputerWorld, 35(13), 36.
Verton, D. (2002a). Congressman says data mining could
have prevented 9-11. ComputerWorld, 36(35), 5.
Verton, D. (2002b). Database woes thwart counterterrorism
work. ComputerWorld, 36(49), 14.
Wilder, C., & Soat, J. (2001). The ethics of data. Informa-
tion Week, 1(837), 37-48.
458
Ethics of Data Mining
Winner, L. (1980). Do artifacts have politics? Daedalus,
109, 121-136.
KEY TERMS
Applied Ethics: The study of a morally controversial
practice, whereby the practice is described and analyzed,
and moral principles and judgments are applied, resulting
in a set of recommendations.
Ethics: The study of the general nature of morals and
values as well as specific moral choices; it also may refer
to the rules or standards of conduct that are agreed upon
by cultures and organizations that govern personal or
professional conduct.
Factual Data: Data that include demographic informa-
tion such as name, gender, and birth date. It also may
contain information derived from transactional data such
as someones favorite beverage.
Factual Data Analysis: Another term for data mining,
often used by government agencies. It uses both factual
and transactional data.
Informational Privacy: The control over ones per-
sonal information in the form of text, pictures, recordings,
and such.
Mass Dataveillance: Suspicion-less surveillance of
large groups of people.
Relational Privacy: The control over ones person
and ones personal environment.
Transactional Data: Data that contains records of
purchases over a given period of time, including such
information as date, product purchased, and any special
requests.
TEAM LinG

Ethnography to Define Requirements and Data
Model
Gary J. DeLorenzo
Robert Morris University, USA
INTRODUCTION
Ethnographic research offers an orientation to under-
stand the process and structure of a social setting and
employs research techniques consistent with this orien-
tation. The ethnographic study is rooted in gaining an
understanding of cultural knowledge used to interpret the
experience and behavior patterns of the researched par-
ticipants. The ethnographic method aids the researcher in
identifying descriptions and in interpreting language,
terms, and meanings as the basis of defining user require-
ments for information systems.
In the business sector, enterprises that use third-
party application providers to support day-to-day opera-
tions have limited independence to access, review, and
analyze the data needed for strategic and tactical decision
making. Because of security and firewall protection con-
trols maintained by the provider, enterprises do not have
direct access to the data needed to build analytical report-
ing solutions for decision support.
This article reports a unique methodology used to
define the conceptual model of one enterprises (PPG
Industries, Inc.) business needs to reduce working capital
when using application service providers (ASPs).
BACKGROUND
The Methodology
Ethnographic research is the study of both explicit and
tacit knowledge within a culture. The technique is based
to gain an understanding of the acquired knowledge
people use to interpret experience and generate behavior.
Whereas explicit cultural knowledge can be communi-
cated at a conscious level and with relative ease, tacit
cultural knowledge remains largely outside of peoples
awareness (Creswell, 1994).
The ethnographic technique aids the researcher in
identifying cultural descriptions and uncovering mean-
ing to language within an organization. From my 12 years
of experience at PPG Industries, Inc. (PPG), a U.S.-based
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
459
-
$8 billion per year manufacturing company specializing in
coatings, chemicals, glass and fiberglass, the normal
workflow process to define user requirements for decision
support is through a quick, low cost, rapid application
development approach.
To gain an in-depth understanding of the culture and
knowledge base of the Credit Services Department, the
ethnographic research technique was selected to con-
struct an understanding to questions such as why do
they do what they do? and more importantly for me,
what reporting solutions are needed to understand cus-
tomer payments so that actions can be prioritized accord-
ingly? The ability to answer the second question was
critical in order to define the conceptual model and infor-
mation system specifications based on Credits user re-
quirements. While using a third-party vendor solution
thorough an ASP model, another challenge was in gaining
access to data that was not readily accessible (Boyd, 2001).
Why the Ethnographic Technique?
Ethnographic research is based on understanding the
language, terms, and meaning within a culture (Agar,
1980). Ethnography is an appealing approach, because it
promotes an understanding of process, terms and mean-
ing, and beliefs of the people in a particular culture.
Interviews, discussions, participant observation, and
social interaction would help me attain an understanding
of a culture. As opposed to the quantitative approach,
which reflects on researching a situation, problem, or
hypothesis at a given point in time, the qualitative ap-
proach using ethnographic techniques is more of an
evolutionary process over time, which covers months, if
not years, for the researcher. The time consideration is
important because of the ever changing and evolving
environment of the enterprise. New business plans, direc-
tion, and project activities frequently change within a
business. Emerging technology processes such as RAD
(rapid application development) to streamline the pro-
cess of developing and implementing information sys-
tems continue to change the way information systems
are created.
TEAM LinG

MAIN THRUST
Cultural Setting and Contextual
Baseline
To gain an understanding of what the Credit Services
Department actually does, an overview of the specific
department goals, objectives, and responsibilities are
noted in this section and called the contextual baseline,
and uses the native, emic terms of the department. A
contextual baseline identifies the driving motivators of
Credit Services to understand what people do and how
they accomplish their jobs. The General Credit Manager
noted how the Credit Services Department is viewed as
the financial conscience of PPG in the relationship be-
tween sales and receiving payments against those sales
by our customers (Camsuzou, 2001) The Credit Services
Department monitors credit ratings to ensure that custom-
ers have the financial stability to pay invoices while also
tracking and researching with outside banking services
on the application of payments against invoices.
Field Work
The interview process included different informants
throughout PPG with informants from strategic associ-
ates (business unit directors), tactical members (credit
managers, business unit managers), and operational mem-
bers (users with the business units). Fourteen individuals
from throughout the enterprise were interviewed as key
informants. Also, general observations were captured in
the field notebook by sitting and working on a daily basis
in the Credit Department area (over 50 employees) from
June, 2001 through December, 2001.
Geertz (1973) suggests that qualitative research often
is presented as working up from data [whereas] in
ethnography, the techniques and procedures [do not]
define the enterprise. [what defines it] is the intellectual
effort [borrowed] from Ryle called thick descriptions (p
6). Thick descriptions are a reference to getting an under-
standing and insight into a culture through the uncover-
ing of facts, at a very detailed level. Its the ability to gain
a truer, real understanding of a culture through an inter-
pretation of language, symbols, and signs. Geertz (1973)
summarized that the challenge of the ethnographer does
not rest on the ability to capture facts and carry them home
like a mask or carving, but on a degree to which he is able
to clarify what goes on in such places, to reduce the
puzzlement [where] unfamiliar acts naturally give rise [to
meaning] (p. 17). Ethnography is a technique based on
a theory where the understanding of a culture can be
gained through the analysis of the cumulative, unstruc-
tured, and disconnected findings captured during the
460
Ethnography to Define Requirements and Data Model
field research. These findings are assembled into a
coherent sequence of bold, structured, and related sto-
ries.
Initial Themes
According to Spradley (1979), the next step is the identi-
fication of initial themes. The themes are thin descriptions
of high-level, overview insights toward a culture. Geertz
(1973) states that systematic modes of assessment [are
needed in] interpretative approaches. For a field of study
to assert itself as a science, the conceptual structure of
a cultural interpretation should be formulable. (p. 24) He
continues to state that theoretical formulations tend to
start as general interpretations, but the scientific, theo-
retical process pushing forward should [include} the
essential task of theory building, not to codify abstract
regularities, but to make thick descriptions possible (p.
26). This theory-building approach, from an overview
perspective, helped me to define the initial themes of the
Credit Services Department.
After analyzing the transcribed interviews, field notes,
and results from categorizing the field-work terms, three
general themes began to surface. The themes, also called
categories by Spradley (1979), are the following: (1) At the
enterprise level of PPG, there is a strategic mission to
reduce working capital through improved cash flow. (2) At
the functional level for both the Credit Services Depart-
ment and the business units, there is a tactical vision to
decrease daily sales outstanding. (3). In pursuing the first
two themes, information systems are used to track cus-
tomer payment patterns and to monitor incoming cash flow.
Domain Analysis
For Spradley (1979), a domain structure includes the
terms that belong to a category . and the semantic
relationship of terms and categories [that] are linked
together(p. 100). The process of domain identification
included contexualizing the terms and categories, draw-
ing overview diagrams of terms and categories, and iden-
tifying how the terms and categories related to each other.
Spradley (1979) summarized domain analysis as the pro-
cess to gain a better understanding, from the gathered
field data, the semantic relationships of terms and catego-
ries (p. 107).
Further review of the language to identify the thick
descriptions and rich points for PPG to reduce working
capital and decrease sales outstanding resulted in defin-
ing 43 critical and essential terms. These terms were used
as a baseline to define the domain analysis. The semantic
relationships are links between two categories that have
boundaries either inside or outside of the domain. The domain
analysis identifies the relationships among categories.
TEAM LinG
Ethnography to Define Requirements and Data Model

The domain analysis concept also is used in informa- term, relationship, and hierarchical structure of each term
tion systems to show how data relate to each other. Peter within the area. Structured questions allow the researcher -
Chen (1976) proposed the Entity-Relationship (ER) model to find out how informants have organized their knowl-
as a way to define and unify relational database views and edge. A review of the transcribed words from the inter-
can be found in popular relational database management views, additional notes, and the categorized terms from
systems such as Oracle and Microsoft SQL Server. In the the field work began to identify key, structured ques-
entity-relationship model, tables are shown as entities, tions that were important to the Credit Services Depart-
data fields are shown as attributes, and the line shows how ment. These in-depth, structured, thick-description ques-
the entities and attributes relate to each other. Entities are tions began to emerge out of the terms and categories
categories such as a person, place, or thing. The domain used in the language.
analysis identified in Figure 1 is based on the initial themes
of reducing working capital and decreasing sales out- Componential Analysis
standing, where rectangles identify primary domain enti-
ties for both the internal PPG associates and external PPG Spradley (1979) states that the development research
customers. sequence evolves through locating informants, con-
ducting interviews, and collecting the terms and symbols
Taxonomic Analysis used for language and communication. This research
included the domain analysis process with an overview
Further analysis of the fields in the domain analysis in- of terms that make up categories and the relationship
cluded a review of the specific terms were considered among categories. The domain analysis provided an
critical because of the frequency and consistency that overview of the language in the cultural scene. The
these terms were used in the language. These terms even- technique of taxonomic analysis identified relationships
tually became the baseline data field attributes required for and differences among terms within each domain. The
the decision support system model. A taxonomic analysis next technique in the process included searching and
is a technique used to capture the essential terms, also organizing the attributes associated with cultural terms
called fields, in a hierarchical fashion to define not only the and categories. Attributes are specific terms that have
relationship among the fields but also to understand what significant meaning in the language. This process is
terms are a subset meaning to a grander, higher term. The called componential analysis.
design of the taxonomic analysis (see Figure 2) is based on Spradley (1973) states that, in drafting a schematic
a spiraling concept where all of the terms and relationships diagram on domains, This thinking process is one of the
defined previously are dependent on the central theme; best strategies for discovering cultural themes (p. 199).
that is, to get dollars for sales through reducing working The schematic diagram in Figure 3 identifies the one
capital, which is noted in the center of the drawing. The central theme that motivates Credit Services and this
taxonomic analysis is broken into five component areas of research project, and is based on one principleunder-
who, what, why, when and how. Each area identifies the standing customer payment behavior.

Figure 1. Domain analysis (DeLorenzo, 2001)

B ra n c h
C u s to m e r
F ie ld S a le s O p e r a t io n s /
S e r v ic e
D is t r ib u t io n

S t r a te g ic B u s in e s s
U n its

C r e d it
P a re n t H d q trs ,
L o c a t io n s

E x te rn a l
C r e d it S e r v ic e s In te rn a l P P G
C u s to m e r
D e p a rtm e n t A u d ie n c e
A u d ie n c e

C a s h A p p lic a t io n
A c c o u n t S ta tu s

P u rc h a s e a n d S e ll
P r o d u c ts / S e r v ic e s

P re s e n t S ta te m e n t
a n d I n v o ic e f o r
Paym ent

D is p u te s

R e c e iv e $ , B a n k $ T ra n s fe r to
A p p ly P a y m e n t PP G Account

461

TEAM LinG
 Ethnography to Define Requirements and Data Model

Figure 2. Taxonomic analysis (DeLorenzo, 2001)

Taxonomic Analysis

Customer
WHAT
Payment
Cash
WHO Application
Location - Dispute
Level 3 Invoice
Credit
Limit Status:
Parent - Product Line
On Hold,
Branch Operations/ Level 1 (Divisional Sales
Active,
Distribution Resp)
Release
Customer
Service Invoices
Customer
Cash 15 SBU's
Field
Sales Application Account
PPG Auto Statements
PPG Paid
Credit Glass Get Dollars
Current In Full
For Sales,
Balance
Reduce Dispute
Oracle Working Cap Past Due Dispute Resolution:
A/R: AG, CARS Online Aging Reason:
Transaction Balance Unearned
Knowledge Over
SBU Processing Discount,
Mgt Credit Limit
Order Entry Price
Term Daily Deduction
Timeline Sales
Dimension
DSS/Business Outstanding
Intelligence Deduct/ Chargeback,
eBilling Short Writeoff,
HOW Invoice Term - Statement Term - Pay Credit Memo WHY
Online (Discounted Price) (Discount on Payment) Reason
Analytical
Processing
Measures
Invoice Date Discount Due Date

Invoice Due Date Payment Date

WHEN

Figure 3. Cultural themes (DeLorenzo, 2002)

S e ll p r o d u c t t o f in a n c ia lly
s ta b le c u s to m e rs ,
A c c u r a te p ric e o n in v o ic e
C u ltu ra l T h e m e s
R e d u c e s w o r k in g c a p it a l a t t h e
e n t e r p r is e , P P G le v e l

R e v ie w a n d m o n it o r f in a n ic a l
s t a b ilit y o f c u s to m e rs

Im p ro v e d c a s h flo w a t th e s tra te g ic
b u s in e s s u n it , d e p a r t m e n t le v e l

R e c e iv e c a s h , e n s u re p a y m e n ts U n d e r s t a n d in g C u s t o m e r
re c e iv e d f o r s e r v ic e s P a y m e n t B e h a v io r

I m p r o v e in v o ic e , s ta te m e n t in te g rity
r e s u lt in g in r e d u c e d d is p u t e s /
d e d u c ts -- - - h ig h e r
c u s to m e r s a t is f a c t io n

D is p u t e s / d e d u c t s a n d r e s o lu t io n s
o n d is c r e p a n c ie s

K n o w le d g e b a s e d d e c is io n
s u p p o rt a n d b u s in e s s
in t e llig e n c e

H y b r id , in t e g r a t e d in f o r m a t io n
s y s te m s

FUTURE TRENDS years that businesses have taken to manage data. It

addresses the information system needs at PPG to better
Requirements Definition understand customer payment behavior (Craig & Jutla,
2001; Forcht & Cochran, 1999; Inmon, 1996).
The final effort in the ethnography included the drafting The accumulation of the domain analysis and impor-
of the decision support system model to assist PPG in tant entities, the taxonomic analysis with critical catego-
reducing working capital. The model is based on evolu- ries and attributes, and the componential analytical ef-
tionary changes in computer architecture over the past 40 forts identifying cultural themes brought me to my last

462

TEAM LinG
Ethnography to Define Requirements and Data Model
activity. The final stage in the ethnographic process was
to define the decision support system model to assist PPG
in tracking customer payments. The model was based on
using an information systems-based approach to capture
the data required to understand customer payment be-
havior and to provide trend analysis capabilities to gain
knowledge and insight from that understanding.
CONCLUSION
An analysis of a cultures language and communication
mechanisms are important areas to understand in building
information systems models. The research included the
gathering of data requirements for the Credit Services
Department at PPG by living and working within the
researched community to gain a better understanding of
its problem-solving and reporting needs to monitor cus-
tomer payment behavior. The data requirements became
the baseline prerequisite in defining the conceptual deci-
sion support model and database schema for the Credit
Services Departments decision support project. In 2003,
the conceptual model and schema were used by PPGs
Information Technology Department to develop an appli-
cation system to assist the Credit Services Department in
tracking and managing customer payment behavior.
REFERENCES
Agar, M. (1980). The professional stranger: An informal intro-
duction to ethnography. San Diego, CA: Academic Press, Inc.
Boyd, J. (2001, April 16). Think ASPs make sense. Internet
Week, 48.
Camsuzou, C. (2001). The ecommerce vision of the credit
services department. PPG Industries IT Strategy, 15-17.
Chen P. (1976). The entity-relationship modelToward a
unified view of data. ACM Transactions on Database
Systems, 1(1), 9-36.
Craig, J., & Jutla, D. (2001). eBusiness readiness: A cus-
tomer-focused framework. Upper Saddle River, NJ:
Addison Wesley Publishers.
Creswell, J. (1994). Research design qualitative and quanti-
tative approaches. Thousand Oaks, CA: Sage Publications.
Forcht, K.A., & Cochran, K. (1999). Using data mining and
data warehousing techniques. Industrial Management
and Data Systems, 189-196.
Geertz, C. (1973). Emphasizing interpretation from the
interpretation of cultures. New York: Basic Books, Inc.
Inmon, W. (1996). Building the data warehouse. New
York: Wiley Computer Publishing. -
Spradley, J.P. (1979). The ethnographic interview. Or-
lando, FL: Harcourt Brace.
KEY TERMS
Application Solution Providers: Third-party vendors
who provide data center, telecommunications, and appli-
cation options for major companies.
Business Intelligence: Synonym for online analytical
processing system.
Data Mining: Decision support technique that can
predict future trends and behaviors based on past historical data.
Data Warehouse: The capturing of attributes from
multiple databases in a centralized repository to assist in
the decision-making and problem-solving needs through-
out the organization.
Decision Support: Evolutionary step in the 1990s with
characteristics to review retrospective, dynamic data.
OLAP is an example of an enabling technology in this area.
Dimensions: Static, fact-oriented information used in
decision support to drill down into measurements.
Domain Analysis Technique: Search for the larger
units of cultural knowledge called domains, a synonym for
person, place, or thing. Used to gain an understanding of
the semantic relationships of terms and categories.
Drill Down: User interface technique to navigate into
lower levels of information in decision support systems.
Ethnography: The work of describing a culture. The
essential core aims to understand another way of life from
the native point of view.
Measurements: Dynamic, numeric values associated
with dimensions found through drilling down into lower
levels of detail within decision support systems.
Online Analytical Processing Systems (OLAP):
Technology-based solution with data delivered at mul-
tiple dimensions to allow drilling down at multiple levels.
Requirements: Specifics based on defined criterion
used as the basis for information systems design.
Systems Development Life Cycle: A controlled, phased
approach in building information systems from under-
standing wants, defining specifications, designing and
coding the system through implementing the final solution.
463
TEAM LinG
464
Evaluation of Data Mining Methods
Paolo Giudici
University of Pavia, Italy
INTRODUCTION
Several classes of computational and statistical meth-
ods for data mining are available. Each class can be
parameterised so that models within the class differ in
terms of such parameters (see, for instance, Giudici,
2003; Hastie et al., 2001; Han & Kamber, 2000; Hand et
al., 2001; Witten & Frank, 1999): for example, the class
of linear regression models, which differ in the number
of explanatory variables; the class of Bayesian net-
works, which differ in the number of conditional depen-
dencies (links in the graph); the class of tree models,
which differ in the number of leaves; and the class multi-
layer perceptrons, which differ in terms of the number
of hidden strata and nodes. Once a class of models has
been established the problem is to choose the best
model from it.
BACKGROUND
A rigorous method to compare models is statistical
hypothesis testing. With this in mind one can adopt a
sequential procedure that allows a model to be chosen
through a sequence of pairwise test comparisons. How-
ever, we point out that these procedures are generally
not applicable in particular to computational data min-
ing models, which do not necessarily have an underlying
probabilistic model and, therefore, do not allow the
application of statistical hypotheses testing theory. Fur-
thermore, it often happens that for a data problem it is
possible to use more than one type of model class, with
different underlying probabilistic assumptions. For ex-
ample, for a problem of predictive classification it is
possible to use both logistic regression and tree models
as well as neural networks.
We also point out that model specification and,
therefore, model choice is determined by the type of
variables used. These variables can be the result of
transformations or of the elimination of observations,
following an exploratory analysis. We then need to
compare models based on different sets of variables
present at the start. For example, how do we compare a
linear model with the original explanatory variables
with one with a set of transformed explanatory vari-
ables?
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
The previous considerations suggest the need for a
systematic study of the methods for comparison and
evaluation of data mining models.
MAIN THRUST
Comparison criteria for data mining models can be
classified schematically into: criteria based on statisti-
cal tests, based on scoring functions, computational
criteria, Bayesian criteria and business criteria.
Criteria Based on Statistical Tests
The first are based on the theory of statistical hypoth-
esis testing and, therefore, there is a lot of detailed
literature related to this topic. See, for example, a text
about statistical inference, such as Mood, Graybill, &
Boes (1991) and Bickel & Doksum (1977). A statistical
model can be specified by a discrete probability func-
tion or by a probability density function, f(x). Such
model is usually left unspecified, up to unknown quan-
tities that have to be estimated on the basis of the data at
hand. Typically, the observed sample it is not sufficient
to reconstruct each detail of f(x), but can indeed be used
to approximate f(x) with a certain accuracy. Often a
density function is parametric so that it is defined by a
vector of parameters =( 1 ,,I ), such that each value
of corresponds to a particular density function,
p (x). In order to measure the accuracy of a parametric
model, one can resort to the notion of distance between
a model f, which underlies the data, and an approximat-
ing model g (see, for instance, Zucchini, 2000). Notable
examples of distance functions are, for categorical
variables: the entropic distance, which describes the
proportional reduction of the heterogeneity of the de-
pendent variable; the chi-squared distance, based on the
distance from the case of independence; and the 0-1
distance, which leads to misclassification rates. For
quantitative variables, the typical choice is the Euclid-
ean distance, representing the distance between two
vectors in a Cartesian space. Another possible choice is
the uniform distance, applied when nonparametric mod-
els are being used.
Any of the previous distances can be employed to
define the notion of discrepancy of an statistical model.
TEAM LinG
Evaluation of Data Mining Methods
The discrepancy of a model, g, can be obtained comparing
the unknown probabilistic model, f, and the best paramet-
ric statistical model. Since f is unknown, closeness can be
measured with respect to a sample estimate of the un-
known density f. A common choice of discrepancy func-
tion is the Kullback-Leibler divergence, which can be
applied to any type of observations. In such context, the
best model can be interpreted as that with a minimal loss
of information from the true unknown distribution.
It can be shown that the statistical tests used for model
comparison are generally based on estimators of the total
Kullback-Leibler discrepancy; the most used is the log-
likelihood score. Statistical hypothesis testing is based
on subsequent pairwise comparisons of log-likelihood
scores of alternative models. Hypothesis testing allows
one to derive a threshold below which the difference
between two models is not significant and, therefore, the
simpler models can be chosen.
Therefore, with statistical tests it is possible make
an accurate choice among the models. The defect of this
procedure is that it allows only a partial ordering of
models, requiring a comparison between model pairs
and, therefore, with a large number of alternatives it is
necessary to make heuristic choices regarding the com-
parison strategy (such as choosing among the forward,
backward and stepwise criteria, whose results may di-
verge). Furthermore, a probabilistic model must be
assumed to hold, and this may not always be possible.
Criteria Based on scoring functions
A less structured approach has been developed in the
field of information theory, giving rise to criteria based
on score functions. These criteria give each model a
score, which puts them into some kind of complete
order. We have seen how the Kullback-Leibler discrep-
ancy can be used to derive statistical tests to compare
models. In many cases, however, a formal test cannot be
derived. For this reason, it is important to develop
scoring functions that attach a score to each model. The
Kullback-Leibler discrepancy estimator is an example
of such a scoring function that, for complex models, can
be often be approximated asymptotically. A problem
with the Kullback-Leibler score is that it depends on the
complexity of a model as described, for instance, by the
number of parameters. It is thus necessary to employ
score functions that penalise model complexity.
The most important of such functions is the AIC
(Akaike Information Criterion) (Akaike, 1974). From
its definition, notice that the AIC score essentially
penalises the loglikelihood score with a term that in-
creases linearly with model complexity. The AIC criterion
is based on the implicit assumption that q remains con-
stant when the size of the sample increases. However this
assumption is not always valid and therefore the AIC
criterion does not lead to a consistent estimate of the -
dimension of the unknown model. An alternative, and
consistent, scoring function is the BIC criterion (Baye-
sian Information Criterion), also called SBC, formulated
by Schwarz (1978). As can be seen from its definition, the
BIC differs from the AIC only in the second part, which
now also depends on the sample size n. Compared to the
AIC, when n increases the BIC favours simpler models. As
n gets large, the first term (linear in n) will dominate the
second term (logarithmic in n). This corresponds to the
fact that, for a large n, the variance term in the mean
squared error expression tends to be negligible. We also
point out that, despite the superficial similarity between
the AIC and the BIC, the first is usually justified by
resorting to classical asymptotic arguments, while the
second by appealing to the Bayesian framework.
To conclude, the scoring function criteria for select-
ing models are easy to calculate and lead to a total
ordering of the models. From most statistical packages
we can get the AIC and BIC scores for all the models
considered. A further advantage of these criteria is that
they can be used also to compare non-nested models
and, more generally, models that do not belong to the
same class (for instance a probabilistic neural network
and a linear regression model).
However, the limit of these criteria is the lack of a
threshold, as well the difficult interpretability of their
measurement scale. In other words, it is not easy to
determine if the difference between two models is
significant or not, and how it compares to another
difference. These criteria are indeed useful in a prelimi-
nary exploratory phase. To examine this criteria and to
compare it with the previous ones see, for instance,
Zucchini (2000), or Hand, Mannila, & Smyth (2001).
Bayesian Criteria
A possible compromise between the previous two cri-
teria is the Bayesian criteria, which could be developed in
a rather coherent way (see, e.g., Bernardo & Smith,
1994). It appears to combine the advantages of the two
previous approaches: a coherent decision threshold and a
complete ordering. One of the problems that may arise is
connected to the absence of a general purpose software.
For data mining works using Bayesian criteria the reader
could see, for instance, Giudici (2001), Giudici & Castelo
(2003) and Brooks et al. (2003).
Computational Criteria
The intensive wide spread use of computational meth-
ods has led to the development of computationally inten-
sive model comparison criteria. These criteria are usually
465
TEAM LinG

based on using dataset different than the one being
analysed (external validation) and are applicable to all
the models considered, even when they belong to differ-
ent classes (for example in the comparison between
logistic regression, decision trees and neural networks,
even when the latter two are non probabilistic). A pos-
sible problem with these criteria is that they take a long
time to be designed and implemented, although general
purpose software has made this task easier.
The most common of such criterion is based on
cross-validation. The idea of the cross-validation method
is to divide the sample into two sub-samples, a training
sample, with n - m observations, and a validation sample,
with m observations. The first sample is used to fit a
model and the second is used to estimate the expected
discrepancy or to assess a distance. Using this criterion
the choice between two or more models is made by
evaluating an appropriate discrepancy function on the
validation sample. Notice that the cross-validation idea
can be applied to the calculation of any distance function.
One problem regarding the cross-validation criterion
is in deciding how to select m, that is, the number of the
observations contained in the validation sample. For
example, if we select m = n/2 then only n/2 observations
would be available to fit a model. We could reduce m but
this would mean having few observations for the valida-
tion sampling group and therefore reducing the accuracy
with which the choice between models is made. In prac-
tice proportions of 75% and 25% are usually used,
respectively, for the training and the validation samples.
To summarise, these criteria have the advantage of
being generally applicable but have the disadvantage of
taking a long time to be calculated and of being sensitive
to the characteristics of the data being examined. A way
to overcome this problem is to consider model combina-
tion methods, such as bagging and boosting. For a thor-
ough description of these recent methodologies, see
Hastie, Tibshirani, & Friedman (2001).
Business Criteria
One last group of criteria seem specifically tailored for
the data mining field. These are criteria that compare the
performance of the models in terms of their relative
losses, connected to the errors of approximation made
by fitting data mining models. Criteria based on loss
functions have appeared recently, although related ideas
are long time known in Bayesian decision theory (see,
for instance, Bernardo & Smith, 1984). They have a great
application potential, although at present they are mainly
concerned with classification problems. For a more de-
tailed examination of these criteria the reader can see, for
example, Hand (1997), Hand, Mannila, & Smyth (2001), or
the reference manuals on data mining software, such as
466
Evaluation of Data Mining Methods
that of SAS Enterprise Miner (SAS Institute, 2004).
The idea behind these methods is to focus the atten-
tion, in the choice among alternative models, to the
utility of the obtained results. The best model is the one
that leads to the least loss.
Most of the loss function based criteria are based
on the confusion matrix. The confusion matrix is used
as an indication of the properties of a classification
rule. On its main diagonal it contains the number of
observations that have been correctly classified for
each class. The off-diagonal elements indicate the num-
ber of observations that have been incorrectly classi-
fied. If it is assumed that each incorrect classification
has the same cost, the proportion of incorrect classifi-
cations over the total number of classifications is
called rate of error, or misclassification error, and it is
the quantity that must be minimised. The assumption of
equal costs can be replaced by weighting errors with
their relative costs.
The confusion matrix gives rise to a number of
graphs that can be used to assess the relative utility of
a model, such as the Lift Chart, and the ROC Curve (see
Giudici, 2003). The lift chart puts the validation set
observations, in increasing or decreasing order, on the
basis of their score, which is the probability of the
response event (success), as estimated on the basis of
the training set. Subsequently, it subdivides such scores
in deciles. It then calculates and graphs the observed
probability of success for each of the decile classes in
the validation set. A model is valid if the observed
success probabilities follow the same order (increas-
ing or decreasing) as the estimated ones. Notice that, in
order to be better interpreted, the lift chart of a model
is usually compared with a baseline curve, for which the
probability estimates are drawn in the absence of a
model, that is, taking the mean of the observed success
probabilities.
The ROC (Receiver Operating Characteristic) curve
is a graph that also measures predictive accuracy of a
model. It is based on four conditional frequencies that
can be derived from a model, and the choice of a cut-off
points for its scores: a) the observations predicted as
events and effectively such (sensitivity); b) the obser-
vations predicted as events and effectively non-events;
c) the observations predicted as non-events and effec-
tively events; d) the observations predicted as non-
events and effectively such (specificity). The ROC
curve is obtained representing, for any fixed cut-off
value, a point in the plane having as x-value the false
positive value (1-specificity) and as y-value the sensitiv-
ity value. Each point in the curve corresponds therefore
to a particular cut-off. In terms of model comparison, the
best curve is the one that is leftmost, the ideal one
coinciding with the y-axis.
TEAM LinG
Evaluation of Data Mining Methods
To summarise, criteria based on loss functions have
the advantage of being easy to understand but, on the
other hand, they still need formal improvements and
mathematical refinements.
FUTURE TRENDS
It is well known that data mining methods can be classi-
fied into exploratory, descriptive (or unsupervised),
predictive (or supervised) and local (see, e.g., Hand et
al., 2001). Exploratory methods are preliminary to oth-
ers and, therefore, do not need a performance measure.
Predictive problems, on the other hand, are the setting
where model comparison methods are most needed,
mainly because of the abundance of the models avail-
able. All presented criteria can be applied to predictive
models: this is a rather important aid for model choice.
For descriptive models aimed at summarising vari-
ables, such as clustering methods, the evaluation of the
results typically proceeds on the basis of the Euclidean
distance, leading at the R2 index. We remark that it is
important to examine the ratio between the between
and total sums of squares that leads to R 2 for each
variable in the dataset. This can give a variable-specific
measure of the goodness of the cluster representation.
Finally, it is difficult to assess local models, such as
association rules, for the bare fact that a global measure
of evaluation of such model contradicts with the very
notion of a local model. The idea that prevails in the
literature is to measure the utility of patterns in terms of
how interesting or unexpected they are to the analyst. As
measures of interest one can consider, for instance, the
support, the confidence and the lift. The former can be
used to assess the importance of a rule, in terms of its
frequency in the database; the second can be used to
investigate possible dependences between variables;
finally the lift can be employed to measure the distance
from the situation of independence.
CONCLUSION
The evaluation of data mining methods requires a great
deal of attention. A valid model evaluation and compari-
son can improve considerably the efficiency of a data
mining process. We have presented several ways to
perform model comparison; each has its advantages and
disadvantages. Choosing which of them is most suited
for a particular application depends on the specific
problem and on the resources (e.g., computing tools and
time) available. Furthermore, the choice must also be
based upon the kind of usage of the final results. This
implies that, for example, in a business setting, com-
parison criteria based on business quantities are ex-
tremely useful. -
REFERENCES
Akaike, H. (1974). A new look at statistical model
identification. IEEE Transactions on Automatic Con-
trol, 19, 716-723.
Bernardo, J.M., & Smith, A.F.M. (1994). Bayesian
theory. New York: Wiley.
Bickel, P.J., & Doksum, K.A. (1977). Mathematical
statistics. New York: Prentice Hall.
Brooks, S.P., Giudici, P., & Roberts, G.O. (2003).
Efficient construction of reversible jump MCMC pro-
posal distributions. Journal of The Royal Statistical
Society Series B, 1, 1-37.
Castelo, R., & Giudici, P. (2003). Improving Markov
chain model search for data mining. Machine Learning,
50, 127-158.
Giudici, P. (2001). Bayesian data mining, with applica-
tion to credit scoring and benchmarking. Applied Sto-
chastic Models in Business and Industry, 17, 69-81.
Giudici, P. (2003). Applied data mining. London, Wiley.
Han, J., & Kamber, M (2001). Data mining: Concepts and
techniques. New York: Morgan Kaufmann.
Hand, D. (1997). Construction and assessment of classi-
fication rules. London: Wiley.
Hand, D.J., Mannila, H., & Smyth, P. (2001). Principles of
data mining. New York: MIT Press.
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The
elements of statistical learning: Data mining, inference
and prediction. New York: Springer-Verlag.
Mood, A.M., Graybill, F.A., & Boes, D.C. (1991). Intro-
duction to the theory of statistics. Tokyo: McGraw
Hill.
SAS Institute Inc. (2004). SAS enterprise miner refer-
ence manual. Cary: SAS Institute.
Schwarz, G. (1978). Estimating the dimension of a
model. Annals of Statistics, 62, 461-464.
Witten, I., & Frank, E. (1999). Data Mining: Practical
machine learning tools and techniques with Java
implementation. New York: Morgan Kaufmann.
Zucchini, W. (2000). An introduction to model selec-
tion. Journal of Mathematical Psychology, 44, 41-61.
467
TEAM LinG
 Evaluation of Data Mining Methods

KEY TERMS Entropic Distance: The entropic distance of a dis-

tribution g from a target distribution f, is:
0-1 Distance: The 0-1 distance between a vector of
fi
predicted values, X g , and a vector of observed values, E d = i f i log
gi
X f , is:
Euclidean Distance: The distance between a vector of
n
predicted values, X g , and a vector of observed values,
0 1 d = 1(X fr X gr )
r =1
X f , is expressed by the equation:
where 1(w,z) =1 if w=z and 0 otherwise.
n
d (X f , X g ) = (X X gr )
2
AIC Criterion: The AIC criterion is defined by the 2
r =1
fr

following equation:

AIC = 2 log L(; x1 ,..., x n ) + 2q

where log L(; x1 ,..., x n ) is the logarithm of the likeli-
hood function calculated in the maximum likelihood
parameter estimate and q is the number of parameters of
the model.
BIC Criterion: The BIC criterion is defined by the
following expression:
Kullback-Leibler Divergence: The Kullback-
Leibler divergence of a parametric model p with re-
spect to an unknown density f is defined by:
f (x i )
K L ( f , p ) = i f (x i ) log
p (x i )
where the suffix indicates the values of the param-

( )
BIC = 2 log L ; x1 ,..., x n + q log(n )
eters, which minimizes the distance with respect to f.

Chi-Squared Distance: The chi-squared distance Log-Likelihood Score: The log-likelihood score
of a distribution g from a target distribution f is: is defined by

( f i g i )2 2 log p (x i ) [ ]
2
d = i i =1
gi

Discrepancy Of A Model: Assume that f represents

the unknown density of the population, and let g= p be Uniform Distance: The uniform distance between
a family of density functions (indexed by a vector of I two distribution functions, F and G, with values in [0, 1]
parameters, q) that approximates it. Using, to exem- is defined by:
plify, the Euclidean distance, the discrepancy of a model
g, with respect to a target model f is: sup F (t ) G (t )
0 t 1

n
( f , p ) = ( f (x i ) p (x i ))
2

i =1

468

TEAM LinG

Evolution of Data Cube Computational
Approaches
Rebecca Boon-Noi Tan
Monash University, Australia
INTRODUCTION
Aggregation is a commonly used operation in decision
support database systems. Users of decision support
queries are interested in identifying trends rather than
looking at individual records in isolation. Decision sup-
port system (DSS) queries consequently make heavy use
of aggregations, and the ability to simultaneously aggre-
gate across many sets of dimensions (in SQL terms, this
translates to many simultaneous group-bys) is crucial for
Online Analytical Processing (OLAP) or multidimensional
data analysis applications (Datta, VanderMeer, &
Ramamritham, 2002; Dehne, Eavis, Hambrusch, & Rau-
Chaplin, 2002; Elmasri & Navathe, 2004; Silberschatz,
Korth & Sudarshan, 2002).
BACKGROUND
Although aggregation functions together with another
operator, Group-by in SQL terms, have been widely used
Figure 1. An example showing how a histogram is formed from results obtained from a data cube
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
469
-
in business application for the past few decades. Common
forms of data analysis include histograms, roll-up totals
and sub-totals for drill-downs and cross tabulation. The
three common forms are difficult to use with these SQL
aggregation constructs (Gray, Bosworth, Lyaman, &
Pirahesh, 1996). An explanation of the three common
problems: (a) Histograms, (b) Roll-up Totals and Sub-
Totals for drill-downs, and (c) Cross tabulation will be
presented.
Firstly the problem with histograms is that the SQL
standard Group-by operator does not allow a direct con-
struction with aggregation over computed categories.
Unfortunately, not all SQL systems directly support his-
tograms including the standard. In standard SQL, histo-
grams are computed indirectly from a table-valued expres-
sion which is then aggregated. However, the cube-by
operator is able to have direct construction. The histo-
gram can be easily obtained from the data cube using the
cube-by operator to compute the raw data. The histogram
on the right-hand side of Figure 1 shows the sales amount
as well as the sub-totals and total of a range of the
TEAM LinG

Figure 2. An example of cross-tabulation from a data cube
products at a range of the time-frame on a particular
location.
The second problem relates to roll-ups totals and sub-
totals for drill-down. Reports commonly aggregate data
initially at a coarse level, and then at successively finer
levels. This type of report is difficult with the normal SQL
construct. However, the Cube-by operator is able to
present the roll-ups totals and sub-totals for drill-down
easily.
The third problem relates to cross-tabulation which is
difficult to construct with the current standard SQL. The
symmetric aggregation result is cross-tabulation table or
cross tab for short (known as a pivot-table in spread-
sheets). Using the cube-by operator, cross tab data can
be readily obtained which is routinely displayed in the
more compact format as shown in Figure 2. This cross tab
is a two dimensional aggregation within the red-dotted
line. If we add another location such as L002, it becomes
a 3D aggregation.
In summary, the problem of representing aggregate
data in a relational data model with the standard SQL can
be a difficult and daunting task. A six dimensional cross-
tab will require a 64 way union of 64 different group-by
operators in order to build the underlying representation.
This is an important reason why the use of Group-bys is
inadequate as the resulting representation of aggregation
is too complex for optimal analysis.
MAIN THRUST
Birth of cube-by operator
To overcome the difficulty with these SQL aggregation
constructs, (Gray et al., 1996) proposed using data cube
470
Evolution of Data Cube Computational Approaches
operators (also known as cube-by) to conveniently sup-
port such aggregates. The data cube is identified as the
core operator in data warehousing and OLAP
(Lakshmanan, Pei & Zhao, 2003). The cube-by operator
computes group-bys corresponding to all possible com-
binations in a list of attributes. An example of data cube
query is as follows:
SELECT Product, Year, City, SUM(amount)
FROM Sales
CUBE BY Product, Year, City
The above query will produce the SUM of amount of
all tuples in the database according the 7 group-bys, i.e.
(Product, Year, City), (Product, Year), (Product, City),
(Year, City), (Product), (Year), (City). Lastly, the 8th group-
by denotes as ALL, which contains an empty attribute so
as to make all group-bys results union compatible. For
example, a cube-by of three attributes (ABC) in data cube
query will generate eight or 23 group-bys of ([ABC], [AB],
[AC], [BC], [A], [B], [C] and [ALL]).
The most straightforward way to execute the data
cube query is to rewrite it as a collection of eight group-
by queries and execute them separately as shown in
Figure 3. This means that the eight group-by queries will
need to access the raw data eight times. It is likely to be
quite expensive in execution time. If the number of dimen-
sion attributes increases, it becomes very expensive to
compute the data cube. This is because the required
computation cost grows exponentially with the increase
of dimension attributes. For instance, N number of
dimension attributes of cube-by will form a 2N number of
group-bys. However, there are a number of ways in which
this simple solution can be improved.
TEAM LinG
Evolution of Data Cube Computational Approaches
Figure 3. A straightforward way to execute each of the
group-by separately
Lattice Framework
Harinarayan, Rajaraman & Ullman (1996) noted that some
of the group-by queries in the data cube query could be
answered using the results of other. This is known as
dependence relation on queries. Consider a situation
where there are two queries Q1 and Q2. We can say that Q1
is a subset of Q 2 when Q1can be answered by using the
result of Q2. In this case, Q1 is dependent on Q2. This results
in a formation of lattice diagram as shown in Figure 4.
The cuboid may or may not have a common prefix with
its parent. For a simple illustration, [AB] and [AC] have
common prefix [A] with their parent [ABC]. However, there
is no common prefix between [BC] and its parent [ABC]. An
edge from a cuboid to its child indicates that the child
cuboid can be computed from its parent.
Figure 5. An example of how A and B can be derived from AB
Figure 4. A lattice of the cube-by operator
-
Figure 4 shows a pattern of the eight group-bys based
on cube-by with 3 attributes. One unique feature of the
cube-by computation is that the computation of the
various group-bys are not independent of each other,
but are closely related as some of them can be computed
using others. Figure 5 shows a simple example where the
results of A and B are both derived from the group-by of
two attributes AB (T# and P#)
Why is AB chosen in preference to AC or BC in order
to get both the results of A and B? A number of factors
need to be considered when choosing the best candidate
to obtain the result of the others group-by. Table 1 shows
how complex it can be especially when the number of
attributes is increased.
471
TEAM LinG

Table 1. An illustration of the increment in the number of paths and group-bys
Table 1 illustrates the increment in the number of paths
and group-bys whenever there is an increase in one
attribute. For example, three attributes [ABC] will result in
3 D cuboid level and 12 paths if we increase one more
attributes [ABCD], this will generate 4 D cuboid level and
32 paths. When the number of attributes is small, the
number of group-bys and number of paths are relatively
similar. However, when the number of attributes increases,
the number of paths increases significantly faster by a
factor of Np-1 where p is the path number. For instance,
when Na is equal to 2, Ng and Np are equal to 4. When Na
is equal to 3, Ng is 23 = 8 and Np = (23 + 4) = 12. It is important
to note that with the number of different paths available,
it is difficult to decide which path should be used since
some of the group-bys can be computed by using others.
Hierarchies in Lattice Diagram
In real-life environments, the dimensions of a data cube
normally consist of more than one attribute, and the
dimensions are organized as hierarchies of these at-
tributes. Harinarayan, Rajaraman, & Ullman (1996) used a
simple time dimension example to illustrate a hierarchy:
day, month, and year in Figure 6. Hierarchies are very
important, as they form a basis of two frequently used
querying operations: drill-down and roll-up. Drill-up
is the process of viewing data at gradually more detailed
levels while roll-up is just the opposite. More details can
be found in (Ramakrishnan & Gehrke, 2003, p. 852).
Figure 7 shows an example of real-life data set, which
can be viewed conceptually as a two-dimensional array
Figure 6. A hierarchy example of time attributes
Day
Week Month
Year
None
472
Evolution of Data Cube Computational Approaches
with hierarchies on the dimensions. Lis represent Stores
and Pis represent Products. Stores L1 L3 are in Churchill,
while L4 L6 are in Morwell, and this roll-up into two
towns. Products P1 P3 are of type Cup, while products
P4 P5 are of type Plate. Both cup and plate are further
grouped into the category Kitchenware. The xs are sales
volumes; entries that are blank correspond to (product,
store) combinations for which there are no sales.
In summary, there are two types of query dependen-
cies: dimension dependency, and attribute dependency.
Dimension dependency is present when there is an inter-
action of the different dimensions with one another as
shown in Figures 4 and 5. Attribute dependency is intro-
duced when it falls within a dimension caused by attribute
hierarchies as shown in Figures 6 and 7.
Optimisations in Existing Approaches
Sarawagi, Agrawal, and Megiddo (1996) adapted these
methods to compute group-bys by incorporating a num-
ber of optimization:
Smallest-Parent: This optimization was first pro-
posed in (Gray et. al, 1997). It aims at computing a
group-by from the smallest (in terms of either clos-
est parent group-by or size of the parent group-by)
of previously computed group-by. Each group-by
can be computed from a number of other group-bys.
Figures 8 and 9 show a three attributes cube (ABC).
There are a number of options for computing a group-
by A from its group-bys parent (ABC, AB, or AC). For
example, A can be computed from ABC, AB or AC. Figure
8 shows an example where it is better choice to compute
A from smallest or closest parent group-bys AB or AC
rather than ABC. Another example shows another sce-
nario where the size of smallest-parent is smaller than the
other (Figure 9). AC is a better choice as compared to AB
in terms of size.
Cache-Results: This optimization aims at caching
(in memory) the results of a group-by from which
TEAM LinG
Evolution of Data Cube Computational Approaches
Figure 7. A hierarchy example of two attributes
Figure 8. AB or AC is clearly better choice of computing
A
Figure 10. Reduction in disk I/O by computing A from AB
instead of AC
Caching Disk
Input/Output
(in memory)
[AC]
[AB]
other group-bys are computed to reduce disk I/O.
Figure 10 shows an example AB is better choice to
compute A as compared to AC. The reason is that
A can be computed while AB is still in memory so
that there is no disk I/O cost involved.
Amortize-Scans: This optimization aims at amortiz-
ing disk reads by computing as many group-bys as
possible, together in memory. In Figure 11 as many
group-bys as possible are computed such as AB,
AC and A from ABC while ABC is still in the memory.
Thus, there is no need to involve disk read.
Share-Sorts: This optimization is specific to the
sort-based algorithms and aims at sharing sorting
cost across multiple group-bys. Share-sort is using
data sorted in a particular order to compute all
group-bys that are prefixes of that order. There is no
need to re-sort for the subsequent group-bys as it
-
Figure 9. An example of different size in AB and AC
Figure 11. Reduction of disk reads by computing as many
group-bys as possible in the memory
[ABC]
Disk Memory
Input/Output
[AB, AC, A]
has brought equal items together, and duplicate
removal will then be easy (Figure 12).
Share-Partitions: This optimization is specific to
the hash-based algorithms and share the partition
costs across multiple group-bys. Data to be aggre-
gated is usually too large for the hash-tables to fit
in memory. Hence, the conventional way to deal
with limited memory when constructing hash tables
is to partition the data on one or more attributes. If
data is partitioned on an attribute, say A, then all
group-bys that contain A can be computed by
independently grouping on each partition.
Unfortunately, the above optimizations are often
contradictory for OLAP databases especially when the
size of the data to be aggregated is usually much larger
than the available main memory. For instance, a group-by
473
TEAM LinG

Figure 12. An example of share-sorts
A sorted
AB sorted
ABC Sort
[A] can be computed from one of the several parent group-
bys ([AB] [ABC] [AC]), but the bigger one AC (in term of
size) is in memory and the smallest one AB is not. In this
case, based on the Cache-results optimization, AC maybe
is a better choice.
With the possible optimization techniques, Agarwal
et al. (1996) suggested two proposed approaches, which
are basically sort-based PipeSort and hash-based
PipeHash. However, there is a need for some global
planning, which uses the search lattice introduced in
(Harinarayan et al., 1996).
Search Lattice
The search lattice is a graph where a vertex represents a
group-by of the cube as shown in Figure 14. A directed
edge connects group-by i to group-by j whenever j can be
generated from i and j has exactly one attribute less than
i (i is called the parent of j, for instance, AB is called the
parent of A). Level k of the search lattice denotes all
group-bys that contain exactly k attributes.
Data Cube Computation Methodology
PipeSort: Agrawal et al. (1996) have incorporated
the share-sorts, cache-results and amortize-scans
into the PipeSort algorithm. The aim is to get mini-
mum total cost and also to use Pipelining to achieve
cache-results and amortize-scans. However, the main
limitation is that it does not scale well with respect
to the number of Cube-by attributes. PipeSort per-
forms one sort operation for the pipelined evalua-
tion of each path. When the underlying relation is
sparse and much larger than available memory, this
results in many of the cuboids that PipeSort sorts
are also larger than the available memory. Hence,
this has resulted a considerable amount of I/O when
performing PipeSort.
PipeHash: The PipeHash algorithm is able to in-
clude the four stated optimizations only if the
memory is available. First optimization is smallest-
parent where PipeHash has fixed the parent group-
474
Evolution of Data Cube Computational Approaches
Figure 13. An example of sharing partitioning cost
A
AB AC
ABC partitioned on A
by for each group-by and this has resulted in biased
towards optimizing for smallest-parent. Second
optimization, share-partitions, is achieved by com-
puting from the same partition all group-bys that
contain the partitioning attribute. Third and fourth
optimizations are achieved when computing a
subtree, the algorithm maintains all hash-tables of
group-bys in the subtree in memory until all its
children are created and also for each group-by,
therefore its children can be computed in one scan
of the group-by. However, the limitation of PipeHash
algorithm relates to the NP-Hard problem especially
in minimizing overall disk scan cost.
Overlap: Deshpande et al. (1998) proposed the
OVERLAP algorithm for data cube computation and
it based on sorting-based method. The Overlap
algorithm is executed in four stages. The overlap
algorithm has minimized the number of sorting steps
required to compute many sub-aggregates and also
minimized the number of disk accesses by overlap-
ping the computation of the cuboids. However, the
limitation of the overlap algorithm is that the memory
may be not large enough to store more than one such
partition simultaneously. This is because each of
the O(k) nodes has O(k) such descendants, so there
are at least O(k2) nodes in the search tree that
involve additional disk I/O. As a result, the total I/
O cost of OVERLAP is at least quadratic in k for
sparse data.
Partitioning: Ross et al. (1997) have taken into
consideration the fact that real data is frequently
sparse. It exists for two reasons: (a) large domain
sizes of some cube-by attributes and (b) large num-
ber of cube-by attributes in the data cube query. In
(Ross et al., 1997), large relations are partitioned into
fragments that fit in memory. Thus, there is always
enough memory to fit in the fragments of large
relation. The memory-cube is similar to PipeSort as
it computes the various cuboids (group-bys) of the
data cube using the idea of pipelined paths. The
memory-cube performs multiple in-memory sorts,
and does not incur any I/O beyond the input of the
relation and the output of the data cube itself.
TEAM LinG
Evolution of Data Cube Computational Approaches
Figure 14. A lattice search of the cube-by operator
Array Computation: Zhao, Deshpande, Naughton
and Shukla (1997) introduced the multi-way array
cubing algorithm in which the cells are visited in the
right order so that a cell does not have to be revisited
for each sub-aggregate. Two main issues: (a) the
array itself is far too large to fit in memory especially
in a multidimensional application; and (b) many of
the cells in the array are empty. The goal is to overlap
the computation of all these group-bys and finish
the cube in one scan of the array with the require-
ment of memory minimized.
Table 2 shows a summary of benefits and optimization
of existing techniques. The existing techniques show that
their performance based on uni-processor environments.
They either heavily relied on estimation of memory re-
quirements, or simply partitioned data into equal-sized
partitions based on a uniform distribution assumption.
FUTURE TRENDS
Other approaches have utilized parallel techniques (Datta,
et al., 2002; Tan, Taniar & Lu, 2003; Taniar & Tan, 2002).
Dehne et al. (2003) have proposed the use of parallel
techniques on data cubes whereas (Lu, Yu, Feng, & Li.,
2003) focused on the handling of data skew in parallel data
cube computations. However, due to the complexity of the
data cube there is still capacity for computations to be
further optimized. There is thus still scope for researchers
to address these new challenges.
-
CONCLUSION
Data cube computation requires all possible combina-
tions of group-bys which correspond to the attributes.
However, due to the complexity associated with data cube
computation, a number of approaches have been pro-
posed. In spite of these approaches, there are a number of
research issues that still need to be resolved.
REFERENCES
Agarwal, S., Agrawal, R., Deshpande, P.M., Gupta, A.,
Naughton, J.F., Ramakrishnan, R. et al. (1996, September).
On the computation of multidimensional aggregates. In-
ternational VLDB Conference (pp. 506-521), Bombay,
India.
Datta, A., VanderMeer, D., & Ramamritham, K. (2002).
Parallel star join + DataIndexes: Efficient query process-
ing in data warehouses and OLAP. IEEE Transactions
Journal on Knowledge & Data Engineering, 14, 1299-
1316.
Dehne, F., Eavis, T., Hambrusch, S., & Rau-Chaplin, A.
(2002). Parallelizing the data cube. Journal of Distributed
& Parallel Databases, 11, 181-201.
Elmasri, R., & Navathe, S.B. (2004). Fundamentals of
database systems. Boston, MA: Addison Wesley.
Gray, J., Bosworth, A., Lyaman, A., & Pirahesh, H. (1996,
June). Data cube: A relational aggregation operator gen-
eralizing GROUP-BY, CROSS-TAB, and SUB-TOTALS.
International ICDE Conference (pp. 152-159), New Or-
leans, Louisiana.
475
TEAM LinG

Table 2. Benefits and optimization of existing techniques
Harinarayan, V., Rajaraman, A., & Ullman, J. (1996, June).
Implementing data cubes efficiently. International ACM
SIGMOD Conference (pp. 205-216), Montreal, Canada.
Lakshmanan, L.V.S., Pei, J., & Zhao, Y. (2003, September).
Efficacious data cube exploration by semantic summariza-
tion and compression. International VLDB Conference
(pp. 1125-1128), Berlin, Germany.
Lu, H.J., Yu, J.X., Feng, L., & Li, Z.X. (2003). Fully dynamic
partitioning: Handling data skew in parallel data cube
computation. Journal Distributed & Parallel Databases,
13, 181-202.
Ramakrishnan, R., & Gehrke, J. (2003). Database manage-
ment systems. NY: McGraw-Hill.
Ross, K.A., & Srivastava, D. (1997, August). Fast compu-
tation of sparse datacubes. International VLDB Confer-
ence (pp. 116-185), Athens, Greece.
Sarawagi, S., Agrawal, R., & Megiddo, N. (1998, March).
Discovery-driven exploration of OLAP data cubes. Inter-
national EDBT Conference (pp. 168-182), Valencia, Spain.
Silberschatz, A., Korth, H., & Sudarshan, S. (2002). Data-
base system concepts. NY: McGraw-Hill.
Tan, R.B.N., Taniar, D., & Lu, G.J. (2003, March). Efficient
execution of parallel aggregate data cube queries in data
warehouse environments. International IDEAL Confer-
ence (pp. 709-7016), Hong Kong, China.
Taniar, D., & Tan, R.B.N. (2002, May). Parallel processing
of multi-join expansion_aggregate data cube query in
high performance database systems. International I-
SPAN Conference (pp. 51-58), Manila, Philippines.
Zhao, Y., Deshpande, P., Naughton, J., & Shukla, A. (1998,
June). Simultaneous optimization and evaluation of mul-
476
Evolution of Data Cube Computational Approaches
tiple dimensional queries. International ACM SIGMOD
Conference (pp. 271-282), Seattle, Washington.
KEY TERMS
Amortize-Scans: Amortizing disk reads by comput-
ing as many group-bys as possible, simultaneously in
memory.
Attribute Dependency: Introduces when it falls within
a dimension caused by attribute hierarchies.
Cache-Results: the result of a group-by is obtained
from other group-bys computation (in memory).
Data Cube: Is known as core operator in data ware-
house and OLAP (Lakshmanan, Pei & Zhao, 2003).
Data Cube Operator: Computes Group-by correspond-
ing to all possible combinations of attributes in the Cube-
by clause.
Dependence Relation: Relates to data cube query
where some of the group-by queries could be answered
using the results of other.
Dimension Dependency: Presents when there is an
interaction of the different dimensions with one another.
OLAP: Describes a technology that uses a multi-
dimensional view of aggregate data to provide quick
access to strategic information for the purposes of ad-
vanced analysis.
Smallest-Parent: In terms of either closest parent
group-by or size of the parent group-by.
TEAM LinG

Evolutionary Computation and Genetic
Algorithms
William H. Hsu
Kansas State University, USA
INTRODUCTION
A genetic algorithm (GA) is a procedure used to find
approximate solutions to search problems through the
application of the principles of evolutionary biology.
Genetic algorithms use biologically inspired techniques,
such as genetic inheritance, natural selection, mutation,
and sexual reproduction (recombination, or crossover).
Along with genetic programming (GP), they are one of the
main classes of genetic and evolutionary computation
(GEC) methodologies.
Genetic algorithms typically are implemented using
computer simulations in which an optimization problem is
specified. For this problem, members of a space of candi-
date solutions, called individuals, are represented using
abstract representations called chromosomes. The GA
consists of an iterative process that evolves a working set
of individuals called a population toward an objective
function, or fitness function (Goldberg, 1989; Wikipedia,
2004). Traditionally, solutions are represented using fixed-
length strings, especially binary strings, but alternative
encodings have been developed.
The evolutionary process of a GA is a highly simpli-
fied and stylized simulation of the biological version. It
starts from a population of individuals randomly gener-
ated according to some probability distribution (usually
uniform) and updates this population in steps called
generations. For each generation, multiple individuals
are selected randomly from the current population, based
upon some application of fitness, bred using crossover,
and modified through mutation, to form a new population.
Crossover: Exchange of genetic material
(substrings) denoting rules, structural components,
features of a machine learning, search, or optimiza-
tion problem.
Selection: The application of the fitness criterion to
choose which individuals from a population will go
on to reproduce.
Replication: The propagation of individuals from
one generation to the next.
Mutation: The modification of chromosomes for
single individuals.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
477
-
This article begins with a survey of the following GA
variants: the simple genetic algorithm, evolutionary algo-
rithms, and extensions to variable-length individuals. It
then discusses GA applications to data-mining problems,
such as supervised inductive learning, clustering, and
feature selection and extraction. It concludes with a dis-
cussion of current issues in GA systems, particularly
alternative search techniques and the role of building
block (schema) theory.
BACKGROUND
The field of genetic and evolutionary computation (GEC)
was first explored by Turing, who suggested an early
template for the genetic algorithm. Holland (1975) per-
formed much of the foundational work in GEC in the 1960s
and 1970s. His goal of understanding the processes of
natural adaptation and designing biologically inspired
artificial systems led to the formulation of the simple
genetic algorithm (Holland, 1975).
State of the Field: To date, GAs have been applied
successfully to many significant problems in ma-
chine learning and data mining, most notably clas-
sification, pattern detectors (Gonzlez & Dasgupta,
2003; Rizki et al., 2002) and predictors (Au et al.,
2003), and payoff-driven reinforcement learning1
(Goldberg, 1989).
Theory of GAs: Current GA theory consists of two
main approachesMarkov chain analysis and
schema theory. Markov chain analysis is primarily
concerned with characterizing the stochastic dy-
namics of a GA system (i.e., the behavior of the
random sampling mechanism of a GA over time). The
most severe limitation of this approach is that, while
crossover is easy to implement, its dynamics are
difficult to describe mathematically. Markov chain
analysis of simple GAs has therefore been more
successful at capturing the behavior of evolution-
ary algorithms with selection and mutation only.
These include evolutionary algorithms (EAs) and
evolutions strategie (Schwefel, 1977).
TEAM LinG

Successful building blocks can become redundant in
a GA population. This can slow down processing and also
can result in a phenomenon called takeover, where the
population collapses to one or a few individuals. Goldberg
(2002) characterizes steady-state innovation in GAs as the
situation where time to produce a new, more highly-fit
building block (the innovation time, ti) is lower than the
expected time for the most fit individual to dominate the
entire population (the takeover time, t*). Steady state
innovation is achieved, facilitating convergence toward an
optimal solution, when ti < t*, because the countdown to
takeover, or race, between takeover and innovation is reset.
MAIN THRUST
The general strengths of genetic algorithms lie in their
ability to explore the search space efficiently through
parallel evaluation of fitness (Cant-Paz, 2000) and mixing
of partial solutions through crossover (Goldberg, 2002); to
maintain a search frontier to seek global optima (Goldberg,
1989); and to solve multi-criterion optimization problems.
The basic units of partial solutions are referred to in the
literature as building blocks, or schemata. Modern GEC
systems also are able to produce solutions of variable
length (De Jong et al., 1993; Kargupta & Ghosh, 2002).
A more specific advantage of GAs is their ability to
represent rule-based, permutation-based, and construc-
tive solutions to many pattern-recognition and machine-
learning problems. Examples of this include induction of
decision trees (Cant-Paz & Kamath, 2003) among several
other recent applications surveyed in the following.
Types of Gas
The simplest genetic algorithm represents each chromo-
some as a bit string (containing binary digits 0 and 1) of
fixed length. Numerical parameters can be represented by
integers, though it is possible to use floating-point rep-
resentations for reals. The simple GA performs crossover
and mutation at the bit level for all of these (Goldberg,
1989; Wikipedia, 2004).
Other variants treat the chromosome as a parameter
list, containing indices into an instruction table or an
arbitrary data structure with pre-defined semantics (e.g.,
nodes in a linked list, hashes, or objects. Crossover and
mutation are required to preserve semantics by respecting
object boundaries, and formal invariants for each genera-
tion can be specified according to these semantics. For
most data types, operators can be specialized, with differ-
ing levels of effectiveness that generally are domain-
dependent (Wikipedia, 2004).
478
Evolutionary Computation and Genetic Algorithms
Applications
Genetic algorithms have been applied to many classifica-
tion and performance tuning applications in the domain of
knowledge discovery in databases (KDD). De Jong, et al.
produced GABIL (Genetic Algorithm-Based Inductive
Learning), one of the first general-purpose GAs for learn-
ing disjunctive normal form concepts (De Jong et al.,
1993). GABIL was shown to produce rules achieving
validation set accuracy comparable to that of decision
trees induced using ID3 and C4.5.
Since GABIL, there has been work on inducing rules
(Zhou et al., 2003) and decision trees (Cant-Paz & Kamath,
2003) using evolutionary algorithms. Other representa-
tions that can be evolved using a genetic algorithm
include predictors (Au et al., 2003) and anomaly detectors
(Gonzlez & Dasgupta, 2003). Unsupervised learning
methodologies, such as data clustering (Hall et al., 1999;
Lorena & Furtado, 2001) also admit GA-based represen-
tation, with application to such current data-mining prob-
lems as gene expression profiling in the domain of compu-
tational biology (Iba, 2004). KDD from text corpora is
another area where evolutionary algorithms have been
applied (Atkinson-Abutridy et al., 2003).
GAs can be used to perform meta-learning, or higher-
order learning, by extracting features (Raymer et al., 2000),
selecting features (Hsu, 2003), or selecting training in-
stances (Cano et al., 2003). They also have been applied
to combine, or fuse, classification functions (Kuncheva &
Jain, 2000).
FUTURE TRENDS
Some limitations of GAs are that, in certain situations,
they are overkill compared to more straightforward opti-
mization methods such as hill-climbing, feed-forward ar-
tificial neural networks using back propagation, and even
simulated annealing and deterministic global search. In
global optimization scenarios, GAs often manifest their
strengths: efficient, parallelizable search; the ability to
evolve solutions with multiple objective criteria (Llor &
Goldberg, 2003); and a characterizable and controllable
process of innovation.
Several current controversies arise from open research
problems in GEC:
Selection is acknowledged to be a fundamentally
important genetic operator. Opinion, however, is
divided over the importance of crossover vs. muta-
tion. Some argue that crossover is the most impor-
tant, while mutation is only necessary to ensure that
potential solutions are not lost. Others argue that
TEAM LinG
Evolutionary Computation and Genetic Algorithms
crossover in a largely uniform population only serves
to propagate innovations originally found by muta-
tion, and that in a non-uniform population, cross-
over is nearly always equivalent to a very large
mutation, which is likely to be catastrophic.
In the field of GEC, basic building blocks for solu-
tions to engineering problems primarily have been
characterized using schema theory, which has been
critiqued as being insufficiently exact to characterize
the expected convergence behavior of a GA. Propo-
nents of schema theory have shown that it provides
useful normative guidelines for design of GAs and
automated control of high-level GA properties (e.g.,
population size, crossover parameters, and selec-
tion pressure).
Recent and current research in GEC relates certain
evolutionary algorithms to ant colony optimization
(Parpinelli, Lopes & Freitas, 2002).
CONCLUSION
Genetic algorithms provide a comprehensive search method-
ology for machine learning and optimization. It has been
shown to be efficient and powerful through many data-
mining applications that use optimization and classification.
The current literature (Goldberg, 2002; Wikipedia, 2004)
contains several general observations about the genera-
tion of solutions using a genetic algorithm:
GAs are sensitive to deceptivity, the irregularity of
the fitness landscape. This includes locally optimal
solutions that are not globally optimal, lack of fitness
gradient for a given step size, and jump discontinuities
in fitness.
In general, GAs have difficulty with adaptation to
dynamic concepts or objective criteria. This phe-
nomenon, called concept drift in supervised learning
and data mining, is a problem, because GAs tradi-
tionally are designed to evolve highly fit solutions
(populations containing building blocks of high rela-
tive and absolute fitness) with respect to stationary
concepts.
GAs are not always effective at finding globally
optimal solutions but can rapidly locate good solu-
tions, even for difficult search spaces. This makes
steady-state GAs (i.e., Bayesian optimization GAs
that collect and integrate solution outputs after
convergence to an accurate representation of build-
ing blocks) a useful alternative to generational GAs
(maximization GAs that seek the best individual of
the final generation after convergence).
Looking ahead to future opportunities and challenges
in data mining, genetic algorithms are widely applicable -
to classification by means of inductive learning. GAs
also provide a practical method for optimization of data
preparation and data transformation steps. The latter
includes clustering, feature selection and extraction, and
instance selection. In data mining, GAs likely are most
useful where high-level, fitness-driven search is needed.
Non-local search (global search or search with an adap-
tive step size) and multi-objective data mining are also
problem areas where GAs have proven promising.
REFERENCES
Atkinson-Abutridy, J., Mellish, C., & Aitken, S. (2003). A
semantically guided and domain-independent evolution-
ary model for knowledge discovery from texts. IEEE Trans-
actions on Evolutionary Computation, 7(6), 546-560.
Au, W.-H., Chan, K.C.C., & Yao, X. (2003). A novel
evolutionary data mining algorithm with applications to
churn prediction. IEEE Transactions on Evolutionary
Computation, 7(6), 532-545.
Cano, J.R., Herrera, F., & Lozano, M. (2003). Using evo-
lutionary algorithms as instance selection for data reduc-
tion in KDD: An experimental study. IEEE Transactions
on Evolutionary Computation, 7(6), 561-575.
Cant-Paz, E. (2000). Efficient and accurate parallel
genetic algorithms. Norwell, MA: Kluwer.
Cant-Paz, E., & Kamath, C. (2003). Inducing oblique
decision trees with evolutionary algorithms. IEEE Trans-
actions on Evolutionary Computation, 7(1), 54-68.
De Jong, K.A., Spears, W.M., & Gordon, F.D. (1993).
Using genetic algorithms for concept learning. Machine
Learning, 13, 161-188.
Goldberg, D.E. (1989). Genetic algorithms in search,
optimization, and machine learning. Reading, MA:
Addison-Wesley.
Goldberg, D.E. (2002). The design of innovation: Lessons
from and for competent genetic algorithms. Norwell,
MA: Kluwer.
Gonzlez, F.A., & Dasgupta, D. (2003). Anomaly detec-
tion using real-valued negative selection. Genetic Pro-
gramming and Evolvable Machines, 4(4), 383-403.
Hall, L.O., Ozyurt, I.B., & Bezdek, J.C. (1999). Clustering
with a genetically optimized approach. IEEE Transac-
tions on Evolutionary Computation, 3(2), 103-112.
479
TEAM LinG

Holland, J.H. (1975). Adaptation in natural and artificial
systems. Ann Arbor, MI: University of Michigan Press.
Hsu, W.H. (2003). Control of inductive bias in supervised
learning using evolutionary computation: A wrapper-
based approach. In J. Wang (Ed.), Data mining: Oppor-
tunities and challenges. Hershey, PA: Idea Group Pub-
lishing.
Iba, H. (2004). Classification of gene expression profile
using combinatory method of evolutionary computation
and machine learning. Genetic Programming and Evolv-
able Machines, Special Issue on Biological Applica-
tions of Genetic and Evolutionary Computation, 5(2),
145-156.
Kargupta, H., & Ghosh, S. (2002). Toward machine learn-
ing through genetic code-like transformations. Genetic
Programming and Evolvable Machines, 3(3), 231-258.
Kuncheva, L.I., & Jain, L.C. (2000). Designing classifier
fusion systems by genetic algorithms. IEEE Transactions
on Evolutionary Computation, 4(4), 327-336.
Llor, X., & Goldberg, D.E. (2003). Bounding the effect of
noise in multiobjective learning classifier systems. Evolu-
tionary Computation, 11(3), 278-297.
Lorena, L.A.N., & Furtado, J.C. (2001). Constructive ge-
netic algorithm for clustering problems. Evolutionary
Computation, 9(3), 309-328.
Parpinelli, R.S., Lopes, H.S., & Freitas, A.A. (2002). Data
mining with an ant colony optimization algorithm. IEEE
Transactions on Evolutionary Computation, 6(4), 321-332.
Raymer, M.L., Punch, W.F., Goodman, E.D., Kuhn, L.A.,
& Jain, A.K. (2000). Dimensionality reduction using ge-
netic algorithms. IEEE Transactions on Evolutionary
Computation, 4(2), 164-171.
Rizki, M.M., Zmuda, M.A., & Tamburino, L.A. (2002).
Evolving pattern recognition systems. IEEE Transac-
tions on Evolutionary Computation, 6(6), 594-609.
Schwefel, H.-P. (1997). Numerische optimierung von com-
puterModellen mittels der evolutionsstrategie. Basel:
Birkhauser Verlag.
Wikipedia. (2004). Genetic algorithm. Retrieved from
http://en.wikipedia.org/wiki/Genetic_algorithm
Zhou, C., Xiao, W., Tirpak, T. M., & Nelson, P.C. (2003).
Evolving accurate and compact classification rules with
gene expression programming. IEEE Transactions on
Evolutionary Computation, 7(6), 519-531.
480
Evolutionary Computation and Genetic Algorithms
KEY TERMS
Crossover: In biology, a process of sexual recombina-
tion, by which two chromosomes are paired up and ex-
change some portion of their genetic sequence. Cross-
over in GAs is highly stylized and typically involves
exchange of strings. These can be performed using a
crossover bit mask in bit-string GAs but require complex
exchanges (i.e., partial-match, order, and cycle crossover)
in permutation GAs.
Evolutionary Computation: A solution approach based
on simulation models of natural selection, which begins
with a set of potential solutions and then iteratively
applies algorithms to generate new candidates and select
the fittest from this set. The process leads toward a model
that has a high proportion of fit individuals.
Generation: The basic unit of progress in genetic and
evolutionary computation, a step in which selection is
applied over a population. Usually, crossover and muta-
tion are applied once per generation, in strict order.
Individual: A single candidate solution in genetic and
evolutionary computation, typically represented using
strings (often of fixed length) and permutations in genetic
algorithms, or using problem-solver representations (pro-
grams, generative grammars, or circuits) in genetic pro-
gramming.
Meta-Learning: Higher-order learning by adapting
the parameters of a machine-learning algorithm or the
algorithm definition itself, in response to data. An example
of this approach is the search-based wrapper of inductive
learning, which wraps a search algorithm around an in-
ducer to find locally optimal parameter values such as the
relevant feature subset for a given classification target
and data set. Validation set accuracy typically is used as
fitness. Genetic algorithms have been used to implement
such wrappers for decision tree and Bayesian network
inducers.
Mutation: In biology, a permanent, heritable change
to the genetic material of an organism. Mutation in GAs
involves string-based modifications to the elements of a
candidate solution. These include bit-reversal in bit-
string GAs and shuffle and swap operators in permutation
GAs.
Permutation GA: A type of GA where individuals
represent a total ordering of elements, such as cities to be
visited in a minimum-cost graph tour (the Traveling Sales-
man Problem). Permutation GAs use specialized cross-
over and mutation operators compared to the more com-
mon bit string GAs.
TEAM LinG
Evolutionary Computation and Genetic Algorithms
Schema (pl. Schemata): An abstract building block of
a GA-generated solution, corresponding to a set of indi-
viduals. Schemata typically are denoted by bit strings
with dont-care symbols # (e.g., 1#01#00# is a schema
with 23 = 8 possible instances, one for each instantiation
of the # symbols to 0 or 1). Schemata are important in GA
research, because they form the basis of an analytical
approach called schema theory, for characterizing build-
ing blocks and predicting their proliferation and survival
probability across generations, thereby describing the
expected relative fitness of individuals in the GA.
Selection: In biology, a mechanism in by which the
fittest individuals survive to reproduce and the basis of
speciation according to the Darwinian theory of evolu-
tion. Selection in GP involves evaluation of a quantitative
criterion over a finite set of fitness cases, with the com-
bined evaluation measures being compared in order to
choose individuals. -
ENDNOTE
1
Payoff-driven reinforcement learning describes a
class of learning problems for intelligent agents that
receive rewards, or reinforcements, from the envi-
ronment in response to actions selected by a policy
function. These rewards are transmitted in the form
of payoffs, sometimes strictly non-negative. A GA
acquires policies by evolving individuals, such as
condition-action rules, that represent candidate
policies.
481
TEAM LinG
482
Evolutionary Data Mining for Genomics
Laetitia Jourdan
LIFL, University of Lille 1, France
Clarisse Dhaenens
LIFL, University of Lille 1, France
El-Ghazali Talbi
LIFL, University of Lille 1, France
INTRODUCTION
Knowledge discovery from genomic data has become an
important research area for biologists. Nowadays, a lot
of data is available on the Web, but it is wrong to say that
corresponding knowledge is also available. For example,
the first draft of the human genome, which contains
3,000,000,000 letters, was achieved in June 2000, but,
up to now, only a small part of the hidden knowledge has
been discovered. This is the aim of bioinformatics,
which brings together biology, computer science, math-
ematics, statistics, and information theory to analyze
biological data for interpretation and prediction. Hence,
many problems encountered while studying genomic
data may be modeled as data mining tasks, such as
feature selection, classification, clustering, or associa-
tion rule discovery.
An important characteristic of genomic applications
is the large amount of data to analyze, and, most of the
time, it is not possible to enumerate all the possibilities.
Therefore, we propose to model these knowledge dis-
covery tasks as combinatorial optimization tasks in
order to apply efficient optimization algorithms to ex-
tract knowledge from large datasets. To design an effi-
cient optimization algorithm, several aspects have to be
considered. The main one is the choice of the type of
resolution method according to the characteristics of
the problem. Is it an easy problem, for which a polyno-
mial algorithm may be found? If the answer is yes, then
let us design such an algorithm. Unfortunately, most of
the time, the response to the question is no, and only
heuristics that may find good but not necessarily opti-
mal solutions can be used. In our approach, we focus on
evolutionary computation, which has already shown an
interesting ability to solve highly complex combinato-
rial problems.
In this article, we will show the efficacy of such an
approach while describing the main steps required to
solve data mining problems from genomics with evolu-
tionary algorithms. We will illustrate these steps with a
real problem.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
Evolutionary data mining for genomics groups three
important fields: evolutionary computation, knowledge
discovery, and genomics.
It is now well known that evolutionary algorithms are
well suited for some data mining tasks (Freitas, 2002).
Here, we want to show the interest of dealing with
genomic data, thanks to evolutionary approaches. A first
proof of this interest may be the recent book by Gary
Fogel and David Corne, Evolutionary Computation in
Bioinformatics, which groups several applications of
evolutionary computation to problems in the biological
sciences and, in particular, in bioinformatics (Fogel &
Corne, 2002). In this article, several data mining tasks
are addressed, such as feature selection or clustering,
and solved, thanks to evolutionary approaches.
Another proof of the interest of such approaches is
the number of sessions around evolutionary computa-
tion in bioinformatics and computational biology that
have been organized during the last Congress on Evolu-
tionary Computation (CEC) in Portland, Oregon in 2004.
The aim of genomic studies is to understand the
function of genes, to determine which genes are in-
volved in a given process, and how genes are related.
Hence, experiments are conducted, for example, to
localize coding regions in DNA sequences and/or to
evaluate the expression level of genes in certain condi-
tions. Resulting from this, data available for the
bioinformatics researcher may deal with DNA sequence
information that are related to other types of data. The
example used to illustrate this article may be classified
in this category.
Another type of data deals with the recent technol-
ogy called microarray, which allows the simultaneous
measurement of the expression level of thousands of
genes under different conditions (i.e., various time
points of a process, absorption of different drugs, etc.).
This new type of data requires specific data mining tasks,
as the number of genes to study is very large and the
TEAM LinG
Evolutionary Data Mining for Genomics
number of conditions may be limited. Classical questions
are the classification or the clustering of genes based on
their expression pattern, and commonly used approaches
may vary from statistical approaches (Yeung & Ruzzo,
2001) to evolutionary approaches (Merz, 2002) and may
use additional biological information, such as gene ontol-
ogy (GO) (Speer, Spieth & Zell, 2004). Recently, the bi-
clustering that allows the grouping of instances having
similar characteristic for a subset of attributes (here,
genes having the same expression patterns for a subset of
conditions) has been applied to this type of data and
evolutionary approaches proposed (Bleuler, Preli &
Ziztler, 2004). In this context of microarray data analysis,
association rule discovery also has been realized using
evolutionary algorithms (Khabzaoui, Dhaenens & Talbi,
2004).
MAIN THRUST
In order to extract knowledge from genomic data using
evolutionary algorithms, several steps have to be con-
sidered:
1. Identification of the knowledge discovery task
from the biological problem under study;
2. Design of this task as an optimization problem;
3. Resolution using an evolutionary approach.
Hence, in this section, we will focus on each of these
steps. First, we will present the genomic application that
we will use to illustrate the rest of the article and
indicate the knowledge discovery tasks that have been
extracted. Then, we will show the challenges and some
proposed solutions for the two other steps.
Genomics Application
The genomic problem under study is to formulate hy-
potheses on predisposition factors of different multi-
factorial diseases, such as diabetes and obesity. In such
diseases, one of the difficulties is that sane people can
become affected during their life, so only the affected
status is relevant. This work has been done in collabora-
tion with the Biology Institute of Lille (IBL, France).
One approach aims to discover the contribution of
environmental factors and genetic factors in the patho-
genesis of the disease under study by discovering com-
plex interactions, such as ([gene A and gene B] or [gene
C and environmental factor D]) in one or more popula-
tion. The rest of the article will use this problem as an
illustration.
To solve such a problem, the first thing is to formu-
late it into a classical data mining task. The difficulty of
such a formulation is to identify the task. This work must
be done through discussions and cooperation with bi- -
ologists in order to agree on the objective of the prob-
lems. For example, in our data, identifying groups of
people can be modeled as a clustering task, as we cannot
take into account non-affected people. Moreover, a lot
of loci have to be studied (3,652 points of comparison
on the 23 chromosomes and two environmental factors)
and classical clustering algorithms are not able to cope
with so many points. So, we decided first to execute a
feature selection in order to reduce the number of loci
in consideration and to extract the most influential
features that will be used for the clustering. Hence, the
model of this problem is decomposed into two phases:
feature selection and clustering.
From a Data Mining Task to an
Optimization Problem
The most difficult aspect of turning a data mining task
into an optimization problem is to define the criterion
to optimize. The choice of the optimization criterion,
which measures the quality of candidate knowledge to
be extracted, is very important, and the quality of the
results of the approach depends on it. Indeed, develop-
ing a very efficient method that does not use the right
criterion will lead to obtaining the right answer to the
wrong question. The optimization criterion either can
be specific to the data mining task or dependent of the
biological application. Several different choices exist.
For example, considering the gene clustering, the opti-
mization criterion can be the minimization of the mini-
mum sum-of-squares (MSS) (Merz, 2002), while for
the determination of the members of a predictive gene
group, the criterion can be the maximization of the
classification success using a maximum likelihood
(MLHD) classification method (Ooi & Tan, 2003).
Once the optimization criterion is defined, the sec-
ond step of the design of the data mining task into an
optimization problem is to define the encoding of a
solution, which may be independent of the resolution
method. For example, for clustering problems in gene
expression mining with evolutionary algorithm,
Faulkenauer and Marchand (2001) use the specific CGA
encoding that is dedicated to grouping problems and is
well suited to clustering.
Regarding the genomic application used to illustrate
this article, two phases have been isolated. For the
feature selection, an optimization approach has been
adopted, using an evolutionary algorithm (see next para-
graph), whereas a classical approach (k-means) has been
chosen for the clustering phase. Determining the opti-
mization criterion for the feature selection was not an easy
task, as it was difficult not to favor small sets of features.
483
TEAM LinG

A corrective factor has been introduced (Jourdan et al.,
2002).
Solving with Evolutionary Algorithms
Once the formalization of the data mining task into an
optimization problem is done, resolution methods either
can be exact methods, specific heuristics, or
metaheuristics. As the space of the potential knowledge
is exponential in genomics problems (Zaki & Ho, 2000),
exact methods are almost always discarded. The draw-
backs of heuristic approaches are that it is difficult to
cope with multiple solutions and not easy to integrate
specific knowledge in a general approach. The advan-
tages of metaheuristics are that you can define a general
framework to solve the problem while specializing some
agents in order to suit a specific problem. Genetic Algo-
rithms (GAs), which represent a class of evolutionary
methods, have given good results on hard combinatorial
problems (Michalewicz, 1996).
In order to develop a genetic algorithm for knowl-
edge discovery, we have to focus on the following:
Operators
Diversification mechanisms
Intensification mechanisms
Operators allow GAs to explore the search space and
must be adapted to the problem. Generally, there are two
classes of operatorsmutation and crossover. The mu-
tation allows diversity. For the feature selection task
under study, the mutation flips n bits (Jourdan, Dhaenens
& Talbi, 2001). The crossover produces one, two, or
more children solutions by recombining two or parents.
The objective of this mechanism is to keep useful infor-
mation of the parents in order to ameliorate the solu-
tions. In the considered problem, the subset-oriented
common feature crossover operator (SSOCF) has been
used. Its objective is to produce offspring that have the
same distribution as the parents. This operator is adapted
well for feature selection (Emmanouilidis, Hunter &
MacIntyre, 2000). Another advantage of evolutionary
algorithms is that you easily can use other data mining
algorithms as an operator; for example, a kmeans itera-
tion may be used as an operator in a clustering problem
(Krishma & Murty, 1999).
Working on knowledge discovery on a particular
domain by optimization leads to definition and use of
several operators, where some may use domain knowl-
edge, and others may be specific to the model and/or the
encoding. To take advantage of all the operators, the idea
is to use adaptive mechanisms (Hong, Wang & Chen,
2000) that help to adapt application probabilities of these
operators according to the progress they produce. Hence,
484
Evolutionary Data Mining for Genomics
operators that are less efficient are less used, which may
change during the search, if they become more efficient.
Diversification mechanisms are designed to avoid
premature convergence. There exist several mecha-
nisms. The more classical are the sharing and the ran-
dom immigrant. The sharing boosts the selection of
individuals that lie in less crowded areas of the search
space (Mafhoud, 1995). To apply such a mechanism, a
distance between solutions has to be defined. In the
feature selection for the genomic association discov-
ery, a distance has been defined by integrating knowl-
edge of the application domain. The distance is corre-
lated to a Hamming distance, which integrates biologi-
cal notions (chromosomal cut, inheritance notion, etc.).
A further approach to the diversification of the
population, the random immigrant, introduces new in-
dividuals. An idea is to generate new individuals by
recording statistics on previous selections.
Assessing the efficiency of such algorithms applied
to genomic data is not easy, as most of the time,
biologists have no exact idea about what must be found.
Hence, one step in this analysis is to develop simulated
data for which optimal results are known. In this man-
ner, it is possible to measure the efficiency of the
proposed method. For example, in the problem under
study, predetermined genomic associations were con-
structed to form simulated data. Then, the algorithm
was tested on these data and found these associations.
FUTURE TRENDS
There has been much work on evolutionary data mining
for genomics. In order to be more efficient and to
propose more interesting solutions for decision mak-
ers, the researchers are investigating multi-criteria de-
sign of the data mining tasks. Indeed, we exposed that
one of the critical phases was the determination of the
optimization criterion, and it may be difficult to select a
single one. In response to this problem, the multi-crite-
ria design allows us to take into account some criteria
dedicated to a specific data mining task and some crite-
ria coming from the application domain. Evolutionary
algorithms that work on a population of solutions are
well adapted to multi-criteria problems, as they can
exploit Pareto approaches and propose several good
solutions (i.e., solutions of best compromise).
For data mining in genomics, rule discovery has not
been very well applied and should be studied carefully,
as this is a very general model. Moreover, an interest-
ing multi-criteria model has been proposed for this
task and starts to give some interesting results by using
multi-criteria genetic algorithms (Jourdan et al., 2004).
TEAM LinG
Evolutionary Data Mining for Genomics
CONCLUSION
Genomic is a real challenge for researchers in data
mining, as most of the problems encountered are diffi-
cult to address for several reasons: (1) the objectives
are not always very clear, neither for the data mining
specialist nor for the biologist, and a real dialog has to
exist between the two; (2) data may be uncertain and
noisy; (3) the number of experiments may not be enough
in comparison to the number of elements that have to be
studied.
In this article, we present the different phases re-
quired to deal with data mining problems arriving in
genomics, thanks to optimization approaches. The objec-
tive was to give for each phase the main challenges and to
illustrate through an example some responses. We saw
that the definition on the optimization criterion is a
central point of this approach, and the multi-criteria
optimization may be used as a response to this difficulty.
REFERENCES
Bleuler, S., Preli, A., & Ziztler, E. (2004). An EA
framework for biclustering of gene expression data.
Proceedings of the Congress on Evolutionary Com-
putation (CEC04), Portland, Oregon.
Emmanouilidis, C., Hunter, A., & MacIntyre, J. (2000).
A multiobjective evolutionary setting for feature selec-
tion and a commonality-based crossover operator. Pro-
ceedings of the Congress on Evolutionary Computa-
tion, San Diego, California.
Falkenauer E., & Marchand, A. (2001). Using k-means?
Consider ArrayMiner. Proceedings of the 2001 Inter-
national Conference on Mathematics and Engineer-
ing Techniques in Medicine and Biological Sciences
(METMBS2001), Las Vegas, Nevada.
Fogel, G., & Corne, D. (2002). Evolutionary computa-
tion in bioinformatics. Morgan Kaufmann.
Freitas, A.A. (2002). Data mining and knowledge dis-
covery with evolutionary algorithms. Springer-Verlag.
Hong, T-P., Wang, H-S., & Chen, W-C. (2000). Simul-
taneously applying multiple mutation operators in ge-
netic algorithms. J. Heuristics, 6(4), 439-455.
Jourdan, L., Dhaenens, C., & Talbi, E.G. (2001). An
optimization approach to mine genetic data. Proceed-
ings of Biological Data Mining and Knowledge Dis-
covery (METMBS01).
Jourdan, L., Dhaenens, C., Talbi, E.-G., & Gallina, S.
(2002). A data mining approach to discover genetic fac-
tors involved in multifactorial diseases. Knowledge Based
Systems (KBS) Journal, 15(4), 235-242. -
Jourdan, L., Khabzaoui, M., Dhaenens, C., & Talbi, E-G.
(2004). A hybrid metaheuristic for knowledge discovery in
microarray experiments. Handbook of bioinspired algo-
rithms and applications. CRC Press.
Khabzaoui, M., Dhaenens, C., & Talbi, E-G. (2004). Asso-
ciation rules discovery for DNA microarray data. Pro-
ceedings of the Bioinformatics Workshop of SIAM Inter-
national Conference on Data Mining, Orlando, Florida.
Krishna, K., & Murty, M. (1999). Genetic K-means algo-
rithm. IEEE Transactions on Systems, Man and Cybernet-
ics, 29(3), 433-439.
Mafhoud, S.W. (1995). Niching method for genetic algo-
rithms [doctoral thesis]. IL: University of Illinois.Merz, P.
(2002). Clustering gene expression profiles with memetic
algorithms. Proceedings of the 7th International Confer-
ence on Parallel Problem Solving from Nature (PPSN
VII).
Michalewicz, Z. (1996). Genetic algorithms + data struc-
tures = evolution programs. Springer-Verlag.
Ooi, C.H., & Tan, P. (2003). Genetic algorithms applied to
multi-class prediction for the analysis of gene expression
data. Bioinformatics, 19, 37-44.
Speer, N., Spieth, C., & Zell, A. (2004). A memetic co-
clustering algorithm for gene expression profiles and
biological annotation. Proceedings of the Congress on
Evolutionary Computation (CEC04), Portland, Oregon.
Yeung, K.Y., & Ruzzo, W.L. (2001). Principal compo-
nent analysis for clustering gene expression data.
Bioinformatics, 17, 763-774.
Zaki, M., & Ho, C.T. (2000). Large-scale parallel data
mining. State-of-the-Art Survey, LNAI.
KEY TERMS
Association Rule Discovery: Implication of the
form X Y, where X and Y are sets of items; implies
that if the condition X is verified, the prediction Y is
valid.
Bioinformatics: Field of science in which biology,
computer science, and information technology merge
into a single discipline.
Clustering: Data mining task in which the system
has to classify a set of objects without any information
on the characteristics of the classes.
485
TEAM LinG

Feature (or Attribute): Quantity or quality describing
an instance.
Feature Selection: Task of identifying and selecting
a useful subset of features from a large set of redundant,
perhaps irrelevant, features.
Genetic Algorithm: Evolutionary algorithm using a
population and based on the Darwinian principle, the
survival of the fittest.
486
Evolutionary Data Mining for Genomics
Multi-Factorial Disease: Disease caused by several
factors. Often, multi-factorial diseases are due to two
kinds of causality that interactone is genetic (and often
polygenetic) and the other is environmental.
Optimization Criterion: Criterion that gives the qual-
ity of a solution of an optimization problem.
TEAM LinG

Evolutionary Mining of Rule Ensembles
Jorge Muruzbal
University Rey Juan Carlos, Spain
INTRODUCTION
Ensemble rule based classification methods have been
popular for a while in the machine-learning literature
(Hand, 1997). Given the advent of low-cost, high-com-
puting power, we are curious to see how far can we go by
repeating some basic learning process, obtaining a vari-
ety of possible inferences, and finally basing the global
classification decision on some sort of ensemble sum-
mary. Some general benefits to this idea have been
observed indeed, and we are gaining wider and deeper
insights on exactly why this is the case in many fronts of
interest.
There are many ways to approach the ensemble-
building task. Instead of locating ensemble members
independently, as in Bagging (Breiman, 1996), or with
little feedback from the joint behavior of the forming
ensemble, as in Boosting (see, e.g., Schapire & Singer,
1998), members can be created at random and then made
subject to an evolutionary process guided by some
fitness measure. Evolutionary algorithms mimic the
process of natural evolution and thus involve popula-
tions of individuals (rather than a single solution itera-
tively improved by hill climbing or otherwise). Hence,
they are naturally linked to ensemble-learning methods.
Based on the long-term processing of the data and the
application of suitable evolutionary operators, fitness
landscapes can be designed in intuitive ways to prime
the ensembles desired properties. Most notably, be-
yond the intrinsic fitness measures typically used in
pure optimization processes, fitness can also be endog-
enous, that is, it can prime the context of each individual
as well.
BACKGROUND
A number of evolutionary mining algorithms are avail-
able nowadays. These algorithms may differ in the na-
ture of the evolutionary process or in the basic models
considered for the data or in other ways. For example,
approaches based on the genetic programming (GP),
evolutionary programming (EP), and classifier system
(CS) paradigms have been considered, while predictive
rules, trees, graphs, and other structures been have
evolved. See Eiben and Smith (2003) for an introduction
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
487
-
to the general GP, EP, and CS frameworks and Koza,
Keane, Streeter, Mydlowec, Yu, and Lanza (2003) for an
idea of performance by GP algorithms at the patent level.
Here I focus on ensemble rule based methods for classi-
fication tasks or supervised learning (Hand, 1997).
The CS architecture is naturally suitable for this sort
of rule assembly problems, for its basic representation
unit is the rule, or classifier (Holland, Holyoak, Nisbett,
& Thagard, 1986). Interestingly, tentative ensembles in
CS algorithms are constantly tested for successful co-
operation (leading to correct predictions). The fitness
measure seeks to reinforce those classifiers leading to
success in each case. However interesting, the CS ap-
proach in no way exhausts the scope of evolutionary
computation ideas for ensemble-based learning; see,
for example, Kuncheva and Jain (2000), Liu, Yao, and
Higuchi (2000), and Folino, Pizzuti, and Spezzano
(2003).
Ensembles of trees or rules are the natural reference
for evolutionary mining approaches. Smaller trees, made
by rules (leaves) with just a few tests, are of particular
interest. Stumps place a single test and constitute an
extreme case (which is nevertheless used often). These
rules are more general and hence tend to make more
mistakes, yet they are also easier to grasp and explain. A
related notion is that of support, the estimated probabil-
ity that new data satisfy a given rules conditions. A great
deal of effort has been done in the contemporary CS
literature to discern the idea of adequate generality, a
recurrent topic in the machine-learning arena.
MAIN THRUST
Evolutionary and Tree-Based Rule
Ensembles
In this section, I review various methods for ensemble
formation. As noted earlier, in this article, I use the
ensemble to build averages of rules. Instead of averag-
ing, one could also select the most suitable classifier in
each case and make the decision on the basis of that rule
alone (Hand, Adams, & Kelly, 2001). This alternative
idea may provide additional insights of interest, but I do
not analyze it further here.
TEAM LinG

It is conjectured that maximizing the degree of interac-
tion amongst the rules already available is critical for
efficient learning (Kuncheva & Jain, 2000; Hand et al.,
2001). A fundamental issue concerns then the extent to
which tentative rules work together and are capable of
influencing the learning of new rules. Conventional
methods like Bagging and Boosting show at most mod-
erate amounts of interaction in this sense. While Bag-
ging and Boosting are useful, well-known data-mining
tools, it is appropriate to explore other ensemble-learn-
ing ideas as well. In this article, I focus mainly on the CS
algorithm. CS approaches provide interesting architec-
tures and introduce complex nonlinear processes to
model prediction and reinforcement. I discuss a spe-
cific CS algorithm and show how it opens interesting
pathways for emergent cooperative behaviour.
Conventional Rule Assembly
In Bagging methods, different training samples are cre-
ated by bootstraping, and the same basic learning proce-
dure is applied on each bootstrapped sample. In Bagging
trees, predictions are decided by majority voting or by
averaging the various opinions available in each case.
This idea is known to reduce the basic instability of
trees (Breiman, 1996).
A distinctive feature of the Boosting approach is the
iterative calling to a basic weak learner (WL) algorithm
(Schapire & Singer, 1998). Each time the WL is in-
voked, it takes as input the training set together with
a dynamic (probability) weight distribution over the data
and returns a single tree. The output of the algorithm
is a weighted sum itself, where the weights are propor-
tional to individual performance error. The WL learning
algorithm needs only to produce moderately successful
models. Thus, trees and simplified trees (stumps) con-
stitute a popular choice. Several weight updating schemes
have been proposed. Schapire and Singer update weights
according to the success of the last model incorporated,
whereas in their LogitBoost algorithm, Friedman, Hastie,
and Tibshirani (2000) let the weights depend on overall
probabilistic estimates. This latter idea better reflects
the joint work of all classifiers available so far and
hence should provide a more effective guide for the WL
in general.
The notion of abstention brings a connection with
the CS approach that will be apparent as I discuss the
match set idea in the followins sections. In standard
boosting trees, each tree contributes a leaf to the overall
prediction for any new x input data vector, so the number
of expressing rules is the number of boosting rounds
independently of x. In the system proposed by Cohen
and Singer (1999), the WL essentially produces rules or
single leaves C (rather than whole trees). Their classi-
488
Evolutionary Mining of Rule Ensembles
fiers are then maps taking only two values, a real number
for those x verifying the leaf and 0 elsewhere. The final
boosting aggregation for x is thus unaffected by all
abstaining rules (with x C), so the number of expressing
rules may be a small fraction of the total number of
rules.
The General CS-Based Evolutionary
Approach
The general classifier system (CS) architecture invented
by John Holland constitutes perhaps one of the most
sophisticated classes of evolutionary computation al-
gorithms (Hollandet et al., 1986). Originally conceived as
a model for cognitive tasks, it has been considered in
many (simplified) forms to address a number of learn-
ing problems. The nowadays standard stimulus-response
(or single-step) CS architecture provides a fascinating
approach to the representation issue. Straightforward
rules (classifiers) constitute the CS building blocks. CS
algorithms maintain a population of such predictive
rules whose conditions are hyperplanes involving the
wild-card character #. If we generalize the idea of hyper-
plane to mean conjunctions of conditions on predic-
tors where each condition involves a single predictor,
We see that these rules are also used by many other
learning algorithms. Undoubtedly, hyperplane interpret-
ability is a major factor behind this popularity.
Critical subsystems in CS algorithms are the perfor-
mance, credit-apportionment, and rule discovery mod-
ules (Eiben & Smith, 2003). As regards credit-appor-
tionment, the question has been recently raised about
the suitability of endogenous reward schemes, where
endogenous refers to the overall context in which clas-
sifiers act, versus other schemes based on intrinsic
value measures (Booker, 2000). A well-known family
of algorithms is XCS (and descendants), some of which
have been previously advocated as data-mining tools
(see, e.g., Wilson, 2001). The complexity of the CS
dynamics has been analyzed in detail in Westerdale
(2001).
The match set M=M(x) is the subset of matched
(concurrently activated) rules, that is, the collection of
all classifiers whose condition is verified by the input
data vector x. The (point) prediction for a new x will be
based exclusively on the information contained in this
ensemble M.
A System Based on Support and
Predictive Scoring
Support is a familiar notion in various data-mining sce-
narios. There is a general trade-off between support and
TEAM LinG
Evolutionary Mining of Rule Ensembles

predictive accuracy: the larger the support, the lower the
accuracy. The importance of explicitly bounding support
(or deliberately seeking high support) has been recog-
nized often in the literature (Greene & Smith, 1994; Fried-
man & Fisher, 1999; Muselli & Liberati, 2002). Because the
world of generality intended by using only high-support
rules introduces increased levels of uncertainty and error,
statistical tools would seem indispensable for its proper
modeling. To this end, classifiers in the BYPASS algorithm
(Muruzbal, 2001) differ from other CS alternatives in that
they enjoy (support-minded) probabilistic predictions (thus
extending the more common single-label predictions).
Support plays an outstanding role: A minimum support
level b is input by the analyst at the outset, and consider-
ation is restricted to rules with (estimated) support above
b. The underlying predictive distributions, R, are easily
constructed and coherently updated following a standard
Bayesian Multinomial-Dirichlet process, whereas their toll
on the system is minimal memory-wise. Actual predictions
are built by first averaging the matched predictive distri-
butions and then picking the maximum a posteriori label of
the result. Hence, by promoting mixtures of probability
distributions, the BYPASS algorithm connects readily
with mainstream ensemble-learning methods.
We can find sometimes perfect regularities, that is,
subsets C for which the conditional distribution of the
response Y (given X C) equals 1 for some output class:
P(Y=j | X C)=1 for some j and 0 elsewhere. In the well-
known multiplexer environment, for example, there ex-
ists a set of such classifiers such that 100% performance
can be achieved. But in real situations, it will be difficult
to locate strictly neat C unless its support is quite small.
Moreover, putting too much emphasis on error-free
behavior may increase the risk of overfitting; that is, we
may infer rules that do not apply (or generalize poorly)
over a test sample. When restricting the search to high-
support rules, probability distributions are well equipped
to represent high-uncertainty patterns. When the largest
P(Y=j | X C) is small, it may be especially important to
estimate P(Y=j | X C) for all j.
Furthermore, the use of probabilistic predictions
R(j) for j=1, ..., k, where k is the number of output classes, -
makes possible a natural ranking of the M=M(x) as-
signed probabilities Ri(y) for the true class y related to
the current x. Rules i with large Ri(y) (scoring high) are
generally preferred in each niche. In fact, only a few rules
are rewarded at each step, so rules compete with each
other for the limited amount of resources. Persistent lack
of reward means extinction to survive, classifiers must
get reward from time to time. Note that newly discovered,
more effective rules with even better scores may cut
dramatically the reward given previously to other rules
in certain niches. The fitness landscape is thus highly
dynamic, and lower scores pi(y) may get reward and
survive provided they are the best so far at some niche.
An intrinsic measure of fitness for a classifier C R (such
as the lifetime average score -log R(Y), where Y is condi-
tioned by X C) could hardly play the same role.
It is worth noting that BYPASS integrates three learn-
ing modes in its classifiers: Bayesian at the data-process-
ing level, reinforcement at the survival (competition)
level, and genetic at the rule-discovery (exploration)
level. Standard genetic algorithms (GAs) are triggered by
system failure and act always circumscribed to M. Be-
cause the Bayesian updating guarantees that in the long
run, predictive distributions R reflect the true conditional
probabilities P(Y=j | X C), scores become highly reli-
able to form the basis of learning engines such as reward
or crossover selection. The BYPASS algorithm is sketched
in Table 1. Note that utility reflects accumulated reward
(Muruzbal, 2001). Because only matched rules get re-
ward, high support is a necessary (but not sufficient)
condition to have high utility. Conversely, low-uncer-
tainty regularities need to comply with the (induced)
bound on support. The background generalization rate
P# (controlling the number of #s in the random C built
along the run) is omitted for clarity, although some
tuning of P# with regard to threshold u is often required
in practice.

Table 1. The skeleton BYPASS algorithm

Input parameters (h: reward intensity) (u: minimum utility)

Initialize classifier population, say P
Iterate for a fixed number of cycles:
o Sample training item (x,y) from file
o Build match set M=M(x) including all rules with x C
o Build point prediction y*
o Check for success y* = y
If successful, reward the h 2 highest scores Ri(y) in M
If unsuccessful, add a new rule to P via a GA or otherwise
o Update predictive distributions R (in M)
o Update utility counters (in P)
o Check for low utility via threshold u and possibly delete some rules

489

TEAM LinG

BYPASS has been tested on various tasks under
demanding b (and not very large h), and the results have
been satisfactory in general. Comparative smaller popu-
lations are used, and low uncertainty but high support
rules are uncovered. Very high values for P# (as high as
0.975) have been tested successfully in some cases
(Muruzbal, 2001). In the juxtaposed (or concatenated)
multiplexer environment, BYPASS is shown to maintain a
compact population of relatively high uncertainty rules
that solves the problem by bringing about appropriate
match sets (nearly) all the time. Recent work by Butz,
Goldberg, and Tharakunnel (2003) shows that XCS also
solves this problem, although working at a lower level of
support (generality).
To summarize, BYPASS does not relay on rule plural-
ity for knowledge encoding because it uses compact
probabilistic predictions (bounded by support). It re-
quires no intrinsic value for rules and no added tailor-
made heuristics. Besides, it tends to keep population size
under control (with increased processing speed and
memory savings). The ensembles (match sets) derived
from evolution in BYPASS have shown good promise of
cooperation.
FUTURE TRENDS
Quick interactive data-mining algorithms and protocols
are nice when human judgment is available. When not,
computer-intensive, autonomous algorithms capable of
thoroughly squeezing the data are also nice for prelimi-
nary exploration and other purposes. In a sense, we
should tend to rely on the latter to mitigate the nearly
ubiquitous data overflow problem. Representation
schemes and learning engines are crucial to the success
of these unmanned agents and need, of course, further
investigation. Ensemble methods have lots of appealing
features and will be subject to further analysis and
testing. Evolutionary algorithms will continue to uprise
and succeed in yet some other application areas. Addi-
tional commercial spin-offs will keep coming. Although
great progress has been made in identifying many key
insights in the CS framework, some central points still
need further discussion. Specifically, the idea of rules
that perform its prediction following some kind of more
elaborate computation is appealing, and indeed more
functional representations of classifiers (such as multi-
layer perceptrons) have been proposed in the CS litera-
ture (see, e.g., Bull, 2002). On the theoretical side, a
formal framework for more rigorous analysis in high-
support learning is much needed. The task is not easy,
however, because the target is somewhat more vague,
and individual as well as collective interests should be
brought to terms when evaluating the generality and
490
Evolutionary Mining of Rule Ensembles
uncertainty associated to rules. Also, further research
should be conducted to clearly delineate the strengths
of the various CS approaches against current alternative
methods for rule ensemble formation and data mining.
CONCLUSION
Evolutionary rule mining is a successful, promising
research area. Evolutionary algorithms constitute by
now a very useful and wide class of stochastic optimiza-
tion methods. The evolutionary CS approach is likely to
provide interesting insights and cross-fertilization of
ideas with other data-mining methods. The BYPASS
algorithm discussed in this article has been shown to
tolerate the high support constraint well, leading to
pleasant and unexpected results in some problems. These
results stress the latent predictive power of the en-
sembles formed by high uncertainty rules.
REFERENCES
Booker, L. B. (2000). Do we really need to estimate rule
utilities in classifier systems? Lecture Notes in Artifi-
cial Intelligence, 1813, 125-142.
Breiman, L. (1996). Bagging predictors. Machine Learn-
ing, 24, 123-140.
Bull, L. (2002). On using constructivism in neural classifier
systems. Lecture Notes in Computer Science, 2439, 558-
567.
Butz, M. V., Goldberg, D. E., & Tharakunnel, K. (2003).
Analysis and improvement of fitness exploitation in
XCS: Bounding models, tournament selection, and bi-
lateral accuracy. Evolutionary Computation, 11(3), 239-
277.
Cohen, W. W., & Singer, Y. (1999). A simple, fast, and
effective rule learner. Proceedings of the 16th Na-
tional Conference on Artificial Intelligence,.
Eiben, A. E., & Smith, J. E. (2003). Introduction to
evolutionary computing. Springer.
Folino, G., Pizzuti, C., & Spezzano, G. (2003). En-
semble techniques for parallel genetic programming
based classifiers. Lecture Notes in Computer Science,
2610, 59-69.
Friedman, J. H., & Fisher, N. (1999). Bump hunting in high-
dimensional data. Statistics and Computing, 9(2), 1-20.
TEAM LinG
Evolutionary Mining of Rule Ensembles
Friedman, J. H., Hastie, T., & Tibshirani, R. (2000). Addi-
tive logistic regression: A statistical view of boosting.
Annals of Statistics, 28(2), 337-407.
Greene, D. P., & Smith, S. F. (1994). Using coverage as a
model building constraint in learning classifier systems.
Evolutionary Computation, 2(1), 67-91.
Hand, D. J. (1997). Construction and assessment of clas-
sification rules. Wiley.
Hand, D. J., Adams, N. M., & Kelly, M. G. (2001). Multiple
classifier systems based on interpretable linear classifi-
ers. Lecture Notes in Computer Science, 2096, 136-147.
Holland, J. H., Holyoak, K. J., Nisbett, R. E., & Thagard, P.
R. (1986). Induction: Processes of inference, learning
and discovery. MIT Press.
Koza, J. R., Keane, M. A., Streeter, M. J., Mydlowec, W.,
Yu, J., & Lanza, G. (Eds.). (2003). Genetic programming
IV: Routine human-competitive machine intelligence.
Kluwer.
Kuncheva, L. I., & Jain, L. C. (2000). Designing classifier
fusion systems by genetic algorithms. IEEE Transactions
on Evolutionary Computation, 4(4), 327-336.
Liu, Y., Yao, X., & Higuchi, T. (2000). Evolutionary en-
sembles with negative correlation learning. IEEE Trans-
actions on Evolutionary Computation, 4(4), 380-387.
Muruzbal, J. (2001). Combining statistical and reinforce-
ment learning in rule-based classification. Computational
Statistics, 16(3), 341-359.
Muselli, M., & Liberati, D. (2002). Binary rule generation
via hamming clustering. IEEE Transactions on Knowl-
edge and Data Engineering, 14, 1258-1268.
Schapire, R. E., & Singer, Y. (1998). Improved boosting
algorithms using confidence-rated predictions. Machine
Learning, 37(3), 297-336.
Westerdale, T. H. (2001). Local reinforcement and recom-
bination in classifier systems. Evolutionary Computa-
tion, 9(3), 259-281.
Wilson, S. W. (2001). Mining oblique data with XCS.
Lecture Notes in Artificial Intelligence, 1996, 158-176. -
KEY TERMS
Classification: The central problem in (supervised)
data mining. Given a training data set, classification
algorithms provide predictions for new data based on
predictive rules and other types of models.
Classifier System: A rich class of evolutionary com-
putation algorithms building on the idea of evolving a
population of predictive (or behavioral) rules under the
enforcement of certain competition and cooperation pro-
cesses. Note that classifier systems can also be under-
stood as systems capable of performing classification.
Not all CSs in the sense meant here qualify as classifier
systems in the broader sense, but a variety of CS algo-
rithms concerned with classification do.
Ensemble-Based Methods: A general technique
that seeks to profit from the fact that multiple rule
generation followed by prediction averaging reduces
test error.
Evolutionary Computation: The solution approach
guided by artificial evolution, which begins with random
populations (of solution models), then iteratively ap-
plies algorithms of various kinds to find the best or
fittest models.
Fitness Landscape: Optimization space due to the
characteristics of the fitness measure used to define the
evolutionary computation process.
Predictive Rules: Standard if-then rules with the
consequent expressing some form of prediction about
the output variable.
Rule Mining: A computer-intensive task whereby
data sets are extensively probed for useful predictive
rules.
Test Error: Learning systems should be evaluated
with regard to their true error rate, which in practice is
approximated by the error rate on test data, or test error.
491
TEAM LinG
492
Explanation-Oriented Data Mining
Yiyu Yao
University of Regina, Canada
Yan Zhao
University of Regina, Canada
INTRODUCTION
Data mining concerns theories, methodologies, and, in
particular, computer systems for knowledge extraction
or mining from large amounts of data (Han & Kamber,
2000). The extensive studies on data mining have led to
many theories, methodologies, efficient algorithms and
tools for the discovery of different kinds of knowledge
from different types of data. In spite of their differ-
ences, they share the same goal, namely, to discover new
and useful knowledge, in order to gain a better under-
standing of nature.
The objective of data mining is, in fact, the goal of
scientists when carrying out scientific research, inde-
pendent of their various disciplines. Data mining, by
combining research methods and computer technology,
should be considered as a research support system. This
goal-oriented view enables us to re-examine data min-
ing in the wider context of scientific research. Such a
re-examination leads to new insights into data mining
and knowledge discovery.
The result, after an immediate comparison between
scientific research and data mining, is that an explana-
tion construction and evaluation task is added to the
existing data mining framework. In this chapter, we
elaborate upon the basic concerns and methods of ex-
planation construction and evaluation. Explanation-ori-
ented association mining is employed as a concrete
example to demonstrate the entire framework.
BACKGROUND
Scientific research and data mining have much in com-
mon in terms of their goals, tasks, processes and meth-
odologies. As a recently emerged area of multi-disci-
plinary study, data mining and knowledge discovery
research can benefit from the long established studies
of scientific research and investigation (Martella,
Nelson, & Marchand-Martella, 1999). By viewing data
mining in a wider context of scientific research, we can
obtain insights into the necessities and benefits of
explanation construction. The model of explanation-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
oriented data mining is a recent result from such an
investigation (Yao, 2003; Yao, Zhao, & Maguire, 2003).
Common Goals of Scientific Research
and Data Mining
Scientific research is affected by the perceptions and
the purposes of science. Martella et al. summarized the
main purposes of science, namely, to describe and pre-
dict, to improve or manipulate the world around us, and
to explain our world (Martella, et al., 1999). The results
of the scientific research process provide a description
of an event or a phenomenon. The knowledge obtained
from this research helps us to make predictions about
what will happen in the future. Research findings are a
useful tool for making an improvement in the subject
matter. Research findings also can be used to determine
the best or the most effective ways of bringing about
desirable changes. Finally, scientists develop models
and theories to explain why a phenomenon occurs.
Goals similar to those of scientific research have
been discussed by many researchers in data mining. For
example, Fayyad, Piatetsky-Shapiro, Smyth and
Uthurusamy identified two high-level goals of data min-
ing as prediction and description (Fayyad, et al., 1996).
Prediction involves the use of some variables to predict
the values of some other variables, while description
focuses on patterns that describe the data. Ling, Chen,
Yang and Cheng studied the issue of manipulation and
action based on discovered knowledge (Ling et al.,
2002). Yao, Zhao, et al. introduced the notion of
explanation-oriented data mining, which focuses on
constructing models for the explanation of data mining
results (2003).
Common Processes of Scientific
Research and Data Mining
Research is a highly complex and subtle human activity,
which is difficult to formally define. It seems impos-
sible to give any formal instruction on how to do re-
search. On the other hand, some lessons and general
principles can be learnt from the experience of scien-
TEAM LinG
Explanation-Oriented Data Mining

Table 1. The model of scientific research processes

Idea-generation phase: to identify a topic of interest.
Problem-definition phase: to precisely and clearly define and formulate vague and general
-
ideas generated in the previous phase.
Procedure-design/planning phase: to make a workable research plan by considering all issues
involved.
Observation/experimentation phase: to observe real world phenomenon, collect data, and carry
out experiments.
Data-analysis phase: to make sense out of the data collected.
Results-interpretation phase: to build rational models and theories that explain the results from
the data-analysis phase.
Communication phase: to present the research results to the research community.

Table 2. The model of data mining processes

Data pre-processing phase: to select and clean working data.
Data transformation phase: to change the working data into the required form.
Pattern discovery and evaluation phase: to apply algorithms to identify knowledge embedded
in data, and to evaluate the discovered knowledge.
Explanation construction and evaluation phase: to construct plausible explanations for
discovered knowledge, and to evaluate different explanations.
Pattern presentation: to present the extracted knowledge and explanations.

tists. There are some basic principles and techniques
that are commonly used in most types of scientific
investigations. We adopt the model of the research
process from Garziano and Raulin (2000), and combine
it with other models (Martella, et al., 1999). The basic
phases and their objectives are summarized in Table 1.
It is possible to combine several phases into one, or to
divide one phase into more detailed steps. The division
between phases is not clear-cut. The research process
does not follow a rigid sequencing of the phases. Itera-
tion of different phrases may be necessary (Graziano &
Raulin, 2000).
Many researchers have proposed and studied models
of data mining processes (Fayyad, et al. 1996; Mannila,
1997; Yao, Zhao, et al., 2003; Zhong, Liu, & Ohsuga,
2001). A model that adds the explanation facility to the
commonly used models has been recently proposed by
Yao, Zhao, et al.; it is remarkably similar to the model of
scientific research. The basic phases and their objec-
tives are summarized in Table 2. Like the research
process, the data mining process is also an iterative
process and there is no clear-cut difference among the
There are bi-directional benefits. The experiences and
results from the studies of research methods can be
applied to data mining problems; the data mining algo-
rithms can be used to support scientific research.
MAIN THRUST
Explanations of data mining address several important
questions. What needs to be explained? How to explain
the discovered knowledge? Moreover, is an explanation
correct and complete? By answering these questions,
one can better understand explanation-oriented data
mining. The ideas and processes of explanation con-
struction and explanation evaluation are demonstrated
by explanation-oriented association mining.
Figure 1. A framework of explanation-oriented data
mining
Pattern discovery & evaluation

different phases. In fact, Zhong, et al. argue that it should Data transformation Explanation construction
be a dynamically organized process (Zhong, et al., 2001). Data preprocessing
& evaluation

The entire framework is illustrated in Figure 1. Data Target Transformed Patterns

Pattern representation
There is a parallel correspondence between the data data

processes of scientific research and data mining. Their

main difference lies in the subjects that perform the
tasks. Research is carried out by scientists, and data
mining is done by computer systems. In particular, data
mining may be viewed as a study of domain-independent Explained
patterns
Knowledge

research methods with emphasis on data analysis. The

higher and more abstract level of comparisons and con- Background Selected
attributes
Explanation
profiles
nections between scientific research and data mining Attribute selection
can be further studied in levels that are more concrete. Profile construction

493

TEAM LinG

Basic Issues
Explanation-oriented data mining explains and in-
terprets the knowledge discovered from data.
Knowledge can be discovered by unsupervised learn-
ing methods. Unsupervised learning studies how sys-
tems can learn to represent, summarize, and organize the
data in a way that reflects the internal structure (namely,
a pattern) of the overall collection. This process does
not explain the patterns, but describes them. The primary
unsupervised techniques include clustering mining, be-
lief (usually Bayesian) networks learning, and associa-
tion mining. The criteria for choosing which pattern to be
explained are directly related to the pattern evaluation
step of data mining.
Explanation-oriented data mining requires back-
ground knowledge to infer features that can possi-
bly explain a discovered pattern.
Understanding the theory of explanation requires
considerations from many branches of inquiry: physics,
chemistry, meteorology, human culture, logic, psychol-
ogy, and, above all, the methodology of science. In data
mining, explanation can be made at a shallow, syntactic
level based on statistical information, or at a deep, se-
mantic level based on domain knowledge. The required
information and knowledge for explanation may not nec-
essarily be inside the original dataset. Additional infor-
mation often is required for an explanation construction.
It is argued that the power of explanation involves the
power of insight and anticipation. One collects certain
features based on the underlying hypothesis that these
may provide explanations of the discovered pattern. That
something is unexplainable may simply be an expression
of the inability to discover an explanation of a desired
sort. The process of selecting the relevant and explana-
tory features may be subjective, and trial-and-error. In
general, the better the background knowledge is, the
more accurate the inferred explanations are likely to be.
Explanation-oriented data mining utilizes induc-
tion, namely, drawing an inference from a set of
acquired training instances, and justifying or pre-
dicting the instances one might observe in the
future.
Supervised learning methods can be applied to this
explanation construction. The goal of supervised learn-
ing is to find a model that will correctly associate the
input patterns with the classes. In real world applica-
tions, supervised learning models are extremely useful
analytic techniques. The widely used supervised learning
494
Explanation-Oriented Data Mining
methods include decision tree learning, rule-based learn-
ing, and decision graph learning. The learned results are
represented as either a tree, or a set of if-then rules.
The constructed explanations provide some evi-
dence about conditions (within the background
knowledge) under which the discovered pattern is
most likely to happen, or how the background
knowledge is related to the pattern.
The role of explanation in data mining is linked to
proper description, relation and causality. Comprehen-
sibility is the key factor. The accuracy of the con-
structed explanations relies on the amount of training
examples. Explanation-oriented data mining performs
poorly with insufficient data or poor presuppositions.
Different background knowledge may infer different
explanations. There is no reason to believe that only
one unique explanation exists. One can use statistical
measures and domain knowledge to evaluate different
explanations.
A Concrete Example: Explanation-
oriented Association Mining
Explanations, also expressed as conditions, can pro-
vide additional semantics to a standard association. For
example, by adding time, place, and/or customer pro-
files as conditions, one can identify when, where, and/
or to whom an association occurs.
Explanation Construction
The approach of explanation-oriented data mining com-
bines unsupervised and supervised learning methods,
namely, forming a concept first, and then explaining the
concept. It consists of two main steps and uses two data
tables. One table is used to learn a pattern. The other
table, an explanation profile constructed with respect
to an interesting pattern, is used to search for explana-
tions of the pattern. In the first step, an unsupervised
learning algorithm, like the Apriori algorithm (Agrawal,
Imielinski, & Swami, 1993), can be applied to discover
frequent associations. To discover other types of
associations, different algorithms can be applied. In
the second step, an explanation profile is constructed.
Objects in the profile are labelled as positive instances if
they satisfy the desired pattern, and negative instances,
if they do not. Conditions that explain the pattern are
searched by using a supervised learning algorithm. A
classical supervised learning algorithm such as ID3
(Quinlan, 1983), C4.5 (Quinlan, 1993), or PRISM
(Cendrowska, 1987), may be used to construct explana-
tions.
TEAM LinG
Explanation-Oriented Data Mining

Table 3. The Apriori-ID3 algorithm

Input: A transaction table, and a related explanation profile table. -
Output: Associations and explained associations.
1. Use the Apriori algorithm to generate a set of frequent associations in the transaction table. For each
association in the set, support ( ) minsup , and confidence( ) minconf .
2. If the association is considered interesting (with respect to the user feedback or interestingness
measures), then
a. Introduce a binary attribute named Decision. Given a transaction, its value on Decision is + if
it satisfies in the original transaction table, otherwise its value is -.
b. Construct an information table by using the attribute Decision and explanation profiles. The new
table is called an explanation table.
c. By treating Decision as the target class, one can apply the ID3 algorithm to derive classification
rules of the form ( Decision = "+ ") . The condition is a formula discovered in the
explanation table, which states that under the association occurs.
d. Evaluate the constructed explanation(s).

The Apriori-ID3 algorithm, which can be regarded as
an example of explanation-oriented association mining
method, is described in Table 3.
Explanation Evaluation
Once explanations are generated, it is necessary to
evaluate them. For explanation-oriented association
mining, we want to compare a conditional association
(explained association) with its unconditional counter-
part, in addition to comparing different conditions.
Let T be a transaction table, and E be an explanation
profile table associated with T. Suppose that for a
desired pattern generated by an unsupervised learning
algorithm from T, there is a set K of conditions (expla-
nations) discovered by a supervised learning algorithm
from E, and K is one explanation. Two points are
noted. First, the set K of explanations can be different
according to various explanation profile tables, or vari-
ous supervised learning algorithms. Second, not all
explanations in K are equally interesting. Different con-
ditions may have different degrees of interestingness.
Suppose is a quantitative measure used to evaluate
plausible explanations, which can be the support mea-
sure for an undirected association, the confidence or
coverage measure for a one-way association, or the
similarity measure for a two-way association (Yao &
Zhong, 1999). A condition K provides an explana-
tion of a discovered pattern if ( | ) > ( ) . One can
further evaluate explanations quantitatively based on
several measures, such as absolute difference (AD),
relative difference (RD) and ratio of change (RC):
AD( | ) = ( | ) ( ),
( | ) ( )
RD( | ) = ,
( )
( | ) ( )
RC ( | ) = .
1 ( )
The absolute difference represents the disparity be-
tween the pattern and the pattern under the condition.
For a positive value, one may say that the condition
supports ; for a negative value, one may say that the
condition rejects . The relative difference is the ratio
of absolute difference to the value of the unconditional
pattern. The ratio of change compares the actual change
and the maximum potential change.
Generality is the measure to quantify the size of a
condition with respect to the whole data, defined by
||
generality ( ) = . When the generality of conditions
|U |
is essential, a compound measure should be applied. For
example, one may be interested in discovering an accu-
rate explanation with a high ratio of change and a high
generality. However, it often happens that an explana-
tion has a high generality but a low RC value, while
another explanation has a low generality but a high RC
value. A trade-off between these two explanations does
not necessarily exist.
A good explanation system must be able to rank the
constructed explanations and be able to reject the bad
explanations. It should be realized that evaluation is a
difficult process because so many different kinds of knowl-
edge can come into play. In many cases, one must rely on
domain experts to reject uninteresting explanations.
FUTURE TRENDS
Considerable research remains to be done for explana-
tion construction and evaluation.
In this chapter, rule-based explanation is con-
structed by inductive supervised learning algorithms.
Considering the structure of explanation, case-based
explanations also need to be addressed. Based on the
case-based explanation, a pattern is explained if an
actual prior case is presented to provide compelling

495

TEAM LinG

support. One of the perceived benefits of case-based
explanation is that the rule generation effort is saved.
Instead, similarity functions need to be studied in order
to evaluate the distance between the description of the
new pattern and an existing case, and retrieve the most
similar case as an explanation.
The constructed explanations of the discovered pat-
tern provide conclusive evidence for the new instances.
In other words, the new instances can be explained and
implied by the explanations. This is normally true when
the explanations are sound and complete. However,
sometimes, the constructed explanations cannot guar-
antee that a certain instance is a perfect fit. Even worse,
a new data set, as a whole, may show a change or a
confliction with the learnt explanations. This is because
the explanations may be context-dependent on certain
spatial and/or temporal intervals. To consolidate the
explanations we have constructed, we cannot simply
logically and, or, or ignore the new explanation. Mining optimal actions for profitable CRM. Proceed-
Instead, a spatial-temporal reasoning model needs to be
introduced to show the trend and evolution of the pattern
to be explained.
The explanations we have introduced so far are not
necessarily the causal interpretation of the discovered
pattern, i.e. the relationships expressed in the form of
deterministic and functional equations. They can be
inductive generalizations, descriptions, or deductive
implications. Explanation as causality is the strongest
explanation and coherence.
We might think of Bayesian networks as an inference
that unveils the internal relationship between attributes.
Searching for an optimal model is difficult and NP-hard.
Arrow direction is not guaranteed. Expert knowledge
could be integrated in the a priori search function, such
as the presence of links and orders.
CONCLUSION
Explanation-oriented data mining offers a new perspec-
tive. It closely relates scientific research and data min-
ing, which have bi-directional benefits. The ideas of
explanation-oriented mining can have a significant im-
pact on the understanding of data mining and effective
applications of data mining results.
REFERENCES
Agrawal, R., Imielinski, T. & Swami, A. (1993). Mining
association rules between sets of items in large data-
bases. Proceedings of ACM Special Interest Group on
Management of Data 1993 (pp. 207-216).
496
Explanation-Oriented Data Mining
Brodie, M. & Dejong, G. (2001). Iterated phantom
induction: A knowledge-based approach to learning con-
trol. Machine Learning, 45(1), 45-76.
Cendrowska, J. (1987). PRISM: An algorithm for induc-
ing modular rules. International Journal of Man-Ma-
chine Studies, 27, 349-370.
Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P. &
Uthurusamy, R. (Eds.). (1996). Advances in Knowl-
edge Discovery and Data Mining. AAAI/MIT Press.
Graziano, A.M. & Raulin, M.L. (2000). Research meth-
ods: A process of inquiry (4th ed.). Boston: Allyn &
Bacon.
Han, J. & Kamber, M. (2000). Data mining: Concept
and techniques. Morgan Kaufmann Publisher.
Ling, C.X., Chen, T., Yang, Q. & Cheng, J. (2002).
ings of International Conference on Data Mining (pp.
767-770).
Mannila, H. (1997). Methods and problems in data mining.
Proceedings of International Conference on Database
Theory (pp. 41-55).
Martella, R.C., Nelson, R. & Marchand-Martella, N.E.
(1999). Research methods: Learning to become a
critical research consumer. Boston: Allyn & Bacon.
Mitchell, T. (1999). Machine learning and data mining.
Communications of the ACM, 42(11), 30-36.
Quinlan, J.R. (1983). Learning efficient classification
procedures. In J.S. Michalski, J.G. Carbonell & T.M.
Mirchell (Eds.), Machine learning: An artificial intel-
ligence approach (pp. 463-482). Palo Alto, CA: Mor-
gan Kaufmann.
Quinlan, J.R. (1993). C4.5: Programs for Machine
learning. Morgan Kaufmann Publisher.
Yao, Y.Y. (2003). A framework for web-based research
support systems. Proceedings of the Twenty-Seventh
Annual International Computer Software and Appli-
cations Conference (pp. 601-606).
Yao, Y.Y., Zhao, Y. & Maguire, R.B. (2003). Explana-
tion-oriented association mining using rough set theory.
Proceedings of Rough Sets, Fuzzy Sets and Granular
Computing (pp. 165-172).
Yao, Y.Y. & Zhong, N. (1999). An analysis of quantita-
tive measures associated with rules. Proceedings Pa-
cific-Asia Conference on Knowledge Discovery and
Data Mining (pp. 479-488).
TEAM LinG
Explanation-Oriented Data Mining
Zhong, N., Liu, C. & Ohsuga, S. (2001). Dynamically
organizing KDD processes. International Journal of
Pattern Recognition and Artificial Intelligence, 15, 451-
473.
KEY TERMS
Absolute Difference: A measure that represents the
difference between an association and a conditional
association based on a given measure. The condition
provides a plausible explanation.
Explanation-Oriented Data Mining: A general
framework includes data pre-processing, data transfor-
mation, pattern discovery and evaluation, pattern expla-
nation and explanation evaluation, and pattern presenta-
tion. This framework is consistent with the general
model of scientific research processes.
Generality: A measure that quantifies the coverage
of an explanation in the whole data set.
Goals of Scientific Research: The purposes of
science are to describe and predict, to improve or to
manipulate the world around us, and to explain our
world. One goal of scientific research is to discover
new and useful knowledge for the purpose of science.
As a specific research field, data mining shares this
common goal, and may be considered as a research
support system.
Method of Explanation-Oriented Data Mining: The
method consists of two main steps and uses two data -
tables. One table is used to learn a pattern. The other table,
an explanation table, is used to explain one desired pat-
tern. In the first step, an unsupervised learning algorithm
is used to discover a pattern of interest. In the second
step, by treating objects satisfying the pattern as positive
instances, and treating the rest as negative instances, one
can search for conditions that explain the pattern by a
supervised learning algorithm.
Ratio of Change: A ratio of actual change (absolute
difference) to the maximum potential change.
Relative Difference: A measure that represents the
difference between an association and a conditional
association relative to the association based on a given
measure.
Scientific Research Processes: A general model
consists of the following phases: idea generation, prob-
lem definition, procedure design/planning, observation/
experimentation, data analysis, results interpretation,
and communication. It is possible to combine several
phases, or to divide one phase into more detailed steps.
The division between phases is not clear-cut. Iteration
of different phrases may be necessary.
497
TEAM LinG
498
Factor Analysis in Data Mining
Zu-Hsu Lee
Montclair State University, USA
Richard L. Peterson
Montclair State University, USA
Chen-Fu Chien
National Tsing Hua University, Taiwan
Ruben Xing
Montclair State University, USA
INTRODUCTION
The rapid growth and advances of information technol-
ogy enable data to be accumulated faster and in much
larger quantities (i.e., data warehousing). Faced with vast
new information resources, scientists, engineers, and
business people need efficient analytical techniques to
extract useful information and effectively uncover new,
valuable knowledge patterns.
Data preparation is the beginning activity of exploring
for potentially useful information. However, there may be
redundant dimensions (i.e., variables) in the data, even
after the data are well prepared. In this case, the perfor-
mance of data-mining methods will be affected negatively
by this redundancy. Factor Analysis (FA) is known to be
a commonly used method, among others, to reduce
data dimensions to a small number of substantial char-
acteristics.
FA is a statistical technique used to find an underlying
structure in a set of measured variables. FA proceeds with
finding new independent variables (factors) that describe
the patterns of relationships among original dependent
variables. With FA, a data miner can determine whether or
not some variables should be grouped as a distinguishing
factor, based on how these variables are related. Thus, the
number of factors will be smaller than the number of
original variables in the data, enhancing the performance
of the data-mining task. In addition, the factors may be
able to reveal underlying attributes that cannot be ob-
served or interpreted explicitly so that, in effect, a recon-
structed version of the data is created and used to make
hypothesized conclusions. In general, FA is used
with many data-mining methods (e.g., neural net-
work, clustering).
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
The concept of FA was created in 1904 by Charles
Spearman, a British psychologist. The term factor analy-
sis was first introduced by Thurston in 1931. Exploratory
FA and confirmatory FA are two main types of modern FA
techniques. The goals of FA are (1) to reduce the number
of variables and (2) to classify variables through detec-
tion of the structure of the relationships between vari-
ables. FA achieves the goals by creating a fewer number
of new dimensions (i.e., factors) with potentially useful
knowledge. The applications of FA techniques can be
found in various disciplines in science, engineering, and
social sciences, such as chemistry, sociology, econom-
ics, and psychology. To sum up, FA can be considered as
a broadly used statistical approach that explores the
interrelationships among variables and determines a
smaller set of common underlying factors. Furthermore,
the information contained in the original variables can be
explained by these factors with a minimum loss of infor-
mation.
MAIN THRUST
In order to represent the important structure of the data
efficiently (i.e., in a reduced number of dimensions), there
are a number of techniques that can be used for data
mining. These generally are referred to as multi-dimen-
sional scaling methods. The most basic one is Principle
Component Analysis (PCA). Through transforming the
original variables in the data into the same number of new
ones, which are mutually orthogonal (uncorrelated), PCA
sequentially extracts most of the variance (variability) of
TEAM LinG
Factor Analysis in Data Mining
the data. The hope is that most of the information in the
data might be contained in the first few components. FA
also extracts a reduced number of new factors from the
original data set, although it has different aims from PCA.
FA usually starts with a survey or a number of ob-
served traits. Before FA is applied, the assumptions of
correlations in the data (normality, linearity, homogeneity
of sample, and homoscedasticity) need to be satisfied. In
addition, the factors to extract should all be orthogonal
to one another. After defining the measured variables to
represent the data, FA considers these variables as a
linear combination of latent factors that cannot be mea-
sured explicitly. The objective of FA is to identify these
unobserved factors, reflect what the variables share in
common, and provide further information about them.
Mathematically, let X represent a column vector that
contains p measured variables and has a mean vector ,
F stand for a column vector which contains q latent
factors, and L be a p q matrix that transforms F to X.
The elements of L (i.e., factor loadings) give the weights
that each factor contributes to each measured variable. In
addition, let be a column vector containing p uncorrelated
random errors. Note that q is smaller than p. The following
equation simply illustrates the general model of FA
(Johnson & Wichern, 1998):
X - = LF + .
FA and PCA yield similar results in many cases, but,
in practice, PCA often is preferred for data reduction,
while FA is preferred to detect structure of the data.
In any experiment, any one scenario may be delineated
by a large number of factors. Identifying important factors
and putting them into more general categories generates
an environment or structure that is more advantageous to
data analysis, reducing the large number of variables to
smaller, more manageable, interpretable factors (Kachigan,
1986). Technically, FA allows the determination of the
interdependency and pattern delineation of data. It un-
tangles the linear relationships into their separate pat-
terns as each pattern will appear as a factor delineating a
distinct cluster of interrelated data (Rummel, 2002, Sec-
tion 2.1). In other words, FA attempts to take a group of
interdependent variables and create separate descriptive
categories and, after this transformation, thereby de-
crease the number of variables that are used in an experi-
Table 1. Measures of the adequacy of FA to the sample: MSA
E I ILK J M M_S
0.75 0.73 0.86 0.78 0.76 0.82
W WG
0.39 0.87
ment (Rummel, 2002). The analysis procedures can be
performed through a geometrical presentation by plotting .
data points in a multi-dimensional coordinate axis (explor-
atory FA) or through mathematical techniques to test the
specified model and suspected relationship among vari-
ables (confirmatory FA).
In order to illustrate how FA proceeds step by step,
here is an example from a case study on the key variables
(or characteristics) for induction machines, conducted by
Mat and Caldern (2000). The sample of a group of
motors was selected from a catalog published by Siemens
(1988). It consists of 134 cases with no missing values and
13 variables that are power (P), speed (W), efficiency (E),
power factor (PF), current (I), locked-rotor current (ILK),
torque (M), locked-rotor torque (MLK), breakdown torque
(MBD), inertia (J), weight (WG), slip (S), and slope of M-
s curve (M_S). FA can be implemented in the following
procedures using this sample data.
Step 1: Ensuring the Adequacy of the
Date
The correlation matrix containing correlations between
the variables is first examined to identify the variables that
are statistically significant. In the case study, this matrix
from the sample data showed that the correlations be-
tween the variables are satisfactory, and thus, all vari-
ables are kept for the next step. Meanwhile, preliminary
tests, such as the Bartlett test, the Kaiser-Meyer-Olkin
(KMO) test, and the Measures of Sampling Adequacy
(MSA) test, are used to evaluate the overall significance
of the correlation. Table 1 shows that the values of MSA
(rounded to two decimal places) are higher than 0.5 for all
variables but variable W. However, the MSA value of
variable W is close to 0.5 (MSA should be higher than 0.5,
according to Hair, et al. (1998)).
Step 2: Finding the Number of Factors
There are many approaches available for this purpose
(e.g., common factor analysis, parallel analysis). The case
study first employed the plot of eigenvalues vs. the factor
number (the number of factors may be 1 to 13) and found
that choosing three factors accounts for 91.3% of the total
variance. Then, it suggested that the solution be checked
MBD MLK P PF S
0.79 0.74 0.74 0.76 0.85
499
TEAM LinG

with the attempt to extract two or three more factors. Based
on the comparison between results from different selected
methods, the study ended up with five factors.
Step 3: Determining Transformation (or
Rotation) Matrix
A commonly used method is orthogonal rotation. Table 2
can represent the transformation matrix, if each cell shows
the factor loading of each variable on each factor. For the
sample size, only loadings with an absolute value bigger
than 0.5 were accepted (Hair et al., 1998), and they are
marked X in Table 2 (other loadings lower than 0.5 are not
listed). From the table, J, I, M, S, M, WG, and P can be
grouped to the first factor; PF, ILK, E, and S belong to the
second factor; W and MLK can be considered another two
single factors, respectively. Note that MBD can go to
factor 2 or be an independent factor (factor 5) of the other
four. The case study settled the need to retain it in the fifth
factor, based on the results obtained from other samples.
In this step, an oblique rotation is another method to
determine the transformation matrix. Since it is a non-
orthogonal rotation, the factors are not required to be
uncorrelated to each other. This gives better flexibility
than an orthogonal rotation. Using the sample data of the
case study, a new transformation matrix may be obtained
from an oblique rotation, which provides new loadings to
group variables into factors.
Step 4: Interpreting the Factors.
In the case study, the factor consisting of J, I, M, S, M, WG,
and P was named size, because the higher value of the
weight (WG) and power (P) reflects the larger size of the
machine. The factor containing PF, ILK, E, and S is ex-
plained as global efficiency.
This example provides a general demonstration of the
application of FA techniques to data analysis. Various
Table 2. Grouping of variables using the orthogonal rotation matrix
Factor 1 Factor 2
E X (0.93540)
I X (0.93314)
ILK X (0.84356)
J X (0.97035)
M X (0.94582)
M_S X (0.98255)
MBD X (0.62605)
MLK
P X (0.92265)
PF X (0.81847)
S X (-0.92848)
W X (0.95222)
WG X (0.94153)
500
Factor Analysis in Data Mining
methods may be incorporated within the concept of FA.
Proper selection of methods should depend on the na-
ture of data and the problem to which FA is applied.
Garson (2003) points out the abilities of FA: determin-
ing the number of different factors needed to explain the
pattern of relationships among the variables, describing
the nature of those factors, knowing how well the hy-
pothesized factors explain the observed data, and find-
ing the amount of purely random or unique variance that
each observed variable includes. Because of these abili-
ties, FA has been used for various data analysis prob-
lems and may be used in a variety of applications in data
mining from science-oriented to business applications.
One of the most important uses is to provide a sum-
mary of the data. The summary facilitates learning the
data structure via an economic description. For example,
Pan et al. (1997) employed Artificial Neural Network
(ANN) techniques combined with FA for spectroscopic
quantization of amino acids. Through FA, the number of
input nodes for neural networks was compressed effec-
tively, which greatly sped up the calculations of neural
networks. Tan and Wisner (2001) used FA to reduce a set
of factors affecting operations management constructs
and their relationships. Kiousis (2004) applied an explor-
atory FA to he New York Times news coverage of eight
major political issues during the 2000 presidential elec-
tion. FA identified two indices that measured the con-
struct of the key independent variable in agenda-setting
research, which could then be used in future investiga-
tions.
Screening variables is another important function of
FA. A co-linearity problem will appear, if the factors of
the variables in the data are very similar to each other. In
order to avoid this problem, a researcher can group
closely related variables into one category and then
extract the one that would have the greatest use in
determining a solution (Kachigan, 1986). For example,
Borovec (1996) proposed a six-step sequential extraction
Factor 3 Factor 4 Factor 5
X (0.67072)
X (0.91674)
TEAM LinG
Factor Analysis in Data Mining
procedure and applied FA, which found three dominant
trace elements from 12 surface stream sediments. These
three factors accounted for 78% of the total variance. In
another example, Chen, et al. (2001) performed an explor-
atory FA on 48 financial ratios from 63 firms. Four critical
financial ratios were concluded, which explained 80% of
the variation in productivity.
FA can be used as a scaling method, as well. Oftentimes,
after the data are collected, the development of scales is
needed among individuals, groups, or nations, when they
are intended to be compared and rated. As the character-
istics are grouped to independent factors, FA assigns
weights to each characteristic according to the observed
relationships among the characteristics. For instance,
Tafeit, et al. (1999, 2000) provided a comparison between
FA and ANN for low-dimensional classification of high-
dimensional body fat topography data of healthy and
diabetic subjects with a high-dimensional and partly
highly intercorrelated set of data. They found that the
analysis of the extracted weights yielded useful informa-
tion about the structure of the data. As the weights for
each characteristic are obtained by FA, the score (by
summing characteristics times these weights) can be used
to represent the scale of the factor to facilitate the rating
of factors.
In addition, FAs ability to divide closely related
variables into different groups is also useful for statistical
hypothesis testing, as Rummel (2002) stated, when hy-
potheses are about the dimensions that can be a group of
highly intercorrelated characteristics, such as personal-
ity, attitude, social behavior, and voting. For instance, in
a study of resource investments in tourism business,
Morais, et al. (2003) use confirmatory FA to find that pre-
established resource investment scales could not fit their
model well. They reexamined each subscale with explor-
atory FA to identify factors that should not have been
included in the original model.
There have been controversies about uses of FA.
Hand, et al. (2001) pointed out that one important reason
is that FAs solutions are not invariant to various trans-
formations. More precisely, the extracted factors are
basically non-unique, unless extra constraints are im-
posed (Hand et al., 2001, p. 84). The same information
may reach different interpretations with personal judg-
ment. Nevertheless, no method is perfect. In some situa-
tions, other statistical methods, such as regression analy-
sis and cluster analysis, may be more appropriate than FA.
However, FA is a well-known and useful tool among data-
mining techniques.
FUTURE TRENDS
.
At the Factor Analysis at 100 Conference held in May
2004, the future of FA was discussed. Millsap and Meredith
(2004) suggested further research in the area of ordinal
measures in multiple populations and technical issues of
small samples. These conditions can generate bias in
current FA methods, causing results to be suspect. They
also suggested further study in the impact of violations
of factorial invariance and explanations for these viola-
tions. Wall and Amemiya (2004) feel that there are chal-
lenges in the area of non-linear FA. Although models exist
for non-linear analysis, there are aspects of this area that
are not fully understood.
However, the flexibility of FA and its ability to reduce
the complexity of the data still make FA one of commonly
used techniques. Incorporated with advances in informa-
tion technologies, the future of FA shows great promise
for the applications in the area of data mining.
CONCLUSION
FA is a useful multivariate statistical technique that has
been applied in a wide range of disciplines. It enables
researchers to effectively extract information from huge
databases and attempts to organize and minimize the
amount of variables used in collecting or measuring data.
However, the applications of FA in business sectors (e.g.,
e-business) is relatively new.
Currently, the increasing volumes of data in databases
and data warehouses are the key issue governing their
future development. Allowing the effective mining of
potentially useful information from huge databases with
many dimensions, FA definitely is helpful in sorting out
the significant parts of information for decision makers, if
it is used appropriately.
REFERENCES
Borovec, Z. (1996). Evaluation of the concentrations of
trace elements in stream sediments by factor and cluster
analysis and the sequential extraction procedure. The
Science of the Total Environment, 117, 237-250.
Chen, L., Liaw, S., & Chen, Y. (2001). Using financial
factors to investigate productivity: An empirical study in
Taiwan. Industrial Management & Data Systems, 101(7),
378-384.
501
TEAM LinG

Garson, D. (2003). Factor analysis. Retrieved from http:/
/www2.chass.ncsu.edu/garson/pa765/factor.htm
Hair, J., Anderson, R., Tatham R., & Black, W. (1998).
Multivariate data analysis with readings. Englewood
Cliffs, NJ: Prentice-Hall, Inc.
Hand, D., Mannila, H., & Smyth, P. (2001). Principles of
data mining. Cambridge, MA. MIT Press.
Johnson, R., & Wichern, D. (1998). Applied multivariate
statistical analysis. Englewood Cliffs, NJ: Prentice Hall,
Inc.
Kachigan, S. (1986). Statistical analysis: An interdisci-
plinary introduction to univariate and multivariate
methods. New York: Radius Press.
Kiousis, S. (2004). Explicating media salience: A factor
analysis of New York Times issue coverage during the
2000 U.S. presidential election. Journal of Communica-
tion, 54(1), 71-87.
Mate, C., & Calderon, R. (2000). Exploring the character-
istics of rotating electric machines with factor analysis.
Journal of Applied Statistics, 27(8), 991-1006.
Millsap, R., & Meredith, W. (2004). Factor invariance:
Historical trends and new developments. Proceedings of
the Factor Analysis at 100: Historical Developments
and Future Directions Conference, Chapel Hill, North
Carolina.
Morais, D., Backman, S., & Dorsch, M. (2003). Toward the
operationalization of resource investments made between
customers and providers of a tourism service. Journal of
Travel Research, 41, 362-374.
Pan, Z. et al. (1997). Spectroscopic quantization of amino
acids by using artificial neural networks combined with
factor analysis. Spectrochimica Acta Part A 53, 1629-
1632.
Rummel, R.J. (2002). Understanding factor analysis. Re-
trieved from http://www.hawaii.edu/powerkills/
UFA.HTM
Tafeit, E., Moller, R., Sudi, K., & Reibnegger, G. (1999). The
determination of three subcutaneous adipose tissue com-
partments in non-insulin-dependent diabetes mellitus
women with artificial neural networks and factor analysis.
Artificial Intelligence in Medicine, 17, 181-193.
Tafeit, E., Moller, R., Sudi, K., & Reibnegger, G. (2000).
Artificial neural networks compared to factor analysis for
low-dimensional classification of high-dimensional body
fat topography data of healthy and diabetic subjects.
Computers and Biomedical Research, 33, 365-374.
502
Factor Analysis in Data Mining
Tan, K., & Wisner, J. (2003). A study of operations
management constructs and their relationships. Interna-
tional Journal of Operations & Production Manage-
ment, 23(11), 1300-1325.
Wall, M., & Amemiya, Y. (2004). A review of nonlinear
factor analysis methods and applications. Proceedings of
the Factor Analysis at 100: Historical Developments
and Future Directions Conference, Chapel Hill, North
Carolina.
Williams, R.H., Zimmerman, D.W., Zumbo, B.D., & Ross,
D. (2003). Charles Spearman: British behavioral scientist.
Human Nature Review. Retrieved from http://human-
nature.com/nibbs/03/spearman.html
KEY TERMS
Cluster Analysis: A multivariate statistical technique
that assesses the similarities between individuals of a
population. Clusters are groups or categories formed so
members within a cluster are less different than members
from different clusters.
Eigenvalue: The quantity representing the variance of
a set of variables included in a factor.
Factor Score: A measure of a factors relative weight
to others, which is obtained using linear combinations of
variables.
Homogeneity: The degree of similarity or uniformity
among individuals of a population.
Homoscedasticity: A statistical assumption for linear
regression models. It requires that the variations around
the regression line be constant for all values of input
variables.
Matrix: An arrangement of rows and columns to
display quantities. A p q matrix contains p q quan-
tities arranged in p rows and q columns (i.e., each row has
q quantities,and each column has p quantities).
Normality: A statistical assumption for linear regres-
sion models. It requires that the errors around the regres-
sion line be normally distributed for each value of input
variable.
Variance: A statistical measure of dispersion around
the mean within the data. Factor analysis divides variance
of a variable into three elementscommon, specific, and
error.
Vector: A quantity having both direction and magni-
tude. This quantity can be represented by an array of
components in a column (column vector) or in a row (row
vector).
TEAM LinG

Financial Ratio Selection for Distress
Classification
Roberto Kawakami Harrop Galvo
Instituto Tecnolgico de Aeronutica, Brazil
Victor M. Becerra
University of Reading, UK
Magda Abou-Seada
Middlesex University, UK
INTRODUCTION
Prediction of corporate financial distress is a subject
that has attracted the interest of many researchers in
finance. The development of prediction models for
financial distress started with the seminal work by Altman
(1968), who used discriminant analysis. Such a tech-
nique is aimed at classifying a firm as bankrupt or
nonbankrupt on the basis of the joint information con-
veyed by several financial ratios.
The assessment of financial distress is usually based
on ratios of financial quantities, rather than absolute
values, because the use of ratios deflates statistics by
size, thus allowing a uniform treatment of different
firms. Moreover, such a procedure may be useful to
reflect a synergy or antagonism between the constitu-
ents of the ratio.
BACKGROUND
The classification of companies on the basis of finan-
cial distress can be performed by using linear discrimi-
nant models (also called Z-score models) of the follow-
ing form (Duda, Hart, & Stork, 2001):
1 2)TS1x
Z(x) = ( (1)
where x = [x1 x2 ... xn]T is a vector of n financial ratios, 1n
and 2n are the sample mean vectors of each group
(continuing and failed companies), and Snn is the common
sample covariance matrix. Equation 1 can also be written
as
Z = w1x 1 + w2x2 + ... + wnxn = wTx (2)
where w = [w1 w2 ... wn]T is a vector of coefficients obtained as
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
503
.
w = S1(m 1 m 2) (3)
The optimal cut-off value for classification z c can be
calculated as
1 2)TS1(
z c = 0.5( 1 + 2) (4)
A given vector x should be assigned to Population 1
if Z(x) > zc, and to Population 2 otherwise.
The generalization (or prediction) performance of
the Z-score model, that is, its ability to classify objects
not used in the modeling phase, can be assessed by using
an independent validation set or cross-validation meth-
ods (Duda et al., 2001). The simplest cross-validation
technique, termed leave-one-out, consists of separat-
ing one of the m modelling objects and obtaining a Z-
score model with the remaining m 1 objects. This model
is used to classify the object that was left out. The
procedure is repeated for each object in the modeling
set in order to obtain a total number of cross-validation
errors.
Resampling techniques (Good, 1999) such as the
Bootstrap method (Davison & Hinkley, 1997) can also
be used to assess the sensitivity of the analysis to the
choice of the training objects.
The Financial Ratio Selection Problem
The selection of appropriate ratios from the available
financial information is an important and nontrivial
stage in building distress classification models. The
best choice of ratios will normally depend on the types
of companies under analysis and also on the economic
context. Although the analysts market insight plays an
important role at this point, the use of data-driven
selection techniques can be of value, because the rel-
evance of certain ratios may only become apparent when
their joint contribution is considered in a multivariate
TEAM LinG

context. Moreover, some combinations of ratios may not
satisfy the statistical assumptions required in the model-
ing process, such as normal distribution and identical
covariances in the groups being classified, in the case of
standard linear discriminant analysis (Duda et al., 2001).
Finally, collinearity between ratios may cause the model
to have poor prediction ability (Naes & Mevik, 2001).
Techniques proposed for ratio selection include nor-
mality tests (Taffler, 1982), and clustering followed by
stepwise discriminant analysis (Alici, 1996).
Most of the works cited in the preceding paragraph
begin with a set of ratios chosen from either popularity
in the literature, theoretical arguments, or suggestions
by financial analysts. However, this article shows that it
is possible to select ratios on the basis of data taken
directly from the financial statements.
For this purpose, we compare two selection methods
proposed by Galvo, Becerra, and Abou-Seada (2004).
A case study involving 60 failed and continuing British
firms in the period from 1997 to 2000 is employed for
illustration.
MAIN THRUST
It is not always advantageous to include all available
variables in the building of a classification model (Duda
et al., 2001). Such an issue has been studied in depth in
the context of spectrometry (Andrade, Gomez-
Carracedo, Fernandez, Elbergali, Kubista, & Prada,
2003), in which the variables are related to the wave-
lengths monitored by an optical instrumentation frame-
work. This concept also applies to the Z-score modeling
process described in the preceding section. In fact,
numerical ill-conditioning tends to increase with (m
n)1, where m is the size of the modeling sample, and n
is the number of variables (Tabachnick & Fidell, 2001).
If n > m, matrix S becomes singular, thus preventing the use
of Equation 1. In this sense, it may be more appropriate to
select a subset of the available variables for inclusion in
the classification model.
The selection procedures to be compared in this
article search for a compromise between maximizing
the amount of discriminating information available for
the model and minimizing collinearity between the clas-
sification variables, which is a known cause of generali-
zation problems (Naes & Mevik, 2001). These goals are
usually conflicting, because the larger the number of
variables, the more information is available, but also the
more difficult it is to avoid collinearity.
504
Financial Ratio Selection for Distress Classification
Algorithm A (Preselection Followed by
Exhaustive Search)
If N variables are initially available for selection, they can
be combined in 2N 1 different subsets (each subset with
a number of variables between 1 and N). Thus, the com-
putational workload can be substantially reduced if
some variables are preliminarily excluded.
In this algorithm, such a preselection is carried out
according to a multivariate relevance index W(x) that
measures the contribution of each variable x to the
classification output when a Z-score model is employed.
This index is obtained by using all variables to build a
model as in Equation 1 and by multiplying the absolute
value of each model weight by the sample standard
deviation (including both groups) of the respective vari-
able.
An appropriate threshold value for the relevance
index W(x) can be determined by augmenting the model-
ing data with artificial uninformative variables (noise)
and then obtaining a Z-score model. Those variables
whose relevance is not considerably larger than the
average relevance of the artificial variables are then
eliminated (Centner, Massart, Noord, Jong, Vandeginste,
& Sterna, 1996).
After the preselection phase, all combinations of the
remaining variables are tested. Subsets with the same
number of variables are compared on the basis of the
number of classification errors on the modelling set for
a Z-score model and the condition number of the matrix
of modeling data. The condition number (the ratio
between the largest and smallest singular value of the
matrix) should be small to avoid collinearity problems
(Navarro-Villoslada, Perez-Arribas, Leon-Gonzalez, &
Polodiez, 1995). After the best subset has been deter-
mined for each given number of variables, a cross-
validation procedure is employed to find the optimum
number of variables.
Algorithm B (Genetic Selection)
The drawback of the preselection procedure employed
in Algorithm A is that some variables that display a small
relevance index when all variables are considered to-
gether could be useful in smaller subsets. An alternative
to such a preselection consists of employing a genetic
algorithm (GA), which tests subsets of variables in an
efficient way instead of performing an exhaustive search
(Coley, 1999; Lestander, Leardi, & Geladi, 2003).
The GA represents subsets of variables as individu-
als competing for survival in a population. The genetic
TEAM LinG
Financial Ratio Selection for Distress Classification

code of each individual is stored in a chromosome, which were extracted from the statements, allowing 28 ratios to
is a string of N binary genes, each gene associated to one be built, as shown in Table 1. Quantities WC, PBIT, EQ, .
of the variables available for selection. The genes with S, TL, ARP, and TA are commonly found in the financial
value 1 indicate the variables that are to be included in the distress literature (Altman, 1968; Taffler, 1982; Alici,
classification model. 1996), and the ratios shown in boxes are those adopted
In the formulation adopted here, the measure F of the by Altman (1968). It is worth noting that the book value
survival fitness of each individual is defined as follows. A of equity was used rather than the market value of equity
Z-score model is obtained from Equation 1 with the vari- to allow the inclusion of firms not quoted in the stock
ables indicated in the chromosome, and then F is calcu- market. Quantity RPY is not typically employed in dis-
lated as tress models, but we include it here to illustrate the ability
of the selection algorithms to discard uninformative
F = (e + r)1 (5) variables. The data employed in this example are given in
Galvo, Becerra, and Abou-Seada (2004).
where e is the number of classification errors in the mod- The data set was divided into a modeling set (21 failed
eling set, r is the condition number associated to the and 21 continuing firms) and a validation set (8 failed and
variables included in the model, and > 0 is a design 10 continuing firms). In what follows, the errors will be
parameter that balances modeling accuracy against col- divided into Type 1 (failed company classified as con-
linearity prevention. The larger is, the more emphasis is tinuing) and 2 (continuing company classified as failed).
placed on avoiding collinearity.
After a random initialization of the population, the Conventional Financial Ratios
algorithm proceeds according to the classic evolution-
ary cycle (Coley, 1999). At each generation, the roulette Previous studies (Becerra, Galvo, & Abou-Seada,
method is used for mating pool selection, followed by 2001) with this data set revealed that when the five
the genetic operators of one-point crossover and point conventional ratios are employed, Ratio 13 (PBIT/TA)
mutation. The population size is kept constant, with each is actually redundant and should be excluded from the
generation replacing the previous one completely. How- Z-score model in order to avoid collinearity problems.
ever, the best-fitted individual is preserved from one Thus, Equation 1 was applied only to the remaining four
generation to the next (elitism) in order to prevent ratios, leading to the results shown in Table 2. It is
good solutions from being lost. worth noting that if Ratio PBIT/TA is not discarded, the
number of validation errors increases from four to
seven.
CASE STUDY
Algorithm A
This example employs financial data from 29 failed and
31 continuing British corporations in the period from The preselection procedure was carried out by aug-
1997 to 2000. The data for the failed firms were taken menting the 28 financial ratios with seven uninforma-
from the last financial statements published prior to the tive variables yielded by an N(0,1) random number
start of insolvency proceedings. Eight financial quantities generator. The relevance index thus obtained is shown
in Figure 1. The threshold value, represented by a hori-
zontal line, was set to five times the average relevance of
Table 1. Numbering of financial ratios (Num/Den). the uninformative variables. As a result, 13 ratios were
Conventional ratios are displayed in boxes. WC = discarded.
working capital, PBIT = profit before interest and tax, After the preselection phase, combinations of the 15
EQ = equity, S = Sales, TL = total liabilities, ARP = remaining ratios were tested for modeling accuracy and
accumulated retained profit, RPY = retained profit for
the year, TA = total assets.
Table 2. Results of a Zscore model using four
Den
Num conventional ratios
WC PBIT EQ S TL ARP RPY
WC
PBIT 1 Type 1 Type 2 Percent
EQ 2 8
Data set
errors errors accuracy
S 3 9 14
Modeling 2 7 79%
TL 4 10 15 19
ARP 5 11 16 20 23 Cross- 3 8 74%
RPY 6 12 17 21 24 26 validation
TA 7 13 18 22 25 27 28 Validation 0 4 78%

505

TEAM LinG

Figure 1. Relevance index of the financial ratios.
Log 10 values are displayed for the convenience of
visualization.
condition number. Figure 2 displays the number of mod-
eling and cross-validation errors for the best subsets
obtained as a function of the number n of ratios included.
The cross-validation curve reaches a minimum between
five and seven ratios. In this situation, the use of five
ratios was deemed more appropriate due to the use of the
well-known Parsimony Principle, which states that given
models with similar prediction ability, the simplest should
be favored (Duda et al., 2001). The selected ratios were 7
(WC/TA), 9 (PBIT/S), 10 (PBIT/TL), 13 (PBIT/TA), and 25
(TL/TA). Interestingly, two of these ratios (WC/TA and
PBIT/TA) belong to the set advocated by Altman (1968).
Note that no ratio involving quantity RPY was selected,
Figure 2. Modeling (dashed line) and cross-validation
(solid line) errors as a function of the number of ratios
included in the Z-score model. The arrow indicates
the final choice of ratios.
506
Financial Ratio Selection for Distress Classification
Table 3. Results of a Z-score model using the five
ratios selected by Algorithm A
Data Type 1 Type 2 Percent
set errors errors accuracy
Modeling 1 3 90%
Cross- 2 5 83%
validation
Validation 1 2 83%
which is in agreement with the fact that in comparison with
the other financial quantities used in this study, RPY has
not often been used in the financial distress literature. The
results of using the five selected ratios are summarized in
Table 3.
Algorithm B
The genetic algorithm was employed with a population
size of 100 individuals. The crossover and mutation
probabilities were set to 60% and 5% , respectively, and
the number of generations was set to 100.
Three values were tested for the design parameter :
10, 1, and 0,1. For each value of , the algorithm was run
five times, starting from different populations, and the
best result (in terms of fitness) was kept. The selected
ratios are shown in Table 4, which also presents the
number of modeling and cross-validation errors in each
case. Regardless of the value of , no ratio involving
quantity RPY was selected (see Table 1).
On the basis of cross-validation performance, = 1
is seen to be the best choice (Ratios 2, 9, and 15). In fact,
a smaller value for , which causes the algorithm to
place less emphasis on collinearity avoidance, results in
the selection of more ratios. As a result, there is a gain
in modeling accuracy but not in generalization ability
(as assessed by cross-validation), which means that the
model has become excessively complex. On the other
hand, a larger value for discouraged the selection of
sets with more than one ratio. In this case, there is not
enough discriminating information, which results in
poor modeling and cross-validation performances.
Table 5 details the results for = 1. Notice that one of
the ratios obtained in this case belongs to the set used by
Altman (EQ/TL). It is also worth pointing out that Ratios
2 and 9 were discarded in the preselection phase of
Algorithm A, which prevented the algorithm from taking
the GA solution {2, 9, 15} into account.
Resampling Study
The results in the preceding two sections were obtained
for one given partition of the available data into model-
ing and validation sets. In order to assess the validity of
TEAM LinG
Financial Ratio Selection for Distress Classification
Table 4. GA results for different values of the weight
parameter
Selected Modeling Cross-validation
ratios errors errors
10 {7} 11 11
1 {2,9,15} 4 5
0.1 {2,9,13,18,19,22,25} 3 7
the ratio selection scheme employed, a resampling study
was carried out. For this purpose, 1,000 different model-
ing/validation partitions were randomly generated with
the same size as the one employed before (42 modeling
companies and 18 validation companies). For each parti-
tion, a Z-score model was built and validated by using the
subsets of ratios obtained in the previous subsections.
Table 6 presents the average number of resulting errors.
As can be seen, the ratios selected by Algorithms A
and B lead to better classifiers than the conventional
ones. The best result, in terms of both modeling and
validation performances, was obtained by Algorithm B.
Such a finding is in line with the parsimony of the
associated classifier, which employs only three ratios.
FUTURE TRENDS
The research on distress prediction has been moving
towards nonparametric modeling in order to circumvent
the limitations of discriminant analysis (such as the
need for the classes to exhibit multivariate normal
distributions with identical covariances). In this con-
text, neural networks have been found to be a useful
alternative, as demonstrated in a number of works (Wil-
son & Sharda, 1994; Alici, 1996; Atiya, 2001; Becerra
et al., 2001).
In light of the utility of data-driven ratio selection
techniques for discriminant analysis, as discussed in
this article, it is to be expected that ratio selection may also
play an important role for neural network models. How-
ever, future developments along this line will have to
address the difficulty that nonlinear classifiers usually
have to be adjusted by numerical search techniques
(training algorithms), which may be affected by local
minima problems (Duda et al., 2001).
Table 5. Results of a Z-score model using the three
ratios selected by Algorithm B
Data Type 1 Type 2 Percent
set errors errors accuracy
Modeling 4 0 90%
Cross- 4 1 88%
validation
Validation 1 3 78%
Table 6. Resampling results (average number of errors)
.
Ratios employed
Data
Conventional Algorithm A Algorithm B
set
(4 ratios) (5 ratios) (3 ratios)
Modeling 8.52 5.70 5.36
Validation 4.69 3.22 2.89
CONCLUSION
The selection of appropriate ratios from the available
financial quantities is an important and nontrivial step in
building models for corporate financial distress classi-
fication. This article shows how data-driven variable
selection techniques can be useful tools in building
distress classification models. The article compares the
results of two such techniques, one involving
preselection followed by exhaustive search, and the
other employing a genetic algorithm.
REFERENCES
Alici, Y. (1996). Neural networks in corporate failure
prediction: The UK experience. In A. P. Refenes, Y.
Abu-Mostafa, & J. Moody (Eds.), Neural networks in
financial engineering. London: World Scientific.
Altman, E. I. (1968). Financial ratios, discriminant analysis
and the prediction of corporate bankruptcy. Journal of
Finance, 23, 505-609.
Andrade, J. M., Gomez-Carracedo, M. P., Fernandez, E.,
Elbergali, A., Kubista, M., & Prada, D. (2003). Classifi-
cation of commercial apple beverages using a minimum
set of mid-IR wavenumbers selected by Procrustes ro-
tation. Analyst, 128(9), 1193-1199.
Atiya, A. F. (2001). Bankruptcy prediction for credit
risk using neural networks: A survey and new results.
IEEE Transactions on Neural Networks, 12(4), 929-935.
Becerra, V. M., Galvo, R. K. H., & Abou-Seada, M. (2001).
Financial distress classification employing neural net-
works. Proceedings of the IASTED International Confer-
ence on Artificial Intelligence and Applications (pp. 45-
49).
Centner, V., Massart, D. L., Noord, O. E., Jong, S.,
Vandeginste, B. M., & Sterna, C. (1996). Elimination of
uninformative variables for multivariate calibration.
Analytical Chemistry, 68, 3851-3858.
507
TEAM LinG

Coley, D. A. (1999). An introduction to genetic algo-
rithms for scientists and engineers. Singapore: World
Scientific.
Davison, A. C., & Hinkley, D. V. (Eds.). (1997). Bootstrap
methods and their application. Cambridge, MA: Cam-
bridge University Press.
Duda, R. O., Hart, P. E., & Stork, D. G. (2001). Pattern
classification (2nd ed.). New York: Wiley.
Galvo, R. K. H., Becerra, V. M., & Abou-Seada, M.
(2004). Ratio selection for classification models. Data
Mining & Knowledge Discovery, 8, 151-170.
Good, P. I. (1999). Resampling methods: A practical
guide to data analysis. Boston, MA: Birkhauser.
Lestander, T. A., Leardi, R., & Geladi, P. (2003). Selec-
tion of near infrared wavelengths using genetic algo-
rithms for the determination of seed moisture content.
Journal of Near Infrared Spectroscopy, 11(6), 433-446.
Naes, T., & Mevik, B. H. (2001). Understanding the col-
linearity problem in regression and discriminant analysis.
Journal of Chemometrics, 15(4), 413-426.
Navarro-Villoslada, F., Perez-Arribas, L. V., Leon-
Gonzalez, M. E., & Polodiez, L. M. (1995). Selection of
calibration mixtures and wavelengths for different mul-
tivariate calibration methods. Analytica Chimica Acta,
313(1-2), 93-101.
Tabachnick, B. G., & Fidell, L. S. (2001). Using multi-
variate statistics (4th ed.). Boston, MA: Allyn & Bacon.
Taffler, R. J. (1982). Forecasting company failure in the
UK using discriminant analysis and financial ratio data.
Journal of the Royal Statistical Society, Series A, 145,
342-358.
508
Financial Ratio Selection for Distress Classification
Wilson, R. L., & Sharda, R. (1994). Bankruptcy prediction
using neural networks. Decision Support Systems, 11,
545-557.
KEY TERMS
Condition Number: Ratio between the largest and
smallest singular values of a matrix, often employed to
assess the degree of collinearity between variables asso-
ciated to the columns of the matrix.
Cross-Validation: Resampling method in which ele-
ments of the modeling set itself are alternately removed
and reinserted for validation purposes.
Financial Distress: A company is said to be under
financial distress if it is unable to pay its debts as they
become due, which is aggravated if the value of the
firms assets is lower than its liabilities.
Financial Ratio: Ratio formed from two quantities
taken from a financial statement.
Genetic Algorithm: Optimization technique inspired
by the mechanisms of evolution by natural selection, in
which the possible solutions are represented as the chro-
mosomes of individuals competing for survival in a popu-
lation.
Linear Discriminant Analysis: Multivariate clas-
sification technique that models the classes under con-
sideration by normal distributions with equal covari-
ances, which leads to hyperplanes as the optimal deci-
sion surfaces.
Resampling: Validation technique employed to as-
sess the sensitivity of the classification method with
respect to the choice of modeling data.
TEAM LinG

Flexible Mining of Association Rules
Hong Shen
Japan Advanced Institute of Science and Technology, Japan
INTRODUCTION
The discovery of association rules showing conditions
of data co-occurrence has attracted the most attention
in data mining. An example of an association rule is the
rule the customer who bought bread and butter also
bought milk, expressed by T(bread; butter)T(milk).
Let I ={x1,x2,,xm} be a set of (data) items, called the
domain; let D be a collection of records (transactions),
where each record, T, has a unique identifier and con-
tains a subset of items in I. We define itemset to be a set
of items drawn from I and denote an itemset containing k
items to be k-itemset. The support of itemset X, denoted
by (X/D), is the ratio of the number of records (in D)
containing X to the total number of records in D. An
association rule is an implication rule X Y, where X;
Y I and XIY=0. The confidence of X Y is the ratio of
(X U Y/D) to (X/D), indicating that the percentage of
those containing X also contain Y. Based on the user-
specified minimum support (minsup) and confidence
(minconf), the following statements are true: An itemset
X is frequent if (X/D)> minsup, and an association rule
( X UY / D )
X Y is strong if X U Y is frequent and ( X /Y )
The problem of mining association rules is to find all
strong association rules, which can be divided into two
subproblems:
1. Find all the frequent itemsets.
2. Generate all strong rules from all frequent itemsets.
Because the second subproblem is relatively straight-
forward we can solve it by extracting every subset from
an itemset and examining the ratio of its support; most of
the previous studies (Agrawal, Imielinski, & Swami, 1993;
Agrawal, Mannila, Srikant, Toivonen, & Verkamo, 1996;
Park, Chen, & Yu, 1995; Savasere, Omiecinski, & Navathe,
1995) emphasized on developing efficient algorithms for
the first subproblem.
This article introduces two important techniques for
association rule mining: (a) finding N most frequent
itemsets and (b) mining multiple-level association rules.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
509
.
BACKGROUND
An association rule is called binary association rule if all
items (attributes) in the rule have only two values: 1 (yes)
or 0 (no). Mining binary association rules was the first
proposed data mining task and was studied most inten-
sively. Centralized on the Apriori approach (Agrawal et
al., 1993), various algorithms were proposed (Savasere et
al., 1995; Shen, 1999; Shen, Liang, & Ng, 1999; Srikant &
Agrawal, 1996). Almost all the algorithms observe the
downward property that all the subsets of a frequent
itemset must also be frequent, with different pruning
strategies to reduce the search space. Apriori works by
finding frequent k-itemsets from frequent (k-1)-itemsets
iteratively for k=1, 2, , m-1.
Two alternative approaches, mining on domain parti-
tion (Shen, L., Shen, H., & Cheng, 1999) and mining based
on knowledge network (Shen, 1999) were proposed. The
first approach partitions items suitably into disjoint
itemsets, and the second approach maps all records to
individual items; both approaches aim to improve the
bottleneck of Apriori that requires multiple phases of
scans (read) on the database.
minconf. Finding all the association rules that satisfy minimal
support and confidence is undesirable in many cases for
a users particular requirements. It is therefore necessary
to mine association rules more flexibly according to the
users needs. Mining different sets of association rules of
a small size for the purpose of predication and classifica-
tion were proposed (Li, Shen, & Topor, 2001; Li, Shen, &
Topor, 2002; Li, Shen, & Topor, 2004; Li, Topor, & Shen,
2002).
MAIN THRUST
Association rule mining can be carried out flexibly to
suit different needs. We illustrate this by introducing
important techniques to solve two interesting problems.
TEAM LinG

Finding N Most Frequent Itemsets
Given x, y I, we say that x is greater than y, or y is less
than x, if ( x / D) > ( y / D) . The largest itemset in D is the
itemset that occurs most frequently in D. We want to find
the N largest itemsets in D, where N is a user-specified
number of interesting itemsets. Because users are usually
interested in those itemsets with larger supports, finding
N most frequent itemsets is significant, and its solution
can be used to generate an appropriate number of inter-
esting itemsets for mining association rules (Shen, L.,
Shen, H., Pritchard, & Topor, 1998).
We define the rank of itemset x, denoted by (x), as
follows: (x) ={(y/D)>(x/D), y I}|+1. Call x a
winner if (x)<N and (x/D)>1, which means that x is one
of the N largest itemsets and it occurs in D at least once.
We dont regard any itemset with support 0 as a winner,
even if it is ranked below N, because we do not need to
provide users with an itemset that doesnt occur in D at
all.
Use W to denote the set of all winners and call the
support of the smallest winner the critical support,
denoted by crisup. Clearly, W exactly contains all
itemsets with support exceeding crisup; we also have
crisup>1. It is easy to see that |W| may be different from
N: If the number of all itemsets occurring in D is less
than N, |W| will be less than N; |W| may also be greater
than N, as different itemsets may have the same support.
The problem of finding the N largest itemsets is to
generate W.
Let x be an itemset. Use Pk(x) to denote the set of
all k-subsets (subsets with size k) of x. Use Uk to denote
P1(I) U U Pk(I), the set of all itemsets with a size not
greater than k. Thus, we introduce the k-rank of x, denoted
by k(x), as follows: k(x) = |{y| {(y/D)>(x/D),
y Uk}|+1. Call x a k-winner if k(x)<N and (x/D)>1,
which means that among all itemsets with a size not greater
than k, x is one of the N largest itemsets and also occurs
in D at least once. Use Wk to denote the set of all k-winners.
We define k-critical-support, denoted by k-crisup, as
follows: If |W k|<N, then k-crisup is 1; otherwise, k-crisup
is the support of the smallest k-winner. Clearly, Wk exactly
contains all itemsets with a size not greater than k and
support not less than k-crisup. We present some useful
properties of the preceding concepts as follows.
Property: Let k and i be integers such that 1<k<k+i <|I|.
(1) Given x Uk, we have x Wk iff (x/D)>k-crisup.
(2) If W k-1 = W k, then W=Wk
(3) Wk+i I Uk Wk.
(4) 1<k-crisup<(k +i)-crisup.
510
Flexible Mining of Association Rules
To find all the winners, the algorithm makes multiple
passes over the data. In the first pass, we count the
supports of all 1-itemsets, select the N largest ones from
them to form W 1, and then use W 1 to generate potential 2-
winners with size (2). Each subsequent pass k involves
three steps: First, we count the support for potential k-
winners with size k (called candidates) during the pass
over D; then select the N largest ones from a pool pre-
cisely containing supports of all these candidates and all
(k-1)-winners to form W k; finally, use W k to generate
potential (k+1)-winners with size k+1, which will be used
in the next pass. This process continues until we cant get
any potential (k+1)-winners with size k+1, which implies
that W k+1 = Wk. From Property 2, we know that the last Wk
exactly contains all winners.
We assume that M k is the number of itemsets with
support equal to k-crisup and a size not greater than k,
where 1<k<|I|, and M is the maximum of all M1~M|I|. Thus,
we have |Wk|=N+M k-1<N+M. It was shown that the time
complexity of the algorithm is proportional to the number
of all the candidates generated in the algorithm, which is
O(log(N+M)*min{N+M,|I|}*(N+M)) (Shen et al., 1998).
Hence, the time complexity of the algorithm is polynomial
for bounded N and M.
Mining Multiple-Level Association Rules
Although most previous research emphasized mining
association rules at a single concept level (Agrawal et
al., 1993; Agrawal et al., 1996; Park et al., 1995; Savasere
et al., 1995; Srikant & Agrawal, 1996), some techniques
were also proposed to mine rules at generalized abstract
(multiple) levels (Han & Fu, 1995). However, they can
only find multiple-level rules in a fixed concept hierar-
chy. Our study in this fold is motivated by the goal of
mining multiple-level rules in all concept hierarchies
(Shen, L., & Shen, H., 1998).
A concept hierarchy can be defined on a set of data-
base attribute domains such as D(a1),,D(an), where, for
i [1, n], ai denotes an attribute, and D(a i) denotes the
domain of ai. The concept hierarchy is usually partially
ordered according to a general-to-specific ordering. The
most general concept is the null description ANY, whereas
the most specific concepts correspond to the specific
attribute values in the database. Given a set of
D(a1),,D(an), we define a concept hierarchy H as fol-
lows: H n H n - 1 H 0 , where
H = D(a1 ) L D(aii ) f o r
i i
i[0,n], and
{ a 1 , , a n } = { a1 ,..., an } { a1 ,..., an 1 } .
n n n n 1
Here, Hn represents the set of concepts at the primitive
level, Hn-1 represents the concepts at one level higher
than those at Hn, and so forth; H0, the highest level
hierarchy, may contain solely the most general concept,
TEAM LinG
Flexible Mining of Association Rules
ANY. We also use { a1n ,..., ann }{ a1n ,..., ann11 }...{ a11 } to
denote H directly, and H0 may be omitted here.
We introduce FML items to represent concepts at any
level of a hierarchy. Let *, called a trivial digit, be a dont-
care digit. An FML item is represented by a sequence of
digits, x = x1x2 xn, xi D(ai) 7 {*}. The flat-set of x is
defined as Sf (x) ={(I, x i) | i [1; n] and xi *}. Given two
items x and y, x is called a generalized item of y if Sf (x) Sf
(y), which means that x represents a higher-level concept
that contains the lower-level concept represented by y.
Thus, *5* is a generalized item of 35* due to Sf(*5*)
={(2,5)} Sf (35*)={(1,3),(2,5)}. If Sf(x)=, then x is called
a trivial item, which represents the most general concept,
ANY.
Let T be an encoded transaction table, t a transaction
in T, x an item, and c an itemset. We can say that (a) t
supports x if an item y exists in t such that x is a generalized
item of y and (b) t supports c if t supports every item in c.
The support of an itemset c in T, (c/T), is the ratio of the
number of transactions (in T) that support c to the total
number of transactions in T. Given a minsup, an itemset c
is large if (c/T)e>minsup; otherwise, it is small.
Given an itemset c, we define its simplest form as
Fs(c)={x c| y c; Sf (x) Sf(y)} and its complete form
as Fc(c)={x|Sf (x) Sf (y), y c}.
Given an itemset c, we call the number of elements in
Fs(c) its size and the number of elements in Fc(c) its weight.
An itemset of size j and weight k is called a (j)-itemset, [k]-
itemset, or (j)[k]-itemset. Let c be a (j)[k]-itemset. Use Gi(c)
to indicate the set of all [i]-generalized-subsets of c, where
i<k. Thus, the set of all [k-1]-generalized-subsets of c can
be generated as follows: Gk-1(c) = {Fs(Fc(c)- {x}) | x Fs(c)};
the size of Gk-1(c) is j. Hence, for k>1, c is a (1)-itemset iff
|Gk-1(c)|=1. With this observation, we call a [k-1]-itemset a
self-extensive if a (1)[k]-itemset b exists such that Gk-1(b)
= {a}; at the same time, we call b the extension result of a.
Thus, all self-extensive itemsets as can be generated from
all (1)-itemsets bs as follows: Fc(a) = Fc(b)-Fs(b).
Let b be a (1)-itemset and Fs(b)={x}. From |Fc(b)| = |{y|Sf
(y) Sf (x)}| = 2| S ( x )| , we know that b is a (1) [ 2| S ( x )| ]-itemset.
f f
Let a be the self-extensive itemset generated by b; that is,
a is a [ 2| S ( x )| -1]-itemset such that Fc(a)=Fc(b)-Fs(b). Thus,
f |m|-sub-outcomes of o versus l. Let o be an |m+1|-outcome
if |Sf (x)| > 1, there exist y, z Fs(a) and yz such that Sf
(x)=Sf(y) 7 Sf(z). Clearly, this property can be used to gen-
erate the corresponding extension result from any self-
extensive [2m-1]-itemset, where m > 1. For simplicity, given
a self-extensive [2m-1]-itemset a, where m>1, we directly
use Er(a) to denote its extension result. For example,
Er({12*,1*3,*23}) = {123}.
The algorithm makes multiple passes over the data-
base. In the first pass, we count the supports of all [2]-
itemsets and then select all the large ones. In pass k-1, we
start with L k-1, the set of all large [k-1]-itemsets, and use
Lk-1 to generate Ck, a superset of all large [k]-itemsets. Call .
the elements in Ck candidate itemsets, and count the
support for these itemsets during the pass over the data.
At the end of the pass, we determine which of these
itemsets are actually large and obtain Lk for the next pass.
This process continues until no new large itemsets are
found. Not that L1={{trivial item}}, because {trivial item}
is the unique [1]-itemset and is supported by all transac-
tions.
The computation cost of the preceding algorithm,
which finds all frequent FML itemsets is O( cC s( x) ),
where C is the set of all candidates, g(c) is the cost for
generating c as a candidate, and s(c) is the cost for
counting the support of c (Shen, L., & Shen, H., 1998).
The algorithm is optimal if the method of support
counting is optimal.
After all frequent FML itemsets have been found,
we can proceed with the construction of strong FML
rules. Use r(l,a) to denote rule Fs(a)Fs(Fc(l)-Fc(a)),
where l is an itemset and a l. Use Fo(l,a) to denote
Fc(l)-Fc(a) and say that Fo(l,a) is an outcome form of l
or the outcome form of r(l,a). Note that Fo(l,a) repre-
sents only a specific form rather than a meaningful
itemset, so it is not equivalent to any other itemset
whose simplest form is Fs(Fo(l,a)). Outcome forms are
also called outcomes directly. Clearly, the correspond-
ing relationship between rules and outcomes is one to
one. An outcome is strong if it corresponds to a strong
rule. Thus, all strong rules related to a large itemset can
be obtained by finding all strong outcomes of this
itemset.
Let l be an itemset. Use O(l) to denote the set of all
outcomes of l; that is, O(l) ={Fo(l,a)|a l}. Thus, from
O(l), we can output all rules related to l: Fs(Fc(l)-o) Fs(o)
(denoted by r(l , o) ), where o O(l). Clearly, r(l,a) and
r(l , o) denote the same rule if o=Fo(l,a). Let o, o O(l ) . We
can say two things: (a) o is a |k|-outcome of l if o exactly
contains k elements and (b) o is a sub-outcome of o
versus l if o o . Use Ok(l) to denote the set of all the |k|-
outcomes of l and use Vm(o,l) to denote the set of all the
of l and me>1. If |Vm(o,l)| =1, then o is called an elementary
outcome; otherwise, o is a non-elementary outcome.
Let r(l,a) and r(l,b) be two rules. We can say that r(l,a)
is an instantiated rule of r(l,b) if b a. Clearly, b a
(l / T ) (l / T )
implies (b/T)>(a/T) and . Hence, all
(a / T ) (b / T )
instantiated rules of a strong rule must also be strong. Let
l be a large itemset, o1, o2 O(l), o1=Fo(l,a), and o2 =
Fo(l,b). The three straightforward conclusions are (a)
511
TEAM LinG

o1 o2 iff b a, (b) r(l , o1 ) = r(l,a), and (c) r(l , o2 ) = r(l,b).
Therefore, r(l , o1 ) is an instantiated rule of r(l , o2 ) iff
o1 o2 , which implies that all the suboutcomes of a strong
outcome must also be strong. This characteristic is similar
to the property that all generalized subsets of a large
itemset must also be large. Hence, the algorithm works as
follows. From a large itemset l, we first generate all the
strong rules with |1|-outcomes (ignore the |0|-outcome ,
which only corresponds to all trivial strong rules; Fs(l)
for every large itemset l). We then use all these strong |1|-
outcomes to generate all the possible strong |2|-outcomes
of l, from which all the strong rules with |2|-outcomes can
be produced, and so forth.
The computation cost of the algorithm constructing
all the strong FML rules is O(|C|t + |C|s) (Shen, L., & Shen,
H., 1998), which is optimal for bounded costs t and s,
where C is the set of all the candidate rules, s is the average
cost for generating one element in C, and t is the average
cost for computing the confidence of one element in C.
FUTURE TRENDS
The discovery of data association is a central topic of
data mining. Mining this association flexibly to suit
different needs has an increasing significance for appli-
cations. Future research trends include mining associa-
tion rules in data of complex types such as multimedia,
time-series, text, and biological databases, in which
certain properties (e.g., multidimensionality), con-
straints (e.g., time variance), and structures (e.g., se-
quence) of the data must be taken into account during the
mining process; mining association rules in databases
containing incomplete, missing, and erroneous data,
which requires people to produce correct results re-
gardless of the unreliability of data; mining association
rules online for streaming data that come from an open-
ended data stream and flow away instantaneously at a
high rate, which usually adopts techniques such as active
learning, concept drift, and online sampling and pro-
ceeds in an incremental manner.
CONCLUSION
We have summarized research activities in mining asso-
ciation rules and some recent results of my research in
different streams of this topic. We have introduced impor-
tant techniques to solve two interesting problems of
mining N most frequent itemsets and mining multiple-level
association rules, respectively. These techniques show
512
Flexible Mining of Association Rules
that mining association rules can be performed flexibly in
user-specified forms to suit different needs. More rel-
evant results can be found in Shen (1999). We hope that
this article provides useful insight to researchers for
better understanding to more complex problems and their
solutions in this research direction.
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
associations between sets of items in massive data-
bases. Proceedings of the ACM SIGMOD International
Conference on Management of Data (pp. 207-216).
Agrawal, R., Mannila, H., Srikant,Toivonen, R. H., &
Verkamo, A. I. (1996). Fast discovery of association
rules. In U. Fayyad (Ed.), Advances in knowledge discov-
ery and data mining (pp. 307-328). MIT Press.
Han, J., & Fu, Y. (1995). Discovery of multiple-level asso-
ciation rules from large databases. Proceedings of the
21st VLDB International Conference (pp. 420-431).
Li, J., Shen, H., & Topor, R. (2001). Mining the smallest associa-
tion rule set for predictions. Proceedings of the IEEE Interna-
tional Conference on Data Mining (pp. 361-368).
Li, J., Shen, H., & Topor, R. (2002). Mining the optimum
class association rule set. Knowledge-Based Systems,
15(7), 399-405.
Li, J., Shen, H., & Topor, R. (2004). Mining the informative
rule set for prediction. Journal of Intelligent Information
Systems, 22(2), 155-174.
Li, J., Topor, R., & Shen, H. (2002). Construct robust rule
sets for classification. Proceedings of the ACM Interna-
tional Conference on Knowledge Discovery and Data
Mining (pp. 564-569).
Park, J. S., Chen, M., & Yu, P. S. (1995). An effective hash
based algorithm for mining association rules. Proceed-
ings of the ACM SIGMOD International Conference on
Management of Data (pp. 175-186).
Savasere, A., Omiecinski, R., & Navathe, S. (1995). An
efficient algorithm for mining association rules in large
databases. Proceedings of the 21st VLDB International
Conference (pp. 432-443).
Shen, H. (1999). New achievements in data mining. In J.
Sun (Ed.), Science: Advancing into the new millenium
(pp. 162-178). Beijing: Peoples Education.
Shen, H., Liang, W., & Ng, J. K-W. (1999). Efficient
computation of frequent itemsets in a subcollection of
multiple set families. Informatica, 23(4), 543-548.
TEAM LinG
Flexible Mining of Association Rules
Shen, L., & Shen, H. (1998). Mining flexible multiple-level
association rules in all concept hierarchies. Proceedings
of the Ninth International Conference on Database and
Expert Systems Applications (pp. 786-795).
Shen, L., Shen, H., & Cheng, L. (1999). New algorithms for
efficient mining of association rules. Information Sci-
ences, 118, 251-268.
Shen, L., Shen, H., Pritchard, P., & Topor, R. (1998). Finding
the N largest itemsets. Proceedings of the IEEE Interna-
tional Conference on Data Mining, 19 (pp. 211-222).
Srikant, R., & Agrawal, R. (1996). Mining quantitative
association rules in large relational tables. ACM
SIGMOD Record, 25(2), 1-8.
KEY TERMS
Association Rule: An implication rule X Y that
shows the conditions of co-occurrence of disjoint
itemsets (attribute value sets) X and Y in a given database.
Concept Hierarchy: The organization of a set of data-
base attribute domains into different levels of abstraction .
according to a general-to-specific ordering.
Confidence of Rule X Y: The fraction of the data-
base containing X that also contains Y, which is the ratio
of the support of X U Y to the support of X.
Flexible Mining of Association Rules: Mining asso-
ciation rules in user-specified forms to suit different
needs, such as on dimension, level of abstraction, and
interestingness.
Frequent Itemset: An itemset that has a support
greater than user-specified minimum support.
Strong Rule: An association rule whose support (of
the union of itemsets) and confidence are greater than
user-specified minimum support and confidence, re-
spectively.
Support of Itemset X: The fraction of the database
that contains X, which is the ratio of the number of
records containing X to the total number of records in
the database.
513
TEAM LinG
514

Formal Concept Analysis Based Clustering

Jamil M. Saquer
Southwest Missouri State University, USA

INTRODUCTION Similarly, the set of objects possessing all the features in

a set of features B is denoted by (B) and is given by {g
Formal concept analysis (FCA) is a branch of applied G | gIm " m B}. The operators and b satisfy the
mathematics with roots in lattice theory (Wille, 1982; assertions given in the following lemma.
Ganter & Wille, 1999). It deals with the notion of a concept
in a given universe, which it calls context. For example, Lemma 1 (Wille, 1982): Let (G, M, I) be a context. Then
consider the context of transactions at a grocery store the following assertions hold:
where each transaction consists of the items bought
together. A concept here is a pair of two sets (A, B). A is 1. A1 A2 implies b(A2) b(A1) for every A1, A2 G,
the set of transactions that contain all the items in B and and B 1 B2 implies (B2) (B 1) for every B1, B 2
B is the set of items common to all the transactions in A. M.
A successful area of application for FCA has been data 2. A (b(A) and A = b((b(A)) for all A G, and B
mining. In particular, techniques from FCA have been b(((B)) and B = (b( (B))) for all B M.
successfully used in the association mining problem and
in clustering (Kryszkiewicz, 1998; Saquer, 2003; Zaki & A formal concept in the context (G, M, I) is defined as
Hsiao, 2002). In this article, we review the basic notions a pair (A, B) where A G, B M, b(A) = B, and (B) = A.
of FCA and show how they can be used in clustering. A is called the extent of the formal concept and B is called
its intent. For example, the pair (A, B) where A = {2, 3, 4}
and B = {a, g, h} is a formal concept in the context given
BACKGROUND in Table 1. A subconcept/superconcept order relation on
concepts is as follows: (A1, B 1) (A2, B 2) iff A1 A2 (or
A fundamental notion in FCA is that of a context, which equivalently, iff B2 B1). The fundamental theorem of
is defined as a triple (G, M, I), where G is a set of objects, FCA states that the set of all concepts on a given context
M is a set of features (or attributes), and I is a binary is a complete lattice, called the concept lattice (Ganter &
relation between G and M. For object g and feature m, gIm Wille, 1999). Concept lattices are drawn using Hasse
if and only if g possesses the feature m. An example of a diagrams, where concepts are represented as nodes. An
context is given in Table 1, where an X is placed in the edge is drawn between concepts C1 and C2 iff C1 C 2 and
i th row and jth column to indicate that the object in row i there is no concept C 3 such that C1 C3 C2. The concept
possesses the feature in column j. lattice for the context in Table 1 is given in Figure 1.
The set of features common to a set of objects A is A less condensed representation of a concept lattice
denoted by (A) and is defined as {m M | gIm " g A}. is possible using reduced labeling (Ganter & Wille, 1999).
Figure 2 shows the concept lattice in Figure 1 with
reduced labeling. It is easier to see the relationships and
Table 1. A context excerpted from (Ganter & Wille, 1999, similarities among objects when reduced labeling is used.
p. 18) a = needs water to live; b = lives in water; c = lives The extent of a concept C in Figure 2 consists of the
on land; d = needs chlorophyll; e = two seeds leaf; f = one objects at C and the objects at the concepts that can be
seed leaf; g = can move around; h = has limbs; i = suckles reached from C going downward following descending
its offsprings paths towards the bottom concept. Similarly, the intent of
C consists of the features at C and the features at the
a b c d e f g h i
1 Leech X X X concepts that can be reached from C going upwards
2 Bream X X X X following ascending paths to the top concept.
3 Frog X X X X X Consider the context presented in Table 1. Let B = {a,
4 Dog X X X X X f}. Then, (B) = {5, 6, 8}, and b((B)) = b({5, 6, 8}) = {a, d,
5 Spike-weed X X X X
f} {a, f}; therefore, in general, ((B)) B. A set of
6 Reed X X X X X
7 Bean X X X X features B that satisfies the condition b((B)) = B is called
8 Maize X X X X a closed feature set. Intuitively, a closed feature set is a

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
Formal Concept Analysis Based Clustering
Figure 1. Concept lattice for the context in Table 1
Figure 2. Concept lattice for the context in Table 1 with reduced labeling
maximal set of features shared by a set of objects. It is easy
to show that intents of the concepts of a concept lattice
are all closed feature sets.
The support of a set of features B is defined as the
percentage of objects that possess every feature in B.
That is, support(B) = |(B)|/|G|, where |B| is the cardinality
of B. Let minSupport be a user-specified threshold value
for minimum support. A feature set B is frequent iff
support(B) minSupport. A frequent closed feature set is
a closed feature set, which is also frequent. For example,
for minSupport = 0.3, {a, f} is frequent, {a, d, f} is frequent
closed, while {a, c, d, f} is closed but not frequent.
CLUSTERING BASED ON FCA
It is believed that the method described below is the first
for using FCA for disjoint clustering. Using FCA for
conceptual clustering to gain more information about data
is discussed in Carpineto & Romano (1999) and Mineau
& Godin (1995). In the remainder of this article we show
how FCA can be used for clustering.
.
Traditionally, most clustering algorithms do not allow
clusters to overlap. However, this is not a valid assump-
tion for many applications. For example, in Web docu-
ments clustering, many documents have more than one
topic and need to reside in more than one cluster (Beil,
Ester, & Xu, 2002; Hearst, 1999; Zamir & Etzioni, 1998).
Similarly, in the market basket data, items purchased in a
transaction may belong to more than one category of
items.
The concept lattice structure provides a hierarchical
clustering of objects, where the extent of each node could
be a cluster and the intent provides a description of that
cluster. There are two main problems, though, that make
it difficult to recognize the clusters to be used. First, not
all objects are present at all levels of the lattice. Second,
the presence of overlapping clusters at different levels is
not acceptable for disjoint clustering. The techniques
described in this chapter solve these problems. For ex-
ample, for a node to be a cluster candidate, its intent must
be frequent (meaning a minimum percentage of objects
must possess all the features of the intent). The intuition
is that the objects within a cluster must contain many
515
TEAM LinG

features in common. Overlapping is resolved by using a
score function that measures the goodness of a cluster for
an object and keeps the object in the cluster where it scores
best.
Formalizing the Clustering Problem
Given a set of objects G = {g1, g2, , gn}, where each object
is described by the set of features it possesses, (i.e., gi is
described by b(gi)), = {C1, C2, , Ck} is a clustering of G
if and only if each Ci is a subset of G and UCi = G, where i
ranges from 1 to k. For disjoint clustering, the additional
condition Ci I Cj = must be satisfied for all i j.
Our method for disjoint clustering consists of two
steps. First, assign objects to their initial clusters. Second,
make these clusters disjoint. For overlapping clustering,
only the first step is needed.
Assigning Objects to Initial Clusters
Each frequent closed feature set (FCFS) is a cluster candi-
date, with the FCFS serving as a label for that cluster. Each
object g is assigned to the cluster candidates described by
the maximal frequent closed feature set (MFCFS) con-
tained in b(g). These initial clusters may not be disjoint
because an object may contain several MFCFS. For ex-
ample, for minSupport = 0.375, object 2 in Table 1 contains
the MFCFSs agh and abg.
Notice that all of the objects in a cluster must contain
all the features in the FCFS describing that cluster (which
is also used as the cluster label). This is always true even
after any overlapping is removed. This means that this
method produces clusters with their descriptions. This is
a desirable property. It helps a domain expert to assign
labels and descriptions to clusters.
Making Clusters Disjoint
To make the clusters disjoint, we find the best cluster for
each overlapping object g and keep g only in that cluster.
To achieve this, a score function is used. The function
score(g, C i) measures the goodness of cluster Ci for object
g. Intuitively, a cluster is good for an object g if g has many
frequent features which are also frequent in Ci. On the other
hand, C i is not good for g if g has frequent features that are
not frequent in Ci.
Define global-support(f) as the percentage of objects
possessing f in the whole database, and cluster-support(f)
as the percentages of objects possessing f in a given
cluster C i. We say f is cluster frequent in C i if cluster-
support(f) is at least a user-specified minimum threshold
value q. For cluster Ci, let positive(g, Ci) be the set of
features in b(g) which are both global-frequent (i.e., fre-
quent in the whole database) and cluster-frequent. Also let
516
Formal Concept Analysis Based Clustering
negative(g, Ci) be the set of features in b(g) which are
global-frequent but not cluster-frequent. The function
score(g, Ci) is then given by the following formula:
score(g, C i) = f positive(g, Ci) cluster-support(f) -
f negative(g, Ci) global-support(f).
The first term in score(g, Ci) favors Ci for every feature
in positive(g, Ci) because these features contribute to
intra-cluster similarity. The second term penalizes Ci for
every feature in negative(g, C i) because these features
contribute to inter-cluster similarities.
An overlapping object will be deleted from all initial
clusters except for the cluster where it scores highest.
Ties are broken by assigning the object to the cluster
with the longest label. If this does not resolve ties, then
one of the clusters is chosen randomly.
An Illustrating Example
Consider the objects in Table 1. The closed feature sets
are the intents of the concept lattice in Figure 1. For
minSupport = 0.35, a feature set must appear in at least 3
objects to be frequent. The frequent closed feature sets
are a, ag, ac, ab, ad, agh, abg, acd, and adf. These are the
candidate clusters. Using the notation C[x] to indicate
the cluster with label x, and assigning objects to MFCFS
results in the following initial clusters: C[agh] = {2, 3, 4},
C[abg] = {1, 2, 3}, C[acd] = {6, 7, 8}, and C[adf] = {5, 6,
8}. To find the most suitable cluster for object 6, we need
to calculate its score in each cluster containing it. For
cluster-support threshold value of 0.7, it is found that,
score(6,C[acd]) = 1 - 0.63 + 1 + 1 - 0.38 = 1.99 and
score[6,C[adf]) = 1 - 0.63 - 0.63 + 1 + 1 = 1.74.
We will use score(6,C[acd]) to explain the calculation.
All features in b(6) are global-frequent. They are a, b, c,
d, and f, with global frequencies 1, 0.63, 0.63, 0.5, and 0.38,
respectively. Their respective cluster-support values are
1, 0.33, 1, 1, and 0.67. For a feature to be cluster-frequent
in C[acd], it must appear in at least [ x |C[acd]|] = 3 of its
objects. Therefore, a, c, d positive(6,C[acd]), and b, f
negative(6,C[acd]). Substituting these values into the
formula for the score function, it is found that the
score(6,C[acd]) = 1.99. Since score(6,C[acd]) >
score[6,C[adf]), object 6 is assigned to C[acd].
Tables 2 and 3 show the score calculations for all
overlapping objects. Table 2 shows initial cluster as-
signments. Features that are both global-frequent and
cluster-frequent are shown in the column labeled
positive(g,C i), and features that are only global-frequent
are in the column labeled negative(g,Ci). For elements in
the column labeled positive(g,Ci), the cluster-support
values are listed between parentheses after each feature
name. The same format is used for global-support values
TEAM LinG
Formal Concept Analysis Based Clustering

Table 2. Initial cluster assignments and support values

(minSupport = 0.37, = 0.7) .
Ci initial clusters Positive(g,Ci) negative(g,Ci)

C[agh] 2: abgh a(1), b(0.63),

3: abcgh g(1), c(0.63)

4: acghi h(1)

C[abg] 1: abg a(1), c(0.63),

2: abgh b(1), h(0.38)

3: abcgh g(1)

C[acd] 6: abcdf a(1), b(0.63),

7: acde c(1), f(0.38)

8: acdf d(1)

C[adf] 5: abdf a(1), b(0.63),

6: abcdf d(1), c(0.63)

8: acdf f(1)

Table 3. Score calculations and final cluster assignments and = 0.5, we get the following final clusters C[agh] = {3,
(minSupport = 0.37, = 0.7) 4}, C[abg] = {1, 2}, C[acd] = {7, 8}, and C[adf] = {5, 6}.

Ci Score(g, Ci) Final clusters

C[agh] 2: 1 - .63 + 1 + 1 = 2.37 FUTURE TRENDS
3: 1 - .63 - .63 + 1 + 1 = 1.74
4: does not overlap 4 We have shown how FCA can be used for clustering.
C[abg] 1: does not overlap 1 Researchers have also used the techniques of FCA and
2: 1 + 1 + 1 - .38 = 2.62 2 frequent itemsets in the association mining problem (Fung,
3: 1 + 1 - .63 + 1 - .38 = 1.99 3
Wang, & Ester, 2003; Wang, Xu, & Liu, 1999; Yun, Chuang,
C[acd] 6: 1 - .63 + 1 +1 - .38 = 2.62 6
5: does not overlap 7
& Chen, 2001). We anticipate this framework to be suitable
8: 1 + 1 + 1 -.38 = 2.62 8 for other problems in data mining such as classification.
C[adf] 5: does not overlap 5 Latest developments in efficient algorithms for generat-
6: 1 - .63 - .63 + 1 + 1 = 1.74 ing frequent closed itemsets make this approach efficient
8: 1 - .63 + 1 + 1 = 2.37 for handling large amounts of data (Gouda & Zaki, 2001;
Pei, Han, & Mao, 2000; Zaki & Hsiao, 2002).

for features in the column labeled negative(g,Ci). It is only
a coincidence that in this example all cluster-support
values are 1 (try = 0.5 for different values). Table 3 shows
score calculations, and final cluster assignments.
Notice that, different threshold values may result in
different clusters. The value of minSupport affects cluster
labels and initial clusters while that of affects final
elements in clusters. For example, for minSupport = 0.375
CONCLUSION
This chapter introduces formal concept analysis (FCA);
a useful framework for many applications in computer
science. We also showed how the techniques of FCA can
be used for clustering. A global support value is used to
specify which concepts can be candidate clusters. A
score function is then used to determine the best cluster
for each object. This approach is appropriate for cluster-

517

TEAM LinG

ing categorical data, transaction data, text data, Web
documents, and library documents. These data usually
suffer from the problem of high dimensionality with only
few items or keywords being available in each transaction
or document. FCA contexts are suitable for representing
this kind of data.
REFERENCES
Beil, F., Ester, M., & Xu, X. (2002). Frequent term-based
text clustering. In The 8th International Conference on
Knowledge Discovery and Data Mining (KDD 2002)
(pp. 436-442).
Carpineto, C., & Romano, G. (1993). GALOIS: An order-
theoretic approach to conceptual clustering. In Proceed-
ings of 1993 International Conference on Machine Learn-
ing (pp. 33-40).
Fung, B., Wang, K., & Ester, M. (2003). Large hierarchical
document clustering using frequent itemsets. In Third
SIAM International Conference on Data Mining (pp. 59-
70).
Ganter, B., & Wille, R. (1999). Formal concept analysis:
Mathematical foundations. Berlin: Springer-Verlag.
Gouda, K., & Zaki, M. (2001). Efficiently mining maximal
frequent itemsets. In First IEEE International Confer-
ence on Data Mining (pp. 163-170). San Jose, USA.
Hearst, M. (1999). The use of categories and clusters for
organizing retrieval results. In T. Strzalkowski (Ed.), Natu-
ral language information retrieval (pp. 333-369). Bos-
ton: Kluwer Academic Publishers.
Kryszkiewicz, M. (1998). Representative association rules.
In Proceedings of PAKDD 98. Lecture Notes in Artifi-
cial Intelligence (Vol. 1394) (pp. 198-209). Berlin: Springer-
Verlag.
Mineau, G., & Godin, R. (1995). Automatic structuring of
knowledge bases by conceptual clustering. IEEE Trans-
actions on Knowledge and Data Engineering, 7 (5), 824-
829.
Pei J., Han J., & Mao R. (2000). CLOSET: an efficient
algorithm for mining frequent closed itemsets. In ACM-
SIGMOD Workshop on Research Issues in Data Mining
and Knowledge Discovery (pp. 21-30). Dallas, USA.
518
Formal Concept Analysis Based Clustering
Saquer, J. (2003). Using concept lattices for disjoint clus-
tering. In The Second IASTED International Conference
on Information and Knowledge Sharing (pp. 144-148).
Wang, K., Xu, C., & Liu, B. (1999). Clustering transactions
using large items. In ACM International Conference on
Information and Knowledge Management (pp. 483-490).
Wille, R. (1982). Restructuring lattice theory: An ap-
proach based on hierarchies of concepts. In I. Rival (Ed.),
Ordered sets (pp. 445-470). Dordecht-Boston: Reidel.
Yun, C., Chuang, K., & Chen, M. (2001). An efficient
clustering algorithm for market basket data based on small
large ratios. In The 25th COMPSAC Conference (pp. 505-
510).
Zaki, M.J., & Hsiao, C. (2002). CHARM: An efficient
algorithm for closed itemset mining. In Second SIAM
International Conference on Data Mining.
Zamir, O., & Etzioni, O. (1998). Web document clustering:
A feasibility demonstration. In The 21st Annual Interna-
tional ACM SIGIR (pp. 46-54).
KEY TERMS
Cluster-Support of Feature F In Cluster Ci: Percent-
age of objects in Ci possessing f.
Concept: A pair (A, B) of a set A of objects and a set
B of features such that B is the maximal set of features
possessed by all the objects in A and A is the maximal set
of objects that possess every feature in B.
Context: A triple (G, M, I) where G is a set of
objects, M is a set of features and I is a binary relation
between G and M such that gIm if and only if object g
possesses the feature m.
Formal Concept Analysis: A mathematical framework
that provides formal and mathematical treatment of the
notion of a concept in a given universe.
Negative(g, Ci): Set of features possessed by g which
are global-frequent but not cluster-frequent.
Positive(g, Ci): Set of features possessed by g which
are both global-frequent and cluster-frequent.
Support or Global-Support of Feature F: Percentage
of object transactions in the whole context (or whole
database) that possess f.
TEAM LinG
 519

Fuzzy Information and Data Analysis .

Reinhard Viertl
Vienna University of Technology, Austria

INTRODUCTION precise. For details see (Viertl, 2002). This kind of uncer-
tainty can be best described by a so-called fuzzy number.
The results of data warehousing and data mining are A fuzzy number x is defined by a so-called characteriz-
depending essentially on the quality of data. Usually ing function : IR [0,1] which obeys the following:
data are assumed to be numbers or vectors, but this is
often not realistic. Especially the result of a measure-
ment of a continuous quantity is always not a precise
number, but more or less non-precise. This kind of x0 IR : (x0 ) = 1 (1)
uncertainty is also called fuzziness and should not be
confused with errors. Data mining techniques have to (0,1] the so-called cut C [ ()] defined by
take care of fuzziness in order to avoid unrealistic
C [ ()] := {x IR : (x ) } = [a , b ] is a finite closed interval.
results.
(2)

BACKGROUND
Examples of non-precise data are results on ana-
logue measurement equipments as well as readings on
In standard data warehousing and data analysis data are
digital instruments.
treated as numbers, vectors, words, or symbols. These
For continuous vector quantities real measurements
data types do not take care of fuzziness of data and prior
are not precise vectors but also non-precise. This im-
information. Whereas some methodology for fuzzy data
analysis was developed, statistical data analysis is usu- precision can result in a vector (x1 ,L , x k ) of fuzzy num-
ally not taking care of fuzziness. Recently some meth- bers xi , or more generally, in a so-called k-dimen-
ods for statistical analysis of non-precise data were
sional fuzzy vector x . Using the notation
published (Viertl, 1996, 2003).
Historically fuzzy sets were first introduced by K. Menger x = (x1 ,L , x k ) IR k a k-dimensional fuzzy vector is
in 1951 (Menger, 1951). Later L. Zadeh made fuzzy models
defined by its vector-characterizing function
popular. For more information on fuzzy modeling compare
(Dubois & Prade, 2000). : IR k [0,1] obeying
Most data analysis techniques are statistical tech-
niques. Only in the last 20 years alternative methods
using fuzzy models were developed. For a detailed discus- x 0 IR k : (x 0 ) = 1 (1)
sion compare (Bandemer & Nther, 1992; Berthold &
Hand, 2003).
(0,1] the cut C [ ()] defined by
C [ ()] := {x IR k : (x ) } is a compact simply con-
MAIN THRUST nected subset of IR k (2)

The main thrust of this chapter is to provide the quanti-

tative description of fuzzy data, as well as generalized
methods for the statistical analysis of fuzzy data.
Non-Precise Data
The result of one measurement of a continuous quantity
is not a precise real number but more or less non-
Remark: A vector of fuzzy numbers is essentially
different from a fuzzy vector. But it is possible to
construct a fuzzy vector x from a vector 1 (),L , k () of
fuzzy numbers. The vector-characterizing function
(,L ,) of x can be obtained by

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
 Fuzzy Information and Data Analysis

(x1 , L , x k ) = min{1 (x1 ),L , k (x k )} (x1 ,L , x k ) IR k The generalized integral of a fuzzy valued function
f () defined on M is a fuzzy number I denoted by

Examples of 2-dimensional fuzzy data are light

points on radar screens. I = f (x )dx
,
M
Descriptive Statistics with Fuzzy Data
which is defined via its cuts
Analysis of variable data by forming histograms has to
take care of fuzziness. This is possible based on the

characterizing functions i () of the observations xi for C [I ] = f (x )dx, f (x )dx (0,1]
i = 1(1)n . The height h j over a class K j , j = 1(1)k of the
M M
histogram is a fuzzy number whose characterizing func-
tion j () is obtained in the following way: in case of integrable level curves f () and f () .
For each the cut Fuzzy probability densities on measurable spaces
level
(M , A ) are special fuzzy valued functions f () defined
[ ] [ ]
C j () = h n , (K j ), h n, (K j ) of j () is defined by
on M with integrable level curves, for which

# {xi: C [ i ()] K j }
f (x )dx = 1

+
,
h n, (K j ) =
M

n
where 1+ is a fuzzy number fulfilling
# {xi: C [ i ()] K j }
h n, (K j ) = . 1 C1 [1+ ] and C [1+ ] (0, ) (0,1] .
n

Based on fuzzy probability densities so-called fuzzy

By the representation lemma of fuzzy numbers hereby probability distributions P on A are defined in the
following way:
the characterizing functions j (), j = 1(1)k are determined:
Denoting the set of all classical probability densi-
ties () on M which are bounded by level curves
j (x ) = max I C [ j ( )] (x ) x IR
(0 ,1] f () and f () by

The resulting generalized histogram is also called

fuzzy histogram. For more details compare (Viertl & {
S = () : f (x ) (x ) f (x ) x M }
Hareter, 2004).
the fuzzy associated probability P (A) for all A A is
Fuzzy Probability Distributions the fuzzy number whose cuts
In standard data analysis probability densities are con- [
C [P ( A)] = P (A), P ( A) are ] defined by
sidered as limits of histograms. For fuzzy data limits of
fuzzy histograms are fuzzy valued functions f () whose
values f (x ) are fuzzy numbers. These fuzzy valued P (A) = sup (x )dx
S A
functions are normalized by generalizing classical inte-
gration with the help of so-called level curves f ()
P (A) = inf (x )dx .
and f () of f () , which are defined by the endpoints of S
A

the cuts of f (x ) for all x Def [ f ()]:

By this definition the probability of the extreme
events and M are precise numbers, i.e.
[
C [ f (x )] = f (x ), f (x ) ] for all x Def [ f ()]

520

TEAM LinG
Fuzzy Information and Data Analysis

P ( ) = 0 and P (M ) = 1 . For more details compare the book (Viertl, 1996)

Remark: For precise data xi IR the characterizing .
Statistical Inference based on Fuzzy functions are the one-point indicator functions I {x } () i

Information and the resulting characterizing functions are the indi-

Statistical inference procedures can be generalized to
the situation of fuzzy data. Moreover also Bayesian
inference procedures can be generalized for fuzzy a-
priori distributions and fuzzy sample data.
For stochastic model X ~ f ( | ) , with observa-
tion space M and sample x1 , L , xn let (x1 ,L , x n ) be an
estimator for the true parameter, i.e. a measurable func-
tion from the sample space M n to . In case of fuzzy
sample x1 ,L , xn the estimator can be adapted using the
so-called extension principle from fuzzy set theory in
the following way. In order to do this first the fuzzy
sample x1 ,L , x n with characterizing functions
1 (), L , n () has to be combined into a fuzzy element x
of the sample space M n with vector-characterizing func-
tion (,L,) whose values are defined by
(x1 , L , x n ) = min i (xi ) (x1 , L , x n ) M n .
i =1(1) n
cator functions of the classical results.
Statistical tests in case of fuzzy data yield fuzzy
values t of test statistics t (x1 ,L , xn ) . Therefore simple
significance tests based on rejection regions dont
apply. A solution to this problem is to use p-values. An
alternative approach is so-called fuzzy p-values as
described in (Filzmoser & Viertl, 2004).
Fuzzy Bayesian Inference
The second kind of fuzzy information appears as fuzzy
a-priori information in the context of Bayesian data
analysis. Fuzzy a-priori information is expressed by
fuzzy probability distributions on the parameter space
in connection with a stochastic model X ~ f ( | ) ,
with observation space M . Frequently such fuzzy
probability distributions are generated by fuzzy prob-
ability densities as described in the subsection Fuzzy
Probability Distributions above.
Using the concept of level curves () and ()

of fuzzy a-priori densities () on the parameter space

Let : M be an estimator for the parameter
n
, it is possible to generalize Bayes theorem to the
of a stochastic model X ~ f ( | ) , .
situation of fuzzy a-priori information and fuzzy data
Using the so-called fuzzy combined sample x the D . The resulting a-posteriori distribution is a fuzzy
characterizing function () of the generalized fuzzy probability distribution with fuzzy density ( | D ) on
estimator is given by its values the parameter space.
The fuzzy density ( | D ) can be used to calculate

sup{ (x ) : (x )= } if 1 ( ) predictive densities p( | D ) by the generalized integra-

( ) = . tion from above in the following way:
0 if 1 ( ) =

p(x | D ) = f (x | ) ( | D )d for all xM .

The generalized estimator is a fuzzy element of the
parameter space .
A similar construction is possible to generalize the This is again a fuzzy probability distribution, now on the
concept of confidence sets. Let (X 1 ,L , X n ) be a confi- observation space M . For more details see (Viertl & Hareter,
dence function for at given confidence level 1 . For 2004).
fuzzy data with fuzzy combined sample x with vector-
Fuzzy Data Analysis
characterizing function (,L ,) a generalized fuzzy con-
fidence region 1 with membership function () is In recent times non-statistical approaches to data analy-
defined by sis problems were developed using ideas from fuzzy
theory. Especially for linguistic data this approach was
successful. Instead of stochastic models to describe
sup{ (x ) : (x )} if x : (x ) uncertainty it is possible to use fuzzy models. Such
( ) = .
0 if /x : (x ) alternative methods are fuzzy clustering, fuzzy regres-

521

TEAM LinG

sion analysis, integration of fuzzy valued functions, fuzzy
optimization, and fuzzy approximation. For more details
compare (Bandemer & Nther, 1992).
FUTURE TRENDS
Up to now fuzzy models where mainly used to describe
vague data in form of linguistic data. But also precision
measurements are connected with uncertainty which is
not only of stochastic nature. Therefore hybrid methods
of data analysis, which take care of the fuzziness of data
as well as the statistical variation of them, have to be
applied and further developed. Especially research on
how to obtain the characterizing functions of non-pre-
cise data are necessary.
CONCLUSION
In this contribution different kinds of uncertainty are
discussed. These are variation and fuzziness. Varia-
tion is usually described by stochastic models and
fuzziness by fuzzy numbers and fuzzy vectors respec-
tively. Looking at real data it is necessary to distinguish
between fuzziness and variability in order to obtain
realistic results from data analyses.
REFERENCES
Bandemer, H., & Nther, W. (1992). Fuzzy data analy-
sis. Dordrecht: Kluwer.
Berthold, M., & Hand, D. (Eds.). (2003). Intelligent
data analysis (2nd ed.). Berlin: Springer.
Bertoluzza, C., Gil, M., & Ralescu, D. (Eds.). (2002).
Statistical modeling, analysis and management of
fuzzy data. Heidelberg: Physica-Verlag.
Dubois, D., & Prade, H. (Eds.). (2000). Fundamentals of
fuzzy sets. Boston: Kluwer.
Filzmoser, P., & Viertl, R. (2004). Testing hypotheses
with fuzzy data: The fuzzy p- value. Metrika, 59, 21-29.
Grzegorzewski, P., Hryniewicz, O., & Gil, M. (Eds.).
(2002). Soft methods in probability, statistics and
data analysis. Heidelberg: Physica-Verlag.
Menger, K. (1951). Ensembles flous et functions
aleatoires. Comptes Rendus Acad. Sci., 232, 2001-2003.
Mller, B., & Beer, M. (2004). Fuzzy randomness
Uncertainty in civil engineering and computational
mechanics. Berlin: Springer.
522
Fuzzy Information and Data Analysis
Ross, R., Booker, J., & Parkinson, J., (Eds.) (2002).
Fuzzy logic and probability applications Bridging
the gap. Philadelphia: American Statistical Association
and Society of Industrial and Applied Mathematics.
Viertl, R. (1996). Statistical methods for non-precise
data. Boca Raton: CRC Press.
Viertl, R. (2002). On the description and analysis of
measurements of continuous quantities. Kybernetika,
38, 353-362.
Viertl, R. (2003). Statistical inference with imprecise
data. Encyclopedia of life support systems. Retrieved
from UNESCO, Paris, Web Site: www.eolss.unesco.org
Viertl, R., & Hareter, D. (2004). Fuzzy information and
imprecise probability. Zeitschrift fr Angewandte
Mathematik und Mechanik, 84(10-11), 1-10.
Wolkenhauer, O. (2001). Data engineering Fuzzy
mathematics in systems theory and data analysis.
New York: Wiley.
KEY TERMS
Characterizing Function: Mathematical descrip-
tion of a fuzzy number.
Fuzzy Estimate: Generalized statistical estimation
technique for the situation of non-precise data.
Fuzzy Histogram: A generalized histogram based
on non-precise data, whose heights are fuzzy numbers.
Fuzzy Information: Information which is not given
in form of precise numbers, precise vectors, or pre-
cisely defined terms.
Fuzzy Number: Quantitative mathematical descrip-
tion of fuzzy information concerning a one-dimensional
numerical quantity.
Fuzzy Statistics: Statistical analysis methods for
the situation of fuzzy information.
Fuzzy Valued Functions: Generalized real valued
functions whose values are fuzzy numbers.
Fuzzy Vector: Mathematical description of non-
precise vector quantities.
Hybrid Data Analysis Methods: Data analysis tech-
niques using methods from statistics and from fuzzy
modeling.
Non-Precise Data: Data which are not precise num-
bers or not precise vectors.
TEAM LinG

A General Model for Data Warehouses
Michel Schneider
Blaise Pascal University, France
INTRODUCTION
Basically, the schema of a data warehouse lies on two
kinds of elements: facts and dimensions. Facts are used
to memorize measures about situations or events. Di-
mensions are used to analyse these measures, particu-
larly through aggregation operations (counting, summa-
tion, average, etc.). To fix the ideas let us consider the
analysis of the sales in a shop according to the product
type and to the month in the year. Each sale of a product
is a fact. One can characterize it by a quantity. One can
calculate an aggregation function on the quantities of
several facts. For example, one can make the sum of
quantities sold for the product type mineral water
during January in 2001, 2002 and 2003. Product type is
a criterion of the dimension Product. Month and Year
are criteria of the dimension Time. A quantity is so
connected both with a type of product and with a month
of one year. This type of connection concerns the orga-
nization of facts with regard to dimensions. On the other
hand a month is connected to one year. This type of
connection concerns the organization of criteria within
a dimension. The possibilities of fact analysis depend
on these two forms of connection and on the schema of
the warehouse. This schema is chosen by the designer in
accordance with the users needs.
Determining the schema of a data warehouse cannot
be achieved without adequate modelling of dimensions
and facts. In this article we present a general model for
dimensions and facts and their relationships. This model
will facilitate greatly the choice of the schema and its
manipulation by the users.
BACKGROUND
Concerning the modelling of dimensions, the objective
is to find an organization which corresponds to the
analysis operations and which provides strict control
over the aggregation operations. In particular it is im-
portant to avoid double-counting or summation of non-
additive data. Many studies have been devoted to this
problem. Most recommend organizing the criteria (we
said also members) of a given dimension into hierar-
chies with which the aggregation paths can be explicitly
defined. In (Pourabbas, 1999), hierarchies are defined
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
523
/
by means of a containment function. In (Lehner, 1998),
the organization of a dimension results from the func-
tional dependences which exist between its members,
and a multi-dimensional normal form is defined. In
(Hsemann, 2000), the functional dependences are also
used to design the dimensions and to relate facts to
dimensions. In (Abello, 2001), relationships between
levels in a hierarchy are apprehended through the Part-
Whole semantics. In (Tsois, 2001), dimensions are
organized around the notion of a dimension path which
is a set of drilling relationships. The model is centered
on a parent-child (one to many) relationship type. A
drilling relationship describes how the members of a
children level can be grouped into sets that correspond
to members of the parent level. In (Vassiliadis, 2000), a
dimension is viewed as a lattice and two functions anc
and desc are used to perform the roll up and the drill
down operations. Pedersen (1999) proposes an ex-
tended multidimensional data model which is also based
on a lattice structure, and which provides non-strict
hierarchies (i.e. too many relationships between the
different levels in a dimension).
Modelling of facts and their relationships has not
received so much attention. Facts are generally consid-
ered in a simple fashion which consists in relating a fact
with the roots of the dimensions. However, there is a
need for considering more sophisticated structures
where the same set of dimensions are connected to
different fact types and where several fact types are
inter-connected. The model described in (Pedersen,
1999) permits some possibilities in this direction but is
not able to represent all the situations.
Apart from these studies it is important to note
various propositions (Agrawal, 1997; Datta, 1999;
Gyssens, 1997; Nguyen, 2000) for cubic models where
the primary objective is the definition of an algebra for
multidimensional analysis. Other works must also be
mentioned. In (Golfarelli, 1998), a solution is proposed to
derive multidimensional structures from E/R shemas. In
(Hurtado, 2001) are established conditions for reasoning
about summarizability in multidimensional structures.
MAIN THRUST
Our objective in this article is to propose a generic
model based on our personal research work and which
TEAM LinG

integrates existing models. This model can be used to
apprehend the sharing of dimensions in various ways and
to describe different relationships between fact types.
Using this model, we will also define the notion of well-
formed warehouse structures. Such structures have de-
sirable properties for applications. We suggest a graph
representation for such structures which can help the
users in designing and requesting a data warehouse.
Modelling Facts
A fact is used to record measures or states concerning an
event or a situation. Measures and states can be analysed
through different criteria organized in dimensions.
A fact type is a structure
fact_name[(fact_key),
(list_of_reference_attributes),
(list_of_fact_attributes)]
where
fact_name is the name of the type;
fact_key is a list of attribute names; the concat-
enation of these attributes identifies each instance
of the type;
list_of_reference_attributes is a list of attribute
names; each attribute references a member in a
dimension or another fact instance;
list_of_fact_attributes is a list of attribute names;
each attribute is a measure for the fact.
The set of referenced dimensions comprises the
dimensions which are directly referenced through the
list_of_reference_attributes, but also the dimensions
which are indirectly referenced through other facts.
Each fact attribute can be analysed along each of the
referenced dimensions. Analysis is achieved through
the computing of aggregate functions on the values of
this attribute.
As an example let us consider the following fact type
for memorizing the sales in a set of stores.
Sales[(ticket_number, product_key), (time_key,
product_key, store_key),
(price_per_unit, quantity)]
The key is (ticket_number, product_key). This means
that there is an instance of Sales for each different
product of a ticket. There are three dimension refer-
ences: time_key, product_key, store_key. There are
two fact attributes: price_per_unit, quantity. The fact
attributes can be analysed through aggregate operations
by using the three dimensions.
524
A General Model for Data Warehouses
There may be no fact attribute; in this case a fact
records the occurrence of an event or a situation. In such
cases, analysis consists in counting occurrences satis-
fying a certain number of conditions.
For the needs of an application, it is possible to
introduce different fact types sharing certain dimen-
sions and having references between them.
Modelling Dimensions
The different criteria which are needed to conduct analy-
sis along a dimension are introduced through members.
A member is a specific attribute (or a group of at-
tributes) taking its values on a well defined domain. For
example, the dimension TIME can include members
such as DAY, MONTH, YEAR, etc. Analysing a fact
attribute A along a member M means that we are inter-
ested in computing aggregate functions on the values of
A for any grouping defined by the values of M. In the
article we will also use the notation Mij for the j-th
member of i-th dimension.
Members of a dimension are generally organized in
a hierarchy which is a conceptual representation of the
hierarchies of their occurrences. Hierarchy in dimen-
sions is a very useful concept that can be used to impose
constraints on member values and to guide the analysis.
Hierarchies of occurrences result from various rela-
tionships which can exist in the real world: categoriza-
tion, membership of a subset, mereology. Figure 1
illustrates a typical situation which can occur. Note that
a hierarchy is not necessarily a tree.
We will model a dimension according to a hierarchi-
cal relationship (HR) which links a child member Mij
(i.e. town) to a parent member Mik (i.e. region) and we
will use the notation MijMik. For the following we
consider only situations where a child occurrence is
linked to a unique parent occurrence in a type. However,
a child occurrence, as in case (b) or (c), can have several
parent occurrences but each of different types. We will
also suppose that HR is reflexive, antisymmetric and
transitive. This kind of relationship covers the great
majority of real situations. Existence of this HR is very
important since it means that the members of a dimension
Figure 1. A typical hierarchy in a dimension
ck1 ck2 ck3 ck4 .
cust_key
town demozone
Dallas Mesquite Akron . North North .
region Texas Ohio .
TEAM LinG
A General Model for Data Warehouses
can be organized into levels and correct aggregation of
fact attribute values along levels can be guaranteed.
Each member of a dimension can be an entry for this
dimension i.e. can be referenced from a fact type. This
possibility is very important since it means that dimen-
sions between several fact types can be shared in various
ways. In particular, it is possible to reference a dimen-
sion at different levels of granularity. A dimension root
represents a standard entry. For the three dimensions in
Figure 1, there is a single root. However, definition 3
authorizes several roots.
As in other models (Hsemann, 2000), we consider
property attributes which are used to describe the mem-
bers. A property attribute is linked to its member through
a functional dependence, but does not introduce a new
member and a new level of aggregation. For example the
member town in the customer dimension may have prop-
erty attributes such as population, administrative posi-
tion, etc. Such attributes can be used in the selection
predicates of requests to filter certain groups.
We now define the notion of member type, which
incorporates the different features presented above.
A member type is a structure:
member_name[(member_key),
dimension_name,
(list_of_reference_attributes)]
where
member_name is the name of the type;
member_key is a list of attribute names; the con-
catenation of these attributes identifies each in-
stance of the type;
list_of_reference_attributes is a list of attribute
names where each attribute is a reference to the
successors of the member instance in the cover
graph of the dimension.
Only the member_key is mandatory.
Using this model, the representations of the members
of dimension customer in Figure 1 are the following:
cust_root[(cust_key), customer, (town_name,
zone_name)]
town[(town_name), customer,(region_name)]
demozone[(zone_name), customer, (region_name)]
region[(region_name), customer, ()]
Well-Formed Structures
In this section we explain how the fact types and the
member types can be interconnected in order to model
various warehouse structures.
First, a fact can directly reference any member of a
dimension. Usually a dimension is referenced through /
one of its roots (as we saw above, a dimension can have
several roots). But it is also interesting and useful to
have references to members other than the roots. This
means that a dimension can be used by different facts
with different granularities. For example, a fact can
directly reference town in the customer dimension and
another can directly reference region in the same
dimension. This second reference corresponds to a
coarser granule of analysis than the first.
Moreover, a fact F1 can reference any other fact F2.
This type of reference is necessary to model real
situations. This means that a fact attribute of F1 can be
analysed by using the key of F2 (acting as the grouping
attribute of a normal member) and also by using the
dimensions referenced by F2.
To formalise the interconnection between facts and
dimensions, we thus suggest extending the HR rela-
tionship of section 3 to the representation of the asso-
ciations between fact types and the associations be-
tween fact types and member types. We impose the
same properties (reflexivity, anti-symmetry, transitiv-
ity). We also forbid cyclic interconnections. This gives
a very uniform model since fact types and member
types are considered equally. To maintain a traditional
vision of the data warehouses, we also ensure that the
members of a dimension cannot reference facts.
Figure 2 illustrates the typical structures we want
to model. Case (a) corresponds to the simple case, also
known as star structure, where there is a unique fact
type F1 and several separate dimensions D1, D2, etc.
Cases (b) and (c) correspond to the notion of facts of
fact. Cases (d), (e) and (f) correspond to the sharing of
the same dimension. In case (f) there can be two differ-
ent paths starting at F2 and reaching the same member
M of the sub-dimension D21. So analysis using these
Figure 2. Typical warehouse structures
F1 F1 F1 F2
D2 D1 F2 D1 D2 F3 D2
D2 D3
(a) (b) (c)
F2
F1 F2 F1 F2
F1
D1 D1
D1
D21
D21
(d) (e) (f)
525
TEAM LinG

Figure 3. The DWG of a star-snowflake structure
sales [(sales_key), (time_key, product_key, cust_key),
(quantity, price_per_unit)]
time_root product_root
[(time_key), ] [(product_key), ]
day category
[(day_no),...] [(category_no), ]
year
[(year_no),] family
[(family_name),]
two paths cannot give the same results when reaching M.
To pose this problem we introduce the DWG and the
path coherence constraint.
To represent data warehouse structures, we suggest
using a graph representation called DWG (data ware-
house graph). It consists in representing each type (fact
type or member type) by a node containing the main
information about this type, and representing each ref-
erence by a directed edge.
Suppose that in the DWG graph, there are two differ-
ent paths P1 and P2 starting from the same fact type F,
and reaching the same member type M. We can analyse
instances of F by using P1 or P2. The path coherence
constraint is satisfied if we obtain the same results when
reaching M.
For example in case of figure 1 this constraint means
the following: for a given occurrence of cust_key,
whether the town path or the demozone path is used, one
always obtains the same occurrence of region.
We are now able to introduce the notion of well-
formed structures.
Definition of a Well-Formed Warehouse
Structure
A warehouse structure is well-formed when the DWG is
acyclic and the path coherence constraint is satisfied
for any couple of paths having the same starting node and
the same ending node.
A well-formed warehouse structure can thus have
several roots. The different paths from the roots can be
always divided into two sub-paths: the first one with
only fact nodes and the second one with only member
nodes. So, roots are fact types.
Since the DWG is acyclic, it is possible to distribute
its nodes into levels. Each level represents a level of
aggregation. Each time we follow a directed edge, the
526
A General Model for Data Warehouses
cust_root
[(cust_key), ]
town demozone
[(town_name), ] [(zone_name), ...]
region
[(region_name),]
level increases (by one or more depending on the used
path). When using aggregate operations, this action corre-
sponds to a ROLLUP operation (corresponding to the
semantics of the HR) and the opposite operation to a
DRILLDOWN. Starting from the reference to a dimension
D in a fact type F, we can then roll up in the hierarchies of
dimensions by following a path of the DWG.
Illustrating the Modelling of a Typical
Case with Well-Formed Structures
Well-formed structures are able to model correctly and
completely the different cases of Figure 2. We illus-
trate in this section the modelling for the star-snow-
flake structure.
We have a star-snowflake structure when:
there is a unique root (which corresponds to the
unique fact type);
each reference in the root points towards a sepa-
rate subgraph in the DWG (this subgraph corre-
sponds to a dimension).
Our model does not differentiate star structures
from snowflake structures. The difference will appear
with the mapping towards an operational model (rela-
tional model for example). The DWG of a star-snow-
flake structure is represented in Figure 3. This repre-
sentation is well-formed. Such a representation can be
very useful to a user for formulating requests.
FUTURE TRENDS
A first opened problem is that concerning the property
of summarizability between the levels of the different
dimensions. For example, the total of the sales of a
TEAM LinG
A General Model for Data Warehouses
product for 2001 must be equal to the sum of the totals
for the sales of this product for all months of 2001. Any
model of data warehouse has to respect this property. In
our presentation we supposed that function HR verified
this property. In practice, various functions were pro-
posed and used. It would be interesting and useful to
begin a general formalization which would regroup all
these propositions.
Another opened problem concerns the elaboration
of a design method for the schema of a data warehouse.
A data warehouse is a data base and one can think that its
design does not differ from that of a data base. In fact a
data warehouse presents specificities which it is neces-
sary to take into account, notably the data loading and
the performance optimization. Data loading can be com-
plex since sources schemas can differ from the data
warehouse schema. Performance optimization arises
particularly when using relational DBMS for imple-
menting the data warehouse.
CONCLUSION
In this paper we propose a model which can describe
various data warehouse structures. It integrates and
extends existing models for sharing dimensions and for
representing relationships between facts. It allows for
different entries in a dimension corresponding to dif-
ferent granularities. A dimension can also have several
roots corresponding to different views and uses. Thanks
to this model, we have also suggested the concept of
Data Warehouse Graph (DWG) to represent a data ware-
house schema. Using the DWG, we define the notion of
well-formed warehouse structures which guarantees
desirable properties.
We have illustrated how typical structures such as
star-snowflake structures can be advantageously repre-
sented with this model. Other useful structures like
those depicted in Figure 2 can also be represented.
The DWG gathers the main information from the
warehouse and it can be very useful to users for formu-
lating requests. We believe that the DWG can be used as
an efficient support for a graphical interface to manipu-
late multidimensional structures through a graphical
language.
The schema of a data warehouse represented with our
model can be easily mapped into an operational model.
Since our model is object-oriented a mapping towards
an object model is straightforward. But it is possible
also to map the schema towards a relational model or an
object relational model. It appears that our model has a
natural place between the conceptual schema of the
application and an object relational implementation of
the warehouse. It can thus serve as a helping support for
the design of relational data warehouses.
REFERENCES
/
Abello, A., Samos, J., & Saltor, F. (2001). Understand-
ing analysis dimensions in a multidimensional object-
oriented model. Intl Workshop on Design and Man-
agement of Data Warehouses, DMDW2000, Interlaken,
Switzerland.
Agrawal, R., Gupta, A., & Sarawagi, S. (1997). Modelling
multidimensional databases. International Conference
on Data Engineering, ICDE97 (pp. 232-243), Birming-
ham, UK.
Datta, A., & Thomas, H. (1999). The cube data model: A
conceptual model and algebra for on-line analytical
processing in data warehouses. Decision Support Sys-
tems, 27(3), 289-301.
Golfarelli, M., Maio, D., & Rizzi, V.S. (1998). Concep-
tual design of data warehouses from E/R schemes. 32th
Hawaii International Conference on System Sciences,
HICSS1998.
Gyssens, M., & Lakshmanan, V.S. (1997). A foundation
for multi-dimensional databases. Intl Conference on
Very Large Databases (pp. 106-115).
Hurtado, C., & Mendelzon, A. (2001). Reasoning about
summarizability in heterogeneous multidimensional
schemas. International Conference on Database Theory,
ICDT01.
Hsemann, B., Lechtenbrger, J., & Vossen, G. (2000).
Conceptual data warehouse design. Intl Workshop on
Design and Management of Data Warehouse,
DMDW2000, Stockholm, Sweden.
Lehner, W., Albrecht, J., & Wedekind, H. (1998). Multidi-
mensional normal forms. l0th Intl Conference on Scien-
tific and Statistical Data Management, SSDBM98, Capri,
Italy.
Nguyen, T., Tjoa, A.M., Wanger, S. (2000). Conceptual
multidimensional data model based on metacube. Intl
Conference on Advances in Information Systems (pp. 24-
33), Izmir, Turkey.
Pedersen, T.B., & Jensen, C.S. (1999). Multidimensional
data modelling for complex data. Intl Conference on Data
Engineering, ICDE 99.
Pourabbas, E., & Rafanelli, M. (1999). Characterization of
hierarchies and some operators in OLAP environment.
ACM Second International Workshop on Data Ware-
housing and OLAP, DOLAP99 (pp. 54-59), Kansas City,
USA.
Tsois, A., Karayannidis, N., & Sellis, T. (2001). MAC:
Conceptual data modeling for OLAP. Intl Workshop on
527
TEAM LinG

Design and Management of Data Warehouses,
DMDW2000, Interlaken, Switzerland.
Vassiliadis, P., & Skiadopoulos, S. (2000). Modelling and
optimisation issues for multidimensional databases. In-
ternational Conference on Advanced Information Sys-
tems Engineering, CAISE2000 (pp. 482-497), Stockholm,
Sweden.
KEY TERMS
Data Warehouse: A data base which is specifically
elaborated to allow different analysis on data. Analysis
consists generally to make aggregation operations (count,
sum, average, etc.). A data warehouse is different from
a transactional data base since it accumulates data along
time and other dimensions. Data of a warehouse are
loaded and updated at regular intervals from the transac-
tional data bases of the company.
Dimension: Set of members (criteria) allowing to
drive the analysis (example for the Product dimension:
528
A General Model for Data Warehouses
product type, manufacturer type). Members are used to
drive the aggregation operations.
Drilldown: Opposite operation of the previous one.
Fact: Element recorded in a warehouse (example: each
product sold in a shop) and whose characteristics (i.e.
measures) are the object of the analysis (example: quan-
tity of a product sold in a shop).
Galaxy Structure: Structure of a warehouse for which
two different types of facts share a same dimension.
Hierarchy: The members of a dimension are gener-
ally organized along levels into a hierarchy.
Member: Every criterion in a dimension is materi-
alized through a member.
Rollup: Operation consisting in going in a hierarchy
at a more aggregated level.
Star Structure: Structure of a warehouse for which
a fact is directly connected to several dimensions and
can be so analyzed according to these dimensions. It is
the most simple and the most used structure.
TEAM LinG

Genetic Programming
William H. Hsu
Kansas State University, USA
INTRODUCTION
Genetic programming (GP) is a subarea of evolutionary
computation first explored by John Koza (1992) and inde-
pendently developed by Nichael Lynn Cramer (1985). It is
a method for producing computer programs through ad-
aptation according to a user-defined fitness criterion, or
objective function.
GP systems and genetic algorithms (GAs) are related
but distinct approaches to problem solving by simulated
evolution. As in the GA methodology, GP uses a represen-
tation related to some computational model, but in GP,
fitness is tied to task performance by specific program
semantics. Instead of strings or permutations, genetic
programs most commonly are represented as variable-
sized expression trees in imperative or functional pro-
gramming languages, as grammars (ONeill & Ryan, 2001)
or as circuits (Koza et al., 1999). GP uses patterns from
biological evolution to evolve programs:
Crossover: Exchange of genetic material such as
program subtrees or grammatical rules.
Selection: The application of the fitness criterion in
order to choose which individuals from a population
will go on to reproduce.
Replication: The propagation of individuals from
one generation to the next.
Mutation: The structural modification of individuals.
To work effectively, GP requires an appropriate set of
program operators, variables, and constants. Fitness in
GP typically is evaluated over fitness cases. In data
mining, this usually means training and validation data,
but cases also can be generated dynamically using a
simulator or be directly sampled from a real-world prob-
lem-solving environment. GP uses evaluation over these
cases to measure performance over the required task,
according to the given fitness criterion.
This article begins with a survey of the design of GP
systems and their applications to data-mining problems,
such as pattern classification, optimization of representa-
tions for inputs and hypotheses in machine learning,
grammar-based information extraction, and problem trans-
formation by reinforcement learning. It concludes with a
discussion of current issues in GP systems (i.e., scalability,
human-comprehensibility, code growth and reuse, and
incremental learning).
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
529
/
BACKGROUND
Although Cramer (1985) first described the use of cross-
over, selection, mutation, and tree representations for
using genetic algorithms to generate programs, Koza, et
al. (1992) is indisputably the fields most prolific and
influential author (Wikipedia, 2004). In four books, Koza,
et al. (1992, 1994, 1999, 2003) have described GP-based
solutions to numerous toy problems and several impor-
tant real-world problems.
State of the Field: To date, GPs have been applied
successfully to a few significant problems in ma-
chine learning and data mining, most notably sym-
bolic regression and feature construction. The
method is very computationally intensive, how-
ever, and it is still an open question in current
research whether simpler methods can be used in-
stead. These include supervised inductive learning,
deterministic optimization, randomized approxima-
tion using non-evolutionary algorithms (i.e., Markov
chain Monte Carlo approaches), genetic algorithms,
and evolutionary algorithms. It is postulated by GP
researchers that the adaptability of GPs to struc-
tural, functional, and structure-generating solutions
of unknown forms makes them more amenable to
solving complex problems. Specifically, Koza, et al.
(1999, 2003) demonstrate that, in many domains, GP
is capable of human-competitive automated discov-
ery of concepts deemed to be innovative through
technical review such as patent evaluation.
MAIN THRUST
The general strengths of genetic programming lie in its
ability to produce solutions of variable functional form,
reuse partial solutions, solve multi-criterion optimization
problems, and explore a large search space of solutions in
parallel. Modern GP systems also are able to produce
structured, object-oriented, and functional programming
solutions involving recursion or iteration, subtyping, and
higher-order functions.
A more specific advantage of GPs is their ability to
represent procedural, generative solutions to pattern
recognition and machine-learning problems. Examples of
TEAM LinG

this include image compression and reconstruction (Koza,
1992) and several of the recent applications surveyed in
the following.
GP for Pattern Classification
GP in pattern classification departs from traditional super-
vised inductive learning in that it evolves solutions whose
functional form is not determined in advance, and in some
cases can be theoretically arbitrary. Koza (1992, 1994)
developed GPs for several pattern reproduction prob-
lems, such as the multiplexer and symbolic regression
problems.
Since then, there has been continuing work on induc-
tive GP for pattern classification (Kishore et al., 2000),
prediction (Brameier & Banzhaf, 2001), and numerical curve
fitting (Nikolaev & Iba, 2001). GP has been used to boost
performance in learning polynomial functions (Nikolaev &
Iba, 2001). More recent work on tree-based multi-crossover
schemes has produced positive results in GP-based design
of classification functions (Muni et al., 2004).
GP for Control of Inductive Bias,
Feature Construction, and Feature
Extraction
GP approaches to inductive learning face the general
problem of optimizing inductive bias: the preference for
groups of hypotheses over others on bases other than
pure consistency with training data or other fitness cases.
Krawiec (2002) approaches this problem by using GP to
preserve useful components of representation (features)
during an evolutionary run, validating them using the
classification data and reusing them in subsequent gen-
erations. This technique is related to the wrapper ap-
proach to knowledge discovery in databases (KDD),
where validation data is held out and used to select
examples for supervised learning or to construct or select
variables given as input to the learning system. Because
GP is a generative problem-solving approach, feature
construction in GP tends to involve production of new
variable definitions rather than merely selecting a subset.
Evolving dimensionally-correct equations on the ba-
sis of data is another area where GP has been applied.
Keijzer and Babovic (2002) provide a study of how GP
formulates its declarative bias and preferential (search-
based) bias. In this and related work, it is shown that a
proper units of measurement (strong typing) approach
can capture declarative bias toward correct equations,
530
Genetic Programming
whereas type coercion can implement even better prefer-
ential bias.
Grammar-Based GP for Data Mining
Not all GP-based approaches use expression tree-based
representations or functional program interpretation as
the computational model. Wong and Leung (2000) survey
data mining using grammars and formal languages. This
general approach has been shown to be effective for some
natural language learning problems, and extension of the
approach to procedural information extraction is a topic of
current research in the GP community.
GP Software Packages: Functionality
and Research Features
A number of GP software packages are available publicly
and commercially. General features common to most GP
systems for research and development include a very
high-period random number generator, such as the
Mersenne Twister for random constant generation and GP
operations; a variety of selection, crossover, and muta-
tion operations; and trivial parallelism (e.g., through multi-
threading).
One of the most popular packages for experimentation
with GP is Evolutionary Computation in Java, or ECJ (Luke
et al., 2004). ECJ implements the previously discussed
features as well as parsimony, strongly-typed GP, migra-
tion strategies for exchanging individual subpopulations
in island mode GP (a type of GP featuring multiple demes
local populations or breeding groups), vector representa-
tions, and reconfigurability using parameter files.
Other Applications: Optimization,
Policy Learning
Like other genetic and evolutionary computation method-
ologies, GP is driven by fitness and suited to optimization
approaches to machine learning and data mining. Its
program-based representation makes it good for acquir-
ing policies by reinforcement learning.1 Many GP prob-
lems are error-driven or payoff-driven (Koza, 1992), in-
cluding the ant trail problems and foraging problems now
explored more heavily by the swarm intelligence and ant
colony optimization communities. A few problems use
specific information-theoretic criteria, such as maximum
entropy or sequence randomization.
TEAM LinG
Genetic Programming
FUTURE TRENDS
Limitations: Scalability and Solution
Comprehensibility
Genetic programming remains a controversial approach
due to its high computational cost, scalability issues, and
current gaps in fundamental theory for relating its perfor-
mance to traditional search methods, such as hill climbing.
While GP has achieved results in design, optimization, and
intelligent control that are as good as and sometimes better
than those produced by human engineers, it is not yet
widely used as a technique due to these limitations in
theory. An additional controversy, stemming from the
intelligent systems community, is the role of knowledge in
search-driven approaches such as GP. Some proponents
of GP view it as a way to generate innovative solutions with
little or no domain knowledge, while critics have expressed
skepticism over original results due to the lower human
comprehensibility of some results. The crux of this debate
is a tradeoff between innovation and originality vs. com-
prehensibility, robustness, and ease of validation. Suc-
cesses in replicating previously patented engineering
designs such as analog circuits using GP (Koza et al., 2003)
have increased its credibility in this regard.
Open Issues: Code Growth, Diversity,
Reuse, and Incremental Learning
Some of the most important open problems in GP deal with
the proliferation of solution code (called code growth or
code bloat), the reuse of previously evolved partial solu-
tions, and incremental learning. Code growth is an increase
in solution size across generations and generally refers to
one that is not matched by a proportionate increase in
fitness. It has been studied extensively in the field of GP
by many researchers. Luke (2000) provides a survey of
known and hypothesized causes of code growth, along
with methods for monitoring and controlling growth. Re-
cently, Burke, et al. (2004) explored the relationship be-
tween diversity (variation among solutions) and code
growth and fitness. Some techniques for controlling code
growth include reuse of partial solutions through such
mechanisms as automatically defined functions, or ADFs
(Koza, 1994) and incremental learning; that is, learning in
stages. One incremental approach in GP is to specify
criteria for a simplified problem and then transfer the
solutions to a new GP population (Hsu & Gustafson, 2002).
CONCLUSION
/
Genetic programming (GP) is a search methodology that
provides a flexible and complete mechanism for machine
learning, automated discovery, and cost-driven optimi-
zation. It has been shown to work well in many optimiza-
tion and policy-learning problems, but scaling GP up to
most real-world data mining domains is a challenge, due
to its high computational complexity. More often, GP is
used to evolve data transformations by constructing
features or to control the declarative and preferential
inductive bias of the machine-learning component. Mak-
ing GP practical poses several key questions dealing with
how to scale up, make solutions comprehensible to
humans and statistically validate them, control the growth
of solutions, reuse partial solutions efficiently, and learn
incrementally.
Looking ahead to future opportunities and challenges
in data mining, genetic programming provides one of the
more general frameworks for machine learning and adap-
tive problem solving. In data mining, they are likely to be
most useful where a generative or procedural solution is
desired, or where the exact functional form of the solution
(whether a mathematical formula, grammar, or circuit) is
not known in advance.
REFERENCES
Brameier, M., & Banzhaf, W. (2001). Evolving teams of
predictors with linear genetic programming. Genetic
Programming and Evolvable Machines, 2(4), 381-407.
Burke, E.K., Gustafson, S., & Kendall, G. (2004). Diversity
in genetic programming: An analysis of measures and
correlation with fitness. IEEE Transactions on Evolu-
tionary Computation, 8(1), 47-62.
Cramer, N.L. (1985). A representation for the adaptive
generation of simple sequential programs. Proceedings
of the International Conference on Genetic Algorithms
and Their Applications.
Hsu, W.H., & Gustafson, S.M. (2002). Genetic program-
ming and multi-agent layered learning by reinforcements.
Proceedings of the Genetic and Evolutionary Computa-
tion Conference (GECCO-2002), New York.
Keijzer, M., & Babovic, V. (2002). Declarative and prefer-
ential bias in GP-based scientific discovery. Genetic
Programming and Evolvable Machines, 3(1), 41-79.
531
TEAM LinG

Kishore, J.K., Patnaik, L.M., Mani, V., & Agrawal, V.K.
(2000). Application of genetic programming for
multicategory pattern classification. IEEE Transactions
on Evolutionary Computation, 4(3), 242-258.
Koza, J.R. (1992). Genetic programming: On the program-
ming of computers by means of natural selection. Cam-
bridge, MA: MIT Press.
Koza, J.R. (1994). Genetic programming II: Automatic
discovery of reusable programs. Cambridge, MA: MIT
Press.
Koza, J.R. et al. (2003). Genetic programming IV: Routine
human-competitive machine intelligence. San Mateo,
CA: Morgan Kaufmann.
Koza, J.R., Bennett III, F.H., Andr, D., & Keane, M.A.
(1999). Genetic programming III: Darwinian invention
and problem solving. San Mateo, CA: Morgan Kaufmann.
Krawiec, K. (2002). Genetic programming-based construc-
tion of features for machine learning and knowledge
discovery tasks. Genetic Programming and Evolvable
Machines, 3(4), 329-343.
Kushchu, I. (2002). Genetic programming and evolution-
ary generalization. IEEE Transactions on Evolutionary
Computation, 6(5), 431-442.
Luke, S. (2000). Issues in scaling genetic programming:
Breeding strategies, tree generation, and code bloat [doc-
toral thesis]. College Park, MD: University of Maryland.
Luke, S. et al. (2004). Evolutionary computation in Java
v11. Retrieved from http://www.cs.umd.edu/projects/plus/
ec/ecj/
Muni, D.P., Pal, N.R., & Das, J. (2004). A novel approach to
design classifiers using genetic programming. IEEE Trans-
actions on Evolutionary Computation, 8(2), 183-196.
Nikolaev, N.Y., & Iba, H. (2001a). Regularization approach
to inductive genetic programming. IEEE Transactions on
Evolutionary Computation, 5(4), 359-375.
Nikolaev, N.Y., & Iba, H. (2001b). Accelerated genetic
programming of polynomials. Genetic Programming and
Evolvable Machines, 2(3), 231-257.
ONeill, M., & Ryan, C. (2001). Grammatical evolution.
IEEE Transactions on Evolutionary Computation.
Wikipedia. (2004). Genetic programming. Retrieved from
http://en.wikipedia.org/wiki/Genetic_programming
Wong, M.L., & Leung, K.S. (2000). Data mining using
grammar based genetic programming and applications.
Norwell, MA: Kluwer.
532
Genetic Programming
KEY TERMS
Automatically-Defined Function (ADF): Parametric
functions that are learned and assigned names for reuse
as subroutines. ADFs are related to the concept of macro-
operators or macros in speedup learning.
Code Growth (Code Bloat): The proliferation of solu-
tion elements (e.g., nodes in a tree-based GP representa-
tion) that do not contribute toward the objective function.
Crossover: In biology, a process of sexual recombina-
tion, by which two chromosomes are paired up and ex-
change some portion of their genetic sequence. Cross-
over in GP is highly stylized and involves structural
exchange, typically using subexpressions (subtrees) or
production rules in a grammar.
Evolutionary Computation: A solution approach based
on simulation models of natural selection, which begins
with a set of potential solutions and then iteratively
applies algorithms to generate new candidates and select
the fittest from this set. The process leads toward a model
that has a high proportion of fit individuals.
Generation: The basic unit of progress in genetic and
evolutionary computation, a step in which selection is
applied over a population. Usually, crossover and muta-
tion are applied once per generation, in strict order.
Individual: A single candidate solution in genetic and
evolutionary computation, typically represented using
strings (often of fixed length) and permutations in genetic
algorithms, or using problem-solver representations (i.e.,
programs, generative grammars, or circuits) in genetic
programming.
Island Mode GP: A type of parallel GP, where multiple
subpopulations (demes) are maintained and evolve inde-
pendently, except during scheduled exchanges of indi-
viduals.
Mutation: In biology, a permanent, heritable change
to the genetic material of an organism. Mutation in GP
involves structural modifications to the elements of a
candidate solution. These include changes, insertion,
duplication, or deletion of elements (subexpressions,
parameters passed to a function, components of a resis-
tor-capacitor-inducer circuit, non-terminals on the right-
hand side of a production rule).
Parsimony: An approach in genetic and evolutionary
computation related to minimum description length, which
rewards compact representations by imposing a penalty
for individuals in direct proportion to their size (e.g.,
number of nodes in a GP tree). The rationale for parsimony
is that it promotes generalization in supervised inductive
TEAM LinG
Genetic Programming
learning and produces solutions with less code, which
can be more efficient to apply.
Selection: In biology, a mechanism by which the
fittest individuals survive to reproduce, and the basis of
speciation, according to the Darwinian theory of evolu-
tion. Selection in GP involves evaluation of a quantitative
criterion over a finite set of fitness cases, with the com-
bined evaluation measures being compared in order to
choose individuals.
ENDNOTE
/
1
Reinforcement learning describes a class of prob-
lems in machine learning, in which an agent explores
an environment, selecting actions based upon per-
ceived state and an internal policy. This policy is
updated, based upon a (positive or negative) reward
that it receives from the environment as a result of
its actions and that it seeks to maximize in the
expected case.
533
TEAM LinG
534
Graph Transformations and Neural Networks
Ingrid Fischer
Friedrich-Alexander University Erlangen-Nrnberg, Germany
INTRODUCTION
As the beginning of the area of artificial neural networks,
the introduction of the artificial neuron of McCulloch and
Pitts is considered. They were inspired by the biological
neuron. Since then, many new networks or new algorithms
for neural networks have been invented with the result
that this area is not very clearly laid out. In most textbooks
on (artificial) neural networks (Rojas, 2000; Silipo, 2002),
there is no general definition on what a neural net is, but
rather an example-based introduction leading from the
biological model to some artificial successors. Perhaps
the most promising approach to define a neural network
is to see it as a network of many simple processors (units),
each possibly having a small amount of local memory. The
units are connected by communication channels (connec-
tions) that usually carry numeric (as opposed to sym-
bolic) data called the weight of the connection. The units
operate only on their local data and on the inputs they
receive via the connections. It is typical of neural net-
works that they have great potential for parallelism, since
the computations of the components are largely indepen-
dent of each other. Neural networks work best if the
system modeled by them has a high tolerance to error.
Therefore, one would not be advised to use a neural
network to balance ones checkbook. However, they work
very well for:
capturing associations or discovering regularities
within a set of patterns;
any application where the number of variables or
diversity of the data is very great;
any application where the relationships between
variables are vaguely understood; or,
any application where the relationships are difficult
to describe adequately with conventional ap-
proaches.
Neural networks are not programmed but can be trained
in different ways. In supervised learning, examples are
presented to an initialized net. From the input and the
output of these examples, the neural net learns somehow.
There are as many learning algorithms as there are types
of neural nets. Also, learning is motivated physiologi-
cally. When an example is presented to a neural network
that it cannot recalculate, several different steps are
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
possible: changes can be done for a neuron, for the
connections weight or new connections, and neurons
can be inserted. For unsupervised learning, the results of
an input are not known.
There are many advantages and limitations to neural
network analysis, and to discuss this subject properly,
one must look at each individual type of network. Never-
theless, there is one specific limitation of neural networks
that potential users should be aware of. Neural networks
are more or less, depending on the different types, the
ultimate black boxes. The final result of the learning
process is a trained network that provides no equations
or coefficients defining a relationship beyond its own
internal mathematics.
Graphs are widely-used concepts within computer
science; in nearly every field, graphs serve as a tool for
visualization, summarization of dependencies, explana-
tion of connections, and so forth. Famous examples are all
kinds of different nets and graphs as semantic nets, petri
nets, flow charts, interaction diagrams, or neural net-
works. Invented first 35 years ago, graph transformations
have been expanding constantly. Wherever graphs are
used, graph transformations also are applied (Blostein &
Schrr, 1999; Ehrig et al., 1999a; Ehrig et al., 1999b;
Rozenberg, 1997).
Graph transformations are a very promising method
for modeling and programming neural networks. The
graph part is automatically given, as the name neural
network already indicates. Having graph transformations
as methodology, it is easy to model algorithms on this
graph structure. Structure-preserving and structure-
changing algorithms can be modeled equally well. This is
not the case for the widely used matrices programmed
mostly in C or C++. In these approaches, modeling
structure change becomes more difficult.
This leads directly to a second advantage. Graph
transformations have proven useful for visualizing the
network and its algorithms. Most modern neural network
simulators have some kind of visualization tool. Graph
transformations offer a basis for this visualization, as the
algorithms are already implemented in visual rules. Also,
in nearly all books, neural networks are visualized as
graphs.
When having a formal methodology at hand, it is also
possible to use it for proving properties of nets and
algorithms. Especially in this area, earlier results for graph
TEAM LinG
Graph Transformations and Neural Networks
transformation systems can be used. Three possibilities
are especially promising: first, it is interesting whether an
algorithm is terminating. Though this question is unde-
cidable in the general case, the formal methods of graph
rewriting and general rewriting offer some chances to
prove termination for neural network algorithms. The
same holds for the question whether the result produced
by an algorithm is useful, whether the learning of a neural
network was successful. Then it helps to prove whether
two algorithms are equivalent. Finally, possible parallel-
ism in algorithms can be detected and described, based on
results for graph transformation systems.
BACKGROUND
A Short Introduction to Graph
Transformations
Despite the different approaches to handling graph trans-
formations, there are some properties that all approaches
have in common. When transforming a graph G somehow,
it is necessary to specify what part of the graph, what
subgraph L, has to be exchanged. For this subgraph, a new
graph R must be inserted. When applying such a rule to
a graph G, three steps are necessary:
Choose an occurrence of L in G.
Delete L from G.
Insert R into the remainder of G.
Figure 1. The application of a graph rewrite rule L R to a graph G
L 1: x
2: y z 3:
G 1: 1 H
2: 2 3
3:
In Figure 1, a sample application of a graph transfor-
mation rule is shown. The left-hand side L consists of /
three nodes (1:, 2:, 3:) and three edges. This graph is
embedded into a graph G. Numbers in G indicate how the
nodes of L are matched. The embedding of edges is
straightforward. In the next step L is deleted from G, and
R is inserted. If L is simply deleted from G, hanging edges
remain. All edges ending/starting at 1:,2:,3: are missing
one node after deletion. With the help of numbers 1:,2:,3:
in the right-hand side R, it is indicated how these hanging
edges are attached to R inserted in G/L. The resulting
graph is H.
Simple graphs are not enough for modeling real-world
applications. Among the different extensions, two are of
special interest. First, graphs and graph rules can be
labeled. When G is labeled with numbers, L is labeled with
variables, and R is labeled with terms over Ls variables.
This way, calculations can be modeled. Taking our ex-
ample and extending G with numbers 1,2,3, the left-hand
side L with variables x,y,z and the right-hand side with
terms x+y, x-y, xy, x y is shown in Figure 1. When L is
embedded in G, the variables are set to the numbers of the
corresponding nodes. The nodes in H are labeled with the
result of the terms in R when the variable settings resulting
from the embedding of L in G are used.
Also, application conditions can be added, restricting
the application of a rule. For example, the existence of a
certain subgraph A in G can be allowed or forbidden. A rule
only can be applied if A can be found resp. not found in
G. Additionally, label-based application conditions are
possible. This rule could be extended by asking for x < y.
Only in this case would the rule be applied.
R 1: x+y x-y
2: xy xy 3:
1: 3 -1
2: 2 1 3:
535
TEAM LinG

Combining Neural Networks and Graph
Transformations
Various proposals exist as to how graph transformations
and neural networks can be combined having different
goals in mind. Several ideas originate from evolutionary
computing (Curran & ORiodan, 2002; De Jong & Pollack,
2001; Siddiqi & Lucas, 1998); others stem from electrical
engineering (Wan & Beaufays, 1998). The approach of
Fischer (2000) has its roots in graph transformations itself.
Despite these sources, only three really different ideas can
be found:
In most of the papers (Curran & ORiodan, 2002; De
Jong & Pollack, 2001; Siddiqi & Lucas 1998), some
basic graph operators like insert-node, delete-node,
change-attribute, and so forth are used. This set of
operators differs in the approaches. In addition to
these operators, some kind of application condition
exists, stating which rule has to be applied when and
on which nodes or edges. These application condi-
tions can be directed acyclic graphs giving the se-
quence of the rule applications. It also can be tree-
based, where newly created nodes are handed to
different paths in the tree. The main application area
of these approaches is to grow neural networks from
just one node. In this field, other grammar-based
approaches also can be found (Cantu-Paz & Kamath,
2002), where matrices are rewritten (Browse, Hussain
& Smilie, 1999), taking attributed grammars. In
Tsakonas and Dounias (2002), feed-forward neural
networks are grown with the help of a grammar in
Backus-Naur-Form.
In Wan and Beaufays (1998), signal flow graphs
known from electrical engineering are used to model
the information flow through the net. With the help
of rewrite rules, the elements can be reversed, so that
the signal flow is going in the opposite direction as
Figure 2. A probabilistic neural network seen as graph.
Please note that not all edges are labeled, due to space
reasons
8 5
9 7
0.4 0.7 0.3 0.2
4 1
2 10 4 8
3 5
1 7 9 2
536
Graph Transformations and Neural Networks
before. This way, gradient descent-based training
methods as back propagation, a famous training
algorithm (Rojas, 2000) can be derived. A disad-
vantage of this method is that no topology chang-
ing algorithms can be modeled.
In Fischer (2000), arbitrary neural nets are modeled
as graphs and transformed by arbitrary transforma-
tion rules. Because this is the most general ap-
proach, it will be explained in detail in the following
sections.
MAIN THRUST
In the remainder of this article, one special sort of neural
network, the so-called probabilistic neural networks,
together with training algorithms, are explained in detail.
Probabilistic Neural Networks
The main purpose of a probabilistic neural network is to
sort patterns into classes. It always has three layersan
input layer, a hidden layer, and an output layer. The input
neurons are connected to each neuron in the hidden
layer. The neurons of the hidden layer are connected to
one output neuron each. Hidden neurons are modeled
with two nodes, as they have an input value and do some
calculations on it, resulting in an output value. Neurons
and connections are labeled with values resp. weights. In
Figure 2, a probabilistic neural network is shown.
Calculations in Probabilistic Neural
Networks
First, input is presented to the input neurons. The next
step is to calculate the input value of the hidden neurons.
The main purpose of the hidden neurons is to represent
examples of classes. Each neuron represents one ex-
ample. This example forms the weights from the input
neurons to this special hidden neuron. When an input is
presented to the net, each neuron in the hidden layer
computes the probability that it is the example it models.
Therefore, first, the Euclidean distance between the
activation of the input neurons and the weight of the
connection to the hidden layers neuron is computed.
The result coming via the connections is summed up to
within a hidden neuron. This is the distance that the
current input has from the example modeled by the
neuron. If the exact example is inserted into the net, the
result is 0. In Figure 3, this calculation is shown in detail.
The given graph rewrite rule can be applied to the net
shown in Figure 2. The labels i are variables modeling the
input values of the input neurons, w models the weights
TEAM LinG
Graph Transformations and Neural Networks
Figure 3. Calculating the input activation of a neuron
L R
in
w1 wn
i1 in
Figure 4. A new neuron is inserted into a net. This
example is taken from Dynamic Decay Adjustment (Silipo,
2002)
1 1 1 activation is equal to or higher than a given threshold +,
d+
1 d
0 neurons are inserted.
i1 i2 i3 i1 i2 i3
of the connections. In the right-hand side, R a formula is
given to calculate the input activation.
Then, a Gaussian function is used to calculate the
probability that the inserted pattern is the example pattern
of the neuron. The radius s of the Gaussian function has
to be chosen during the training of the net, or it has to be
adapted by the user. The result is the output activation of
the neuron.
The main purpose of the connection between hidden
and output layer is to sum up the activations of the hidden
layers neurons for one class. These values are multiplied
with a weight associated to the corresponding connec-
tion. The output neuron with the highest activation rep-
resents the class of the input pattern.
Training Neural Networks
For the probabilistic neural networks, different methods
are used. The classical training method is the following:
The complete net is built up by inserting one hidden
neuron for each training pattern. As weights of the con-
nections from the input neuron to the hidden neuron
representing this example, the training pattern itself is
taken. The connections from the hidden layer to the
output layer are weighted with 1/m if there are m training
patterns for one class. s, the radius of the Gaussian
function used, is usually simply set to the average dis-
tance between centers of Gaussians modeling training
patterns.
/
||ij - wj||
w1 wn
i1 in
A more sophisticated training method is called Dy-
namic Decay Adjustment (Silipo, 2002). The algorithm
also starts with no neuron in the hidden layer. If a pattern
is presented to the net, first, the activation of all existing
hidden neurons is calculated. If there is a neuron whose
this neuron covers the input pattern. If this is not the case,
a new hidden neuron is inserted into the net, as shown in
Figure 4. With the help of this algorithm, fewer hidden
FUTURE TRENDS
The area of graph grammars and graph transformation
systems several programming environments, especially
AGG and Progress (Ehrig, Engels, Kreowski, & Rozenberg,
1999) have been developed. They can be used to obtain
new implementations of neural network algorithms, espe-
cially of algorithms that change the nets topology.
Pruning means the deletion of neurons and connec-
tions between neurons in already trained neural networks
to minimize their size. Several mechanisms exist that mainly
differ in the evaluation function determining the neurons/
connections to be deleted. It is an interesting idea to
model pruning with graph transformation rules.
A factor graph is a bipartite graph that expresses how
a global function of many variables factors into a product
of local functions. It subsumes other graphical means of
artificial intelligence as Bayesian networks or Markov
random fields. It would be worth checking whether graph
transformation systems are helpful in formulating algo-
rithms for these soft computing approaches. Also, for
more biological applications, the use of graph transforma-
tion is interesting.
CONCLUSION
Modeling neural networks as graphs and algorithms on
neural networks as graph transformations is an easy-to-
use, straightforward method. It has several advantages.
First, the structure of neural nets is supported. When
modeling the net topology and topology changing algo-
537
TEAM LinG

rithms, graph transformation systems can present their
full power. This might be of special interest for educa-
tional purposes where it is useful to visualize step-by-
step what algorithms do. Finally, the theoretical back-
ground of graph transformation and rewriting systems
offers several possibilities for proving termination, equiva-
lence, and the like, of algorithms.
REFERENCES
Blostein, D., & Schrr, A. (1999). Computing with graphs
and graph transformation. Software Practice and Experi-
ence, 29(3), 1-21.
Browse, R.A., Hussain, T.S., & Smillie, M.B. (1999). Using
attribute grammars for the genetic selection of
backpropagation networks for character recognition. Pro-
ceedings of Applications of Artificial Neural Networks
in Image Processing IV. San Jose, California.
Cantu-Paz, E., & Kamath, C. (2002). Evolving neural net-
works for the classification of galaxies. Proceedings of
the Genetic and Evolutionary Computation Conference.
Curran, D., & ORiordan, C. (2002). Applying evolution-
ary computation to designing neural networks: A study
of the state of the art [technical report]. Galway, Ireland:
Department of Information Technology, National Univer-
sity of Ireland.
De Jong, E., & Pollack, J. (2001). Utilizing bias to evolve
recurrent neural networks. Proceedings of the Interna-
tional Joint Conference on Neural Networks, Washing-
ton, D.C.
Ehrig, H., Engels, G., Kreowski, H.-J., & Rozenberg, G.
(Eds.). (1999a). Handbook on graph grammars and
computing by graph transformation. Singapore: World
Scientific.
Ehrig, H., Kreowski, H.-J., Montanari, U., & Rozenberg, G.
(Eds.). (1999b). Handbook on graph grammars and
computing by graph transformation. Singapore: World
Scientific.
Fischer, I. (2000). Describing neural networks with graph
transformations [doctoral thesis]. Nuremberg: Friedrich-
Alexander University Erlangen-Nuremberg.
Klop, J.W., De Vrijer, R.C., & Bezem, M. (2003). Term
rewriting systems. Cambridge, MA: Cambridge Univer-
sity Press.
Nipkow, T., & Baader, F. (1999). Term rewriting and all
that. Cambridge, MA: Cambridge University Press.
538
Graph Transformations and Neural Networks
Rojas, R. (2000). Neural networks. A systematic introduc-
tion. New York, NY: Springer Verlag.
Rozenberg, G. (Ed.). (1997). Handbook of graph gram-
mars and computing by graph transformations. Singapore:
World Scientific.
Siddiqi, A., & Lucas, S.M. (1998). A comparison of matrix
rewriting versus direct encoding for evolving neural net-
works. Proceedings of the IEEE International Confer-
ence on Evolutionary Computation, Anchorage, Alaska.
Silipo, R. (2002). Artificial neural networks. In M. Berthold,
& D. Hand (Eds.), Intelligent data analysis (pp. 269-319).
New York, NY: Springer Verlag.
Tsakonas, A., & Dounias, D. (2002). A scheme for the
evolution of feedforward neural Networks using BNF-
grammar driven genetic programming. Proceedings of the
EUNITEEUropean Network on Intelligent Technolo-
gies for Smart Adaptive Systems, Algarve, Portugal.
Wan, E., & Beaufays, F. (1998). Diagrammatic methods for
deriving and relating temporal neural network algorithms.
In C. Giles, & M. Gori (Eds.), Adaptive processing of
sequences and data structures (pp. 63-98). Salerno, Italy:
International Summer School on Neural Networks.
KEY TERMS
Confluence: A rewrite system is confluent, if no mat-
ter in which order rules are applied, they lead to the same
result.
Graph: A graph consists of vertices and edges. Each
edge is connected to a source node and a target node.
Vertices and edges can be labeled with numbers and
symbols.
Graph Production: Similar to productions in general
Chomsky grammars, a graph production consists of a left-
hand side and a right-hand side. The left-hand side is
embedded in a host graph. Then, it is removed, and in the
resulting hole, the right-hand side of the graph produc-
tion is inserted. To specify how this right-hand side is
attached into this hole, how edges are connected to the
new nodes, some additional information is necessary.
Different approaches exist how to handle this problem.
Graph Rewriting: The application of a graph produc-
tion to a graph is also called graph rewriting.
Neural Networks: Learning systems, designed by
analogy with a simplified model of the neural connections
in the brain, which can be trained to find nonlinear rela-
tionships in data. Several neurons are connected to form
the neural networks.
TEAM LinG
Graph Transformations and Neural Networks

Neuron: The smallest processing unit in a neural
network.
Probabilistic Neural Network: One of the many dif-
ferent kinds of neural networks with the application area
to classify input data into different classes.
Rewrite System: Consists of a set of configurations
and a relation x y denoting that the configuration x
follows the configuration y with the help of a rule appli-
cation. /
Termination: A rewrite system terminates if it has no
infinite chain.
Weight: Connections between neurons of neural net-
works have a weight. This weight can be changed during
the training of the net.

539

TEAM LinG
540
Graph-Based Data Mining
Lawrence B. Holder
University of Texas at Arlington, USA
Diane J. Cook
University of Texas at Arlington, USA
INTRODUCTION
Graph-based data mining represents a collection of
techniques for mining the relational aspects of data
represented as a graph. Two major approaches to graph-
based data mining are frequent subgraph mining and
graph-based relational learning. This article will fo-
cus on one particular approach embodied in the Subdue
system, along with recent advances in graph-based su-
pervised learning, graph-based hierarchical conceptual
clustering, and graph-grammar induction.
Most approaches to data mining look for associa-
tions among an entitys attributes, but relationships
between entities represent a rich source of information,
and ultimately knowledge. The field of multi-relational
data mining, of which graph-based data mining is a part,
is a new area investigating approaches to mining this
relational information by finding associations involving
multiple tables in a relational database. Two main ap-
proaches have been developed for mining relational
information: logic-based approaches and graph-based
approaches.
Logic-based approaches fall under the area of induc-
tive logic programming (ILP). ILP embodies a number
of techniques for inducing a logical theory to describe
the data, and many techniques have been adapted to
multi-relational data mining (Dzeroski & Lavrac, 2001;
Dzeroski, 2003). Graph-based approaches differ from
logic-based approaches to relational mining in several
ways, the most obvious of which is the underlying rep-
resentation. Furthermore, logic-based approaches rely
on the prior identification of the predicate or predicates
to be mined, while graph-based approaches are more
data-driven, identifying any portion of the graph that has
high support. However, logic-based approaches allow
the expression of more complicated patterns involving,
for example, recursion, variables, and constraints among
variables. These representational limitations of graphs
can be overcome, but at a computational cost.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
Graph-based data mining (GDM) is the task of finding
novel, useful, and understandable graph-theoretic pat-
terns in a graph representation of data. Several ap-
proaches to GDM exist based on the task of identifying
frequently occurring subgraphs in graph transactions,
that is, those subgraphs meeting a minimum level of
support. Washio & Motoda (2003) provide an excellent
survey of these approaches. We here describe four
representative GDM methods.
Kuramochi and Karypis (2001) developed the FSG
system for finding all frequent subgraphs in large graph
databases. FSG starts by finding all frequent single and
double edge subgraphs. Then, in each iteration, it gener-
ates candidate subgraphs by expanding the subgraphs
found in the previous iteration by one edge. In each
iteration the algorithm checks how many times the
candidate subgraph occurs within an entire graph. The
candidates, whose frequency is below a user-defined
level, are pruned. The algorithm returns all subgraphs
occurring more frequently than the given level.
Yan and Han (2002) introduced gSpan, which com-
bines depth-first search and lexicographic ordering to
find frequent subgraphs. Their algorithm starts from all
frequent one-edge graphs. The labels on these edges
together with labels on incident vertices define a code
for every such graph. Expansion of these one-edge
graphs maps them to longer codes. Since every graph
can map to many codes, all but the smallest code are
pruned. Code ordering and pruning reduces the cost of
matching frequent subgraphs in gSpan. Yan & Han (2003)
describe a refinement to gSpan, called CloseGraph,
which identifies only subgraphs satisfying the minimum
support, such that no supergraph exists with the same
level of support.
Inokuchi et al. (2003) developed the Apriori-based
Graph Mining (AGM) system, which searches the space
of frequent subgraphs in a bottom-up fashion, beginning
TEAM LinG
Graph-Based Data Mining
with a single vertex, and then continually expanding by a
single vertex and one or more edges. AGM also employs
a canonical coding of graphs in order to support fast
subgraph matching. AGM returns association rules sat-
isfying user-specified levels of support and confidence.
The last approach to GDM, and the one discussed in
the remainder of this chapter, is embodied in the Subdue
system (Cook & Holder, 2000). Unlike the above sys-
tems, Subdue seeks a subgraph pattern that not only
occurs frequently in the input graph, but also signifi-
cantly compresses the input graph when each instance of
the pattern is replaced by a single vertex. Subdue per-
forms a greedy search through the space of subgraphs,
beginning with a single vertex and expanding by one
edge. Subdue returns the pattern that maximally com-
presses the input graph. Holder & Cook (2003) describe
current and future directions in this graph-based rela-
tional learning variant of GDM.
MAIN THRUST
As a representative of GDM methods, this section will
focus on the Subdue graph-based data mining system.
The input data is a directed graph with labels on vertices
and edges. Subdue searches for a substructure that best
compresses the input graph. A substructure consists of
a subgraph definition and all its occurrences throughout
the graph. The initial state of the search is the set of
substructures consisting of all uniquely labeled verti-
ces. The only operator of the search is the Extend
Substructure operator. As its name suggests, it extends
a substructure in all possible ways by a single edge and
a vertex, or by only a single edge if both vertices are
already in the subgraph.
Subdues search is guided by the minimum descrip-
tion length (MDL) principle, which seeks to minimize
the description length of the entire data set. The evalu-
ation heuristic based on the MDL principle assumes that
the best substructure is the one that minimizes the
description length of the input graph when compressed
by the substructure. The description length of the sub-
structure S given the input graph G is calculated as
DL(G,S) = DL(S)+DL(G|S), where DL(S) is the descrip-
tion length of the substructure, and DL(G|S) is the
description length of the input graph compressed by the
substructure. Subdue seeks a substructure S that mini-
mizes DL(G,S).
The search progresses by applying the Extend Sub-
structure operator to each substructure in the current
state. The resulting state, however, does not contain all
the substructures generated by the Extend Substructure
operator. The substructures are kept on a queue and are
ordered based on their description length (or some-
times referred to as value) as calculated using the MDL
principle. The queues length is bounded by a user- /
defined constant.
The search terminates upon reaching a user-speci-
fied limit on the number of substructures extended, or
upon exhaustion of the search space. Once the search
terminates and Subdue returns the list of best substruc-
tures found, the graph can be compressed using the best
substructure. The compression procedure replaces all
instances of the substructure in the input graph by single
vertices, which represent the substructures instances.
Incoming and outgoing edges to and from the replaced
instances will point to, or originate from the new vertex
that represents the instance. The Subdue algorithm can
be invoked again on this compressed graph.
Figure 1 illustrates the GDM process on a simple
example. Subdue discovers substructure S1, which is
used to compress the data. Subdue can then run for a
second iteration on the compressed graph, discovering
substructure S2. Because instances of a substructure can
appear in slightly different forms throughout the data, an
inexact graph match, based on graph edit distance, is
used to identify substructure instances.
Most GDM methods follow a similar process. Varia-
tions involve different heuristics (e.g., frequency vs.
MDL) and different search operators (e.g., merge vs.
extend).
Graph-Based Hierarchical Conceptual
Clustering
Given the ability to find a prevalent subgraph pattern in
a larger graph and then compress the graph with this
pattern, iterating over this process until the graph can no
longer be compressed will produce a hierarchical, con-
ceptual clustering of the input data. On the i th iteration,
the best subgraph Si is used to compress the input graph,
introducing new vertices labeled Si in the graph input to
the next iteration. Therefore, any subsequently-discov-
ered subgraph Sj can be defined in terms of one or more
of Sis, where i < j. The result is a lattice, where each
cluster can be defined in terms of more than one parent
Figure 1. Graph-based data mining: A simple example
S2
S1 S1
S1 S1 S1
S2 S2
541
TEAM LinG

subgraph. For example, shows such a clustering done on
a DNA molecule. Note that the ordering of pattern dis-
covery can affect the parents of a pattern. For instance,
the lower-left pattern in could have used the C-C-O
pattern, rather than the C-C pattern, but in fact, the lower-
left pattern is discovered before the C-C-O pattern. For
more information on graph-based clustering, see Jonyer
et al. (2001).
Graph-Based Supervised Learning
Extending a graph-based data mining approach to per-
form supervised learning involves the need to handle
negative examples (focusing on the two-class scenario).
In the case of a graph the negative information can come
in three forms. First, the data may be in the form of
numerous smaller graphs, or graph transactions, each
labeled either positive or negative. Second, data may be
composed of two large graphs: one positive and one
negative. Third, the data may be one large graph in which
the positive and negative labeling occurs throughout. We
will talk about the third scenario in the section on future
directions. The first scenario is closest to the standard
supervised learning problem in that we have a set of
clearly defined examples. Let G + represent the set of
positive graphs, and G- represent the set of negative
graphs. Then, one approach to supervised learning is to
find a subgraph that appears often in the positive graphs,
but not in the negative graphs. This amounts to replacing
the information-theoretic measure with simply an er-
ror-based measure. This approach will lead the search
toward a small subgraph that discriminates well. How-
ever, such a subgraph does not necessarily compress
well, nor represent a characteristic description of the
target concept.
Figure 2. Graph-based hierarchical, conceptual
clustering of a DNA molecule
DNA
CN CC
O == P OH
CC
C
\ \
O
NC
\
O == P OH
C |
O one or more edges, then a recursive production rule
\
C generating an infinite sequence of such connected sub-
CC
/ \
NC
O
/ \
C information-theoretic measure taking into account the
542
Graph-Based Data Mining
We can bias the search toward a more characteristic
description by using the information-theoretic mea-
sure to look for a subgraph that compresses the positive
examples, but not the negative examples. If I(G) repre-
sents the description length (in bits) of the graph G, and
I(G|S) represents the description length of graph G
compressed by subgraph S, then we can look for an S
that minimizes I(G +|S) + I(S) + I(G-) I(G-|S), where the
last two terms represent the portion of the negative
graph incorrectly compressed by the subgraph. This
approach will lead the search toward a larger subgraph
that characterizes the positive examples, but not the
negative examples.
Finally, this process can be iterated in a set-cover-
ing approach to learn a disjunctive hypothesis. If using
the error measure, then any positive example contain-
ing the learned subgraph would be removed from subse-
quent iterations. If using the information-theoretic
measure, then instances of the learned subgraph in both
the positive and negative examples (even multiple in-
stances per example) are compressed to a single ver-
tex. Note that the compression is a lossy one, that is we
do not keep enough information in the compressed
graph to know how the instance was connected to the
rest of the graph. This approach is consistent with our
goal of learning general patterns, rather than mere
compression. For more information on graph-based
supervised learning, see Gonzalez et al. (2002).
Graph Grammar Induction
As mentioned earlier, two of the advantages of logic-
based approach to relational learning are the ability to
learn recursive hypotheses and constraints among vari-
ables. However, there has been much work in the area of
graph grammars, which overcome this limitation. Graph
grammars are similar to string grammars except that
terminals can be arbitrary graphs rather than symbols
from an alphabet. While much of the work on graph
grammars involves the analysis of various classes of
graph grammars, recent research has begun to develop
techniques for learning graph grammars (Doshi et al.,
O
| 2002; Jonyer et al., 2002).
Figure 3b shows an example of a recursive graph
grammar production rule learned from the graph in a.
O A GDM approach can be extended to consider graph
|
grammar productions by analyzing the instances of a
O subgraph to see how they are related to each other. If
|
CH2 two or more instances are connected to each other by
graphs can be constructed. A slight modification to the
extra information needed to describe the recursive
TEAM LinG
Graph-Based Data Mining
component of the production is all that is needed to
allow such a hypothesis to compete along side simple
subgraphs (i.e., terminal productions) for maximizing
compression.
These graph grammar productions can include non-
terminals on the right-hand side. These productions can
be disjunctive, as in c, which represents the final produc-
tion learned from a using this approach. The disjunction
rule is learned by looking for similar, but not identical,
extensions to the instances of a subgraph. A new rule can
be constructed that captures the disjunctive nature of
this extension, and included in the pool of production
rules competing based on their ability to compress the
input graph. With a proper encoding of this disjunction
information, the MDL criterion will tradeoff the com-
plexity of the rule with the amount of compression it
affords in the input graph. An alternative to defining
these disjunction non-terminals is to instead construct
a variable whose range consists of the different disjunc-
tive values of the production. In this way we can intro-
duce constraints among variables contained in a sub-
graph by adding a constraint edge to the subgraph. For
example, if the four instances of the triangle structure in
a each had another edge to a c, d, f and f vertex respec-
tively, then we could propose a new subgraph, where
these two vertices are
represented by variables, and an equality constraint
is introduced between them. If the range of the variable
is numeric, then we can also consider inequality con-
straints between variables and other vertices or vari-
ables in the subgraph pattern.
Jonyer (2003) has developed a graph grammar learn-
ing approach with the above capabilities. The approach
has shown promise both in handling noise and learning
Figure 3. Graph grammar learning example with (a)
the input graph, (b) the first grammar rule learned,
and (c) the second and third grammar rules learned
(a) a a
b c b d b f
k
x y x y r x
z q z q
(b) S1 x y
z q
(c) S2 a S2
b S3
S3 c d
recursive hypotheses in many different domains includ-
ing learning the building blocks of proteins and commu- /
nication chains in organized crime.
FUTURE TRENDS
The field of graph-based relational learning is still
young, but the need for practical algorithms is growing
fast. Therefore, we need to address several challenging
scalability issues, including incremental learning in
dynamic graphs. Another issue regarding practical ap-
plications involves the blurring of positive and negative
examples in a supervised learning task, that is, the graph
has many positive and negative parts, not easily sepa-
rated, and with varying degrees of class membership.
Partitioning and Incremental Mining for
Scalability
Scaling GDM approaches to very large graphs, graphs
too big to fit in main memory, is an ever-growing
challenge. Two approaches to address this challenge are
being investigated. One approach involves partitioning
the graph into smaller graphs that can be processed in a
distributed fashion (Cook et al., 2001). A second ap-
proach involves implementing GDM within a relational
database management system, taking advantage of user-
defined functions and the optimized storage capabilities
of the RDBMS.
A newer issue regarding scalability involves dy-
namic graphs. With the advent of real-time streaming
data, many data mining systems must mine incremen-
tally, rather than off-line from scratch. Many of the
domains we wish to mine in graph form are dynamic
domains. We do not have the time to periodically re-
build graphs of all the data to date and run a GDM system
from scratch. We must develop methods to incremen-
tally update the graph and the patterns currently preva-
a a
lent in the graph. One approach is similar to the graph
b f
partitioning approach for distributed processing. New
data can be stored in an increasing number of partitions.
y x y Information within partitions can be exchanged, or a
z q z q repartitioning can be performed if the information loss
exceeds some threshold. GDM can be used to search the
S1 x y new partitions, suggesting new subgraph patterns as they
z q
evaluate highly in new and old partitions.
Learning with Supervised Graphs
a
b S3
In a highly relational domain the positive and negative
examples of a concept are not easily separated. Such a
f graph is called a supervised graph, in that the graph as a
543
TEAM LinG

whole contains class information, but perhaps not indi-
vidual components of the graph. This scenario presents
a challenge to any data mining system, but especially to
a GDM system, where clearly classified data may be
tightly related to less classified data. Two approaches to
this task are being investigated. The first involves modi-
fying the MDL encoding to take into account the amount
of information necessary to describe the class member-
ship of compressed portions of the graph. The second
approach involves treating the class membership of a
vertex or edge as a cost and weighting the information-
theoretic value of the subgraph patterns by the costs of
the instances of the pattern. The ability to learn from
supervised graphs will allow the user more flexibility in
indicating class membership where known, and to varying
degrees, without having to clearly separate the graph into
disjoint examples.
CONCLUSION
Graph-based data mining (GDM) is a fast-growing field
due to the increasing interest in mining the relational
aspects of data. We have described several approaches
to GDM including logic-based approaches in ILP sys-
tems, graph-based frequent subgraph mining approaches
in AGM, FSG and gSpan, and a graph-based relational
learning approach in Subdue. We described the Subdue
approach in detail along with recent advances in super-
vised learning, clustering, and graph-grammar induc-
tion.
However, much work remains to be done. Because
many of the graph-theoretic operations inherent in GDM
are NP-complete or definitely not in P, scalability is a
constant challenge. With the increased need for mining
streaming data, the development of new methods for
incremental learning from dynamic graphs is important.
Also, the blurring of example boundaries in a supervised
learning scenario gives rise to graphs, where the class
membership of even nearby vertices and edges can vary
considerably. We need to develop better methods for
learning in these supervised graphs.
Finally, we discussed several domains throughout
this paper that benefit from a graphical representation
and the use of GDM to extract novel and useful patterns.
As more and more domains realize the increased predic-
tive power of patterns in relationships between entities,
rather than just attributes of entities, graph-based data
mining will become foundational to our ability to better
understand the ever-increasing amount of data in our
world.
544
Graph-Based Data Mining
REFERENCES
Cook, D., & Holder, L. (2000). Graph-based data mining.
IEEE Intelligent Systems, 15(2), 32-41.
Cook, D., Holder, L., Galal, G., & Maglothin, R. (2001).
Approaches to parallel graph-based knowledge discov-
ery. Journal of Parallel and Distributed Computing,
61(3), 427-446.
Doshi, S., Huang, F., & Oates, T. (2002). Inferring the
structure of graph grammars from data. In Proceedings
of the International Conference on Knowledge-based
Computer Systems.
Deroski, S. (2003). Multi-relational data mining: An intro-
duction. SIGKDD Explorations, 5(1), 1-16.
Deroski, S., & Lavra, N. (2001). Relational data mining.
Berlin: Springer Verlag.
Gonzalez, J., Holder, L., & Cook, D. (2002). Graph-based
relational concept learning. In Proceedings of the Nine-
teenth International Conference on Machine Learning.
Holder, L., & Cook, D. (2003). Graph-based relational
learning: Current and future directions. SIGKDD Ex-
plorations, 5(1), 90-93.
Inokuchi, A., Washio, T., & Motoda, H. (2003). Com-
plete mining of frequent patterns from graphs: Mining
graph data. Machine Learning, 50, 321-254.
Jonyer, I. (2003). Context-free graph grammar induction
using the minimum description length principle. Ph.D.
thesis. Department of Computer Science and Engineering,
University of Texas at Arlington.
Jonyer, I., Cook, D., & Holder, L. (2001). Graph-based
hierarchical conceptual clustering. Journal of Machine
Learning Research, 2, 19-43.
Jonyer, I., Holder, L., & Cook, D. (2002). Concept
formation using graph grammars. In Proceedings of the
KDD Workshop on Multi-Relational Data Mining.
Kuramochi, M., & Karypis, G. (2001). Frequent subgraph
discovery. In Proceedings of the First IEEE Conference
on Data Mining.
Washio, T., & Motoda, H. (2003). State of the art of graph-
based data mining. SIGKDD Explorations, 5(1), 59-68.
Yan, X., & Han, J. (2002). Graph-based substructure pat-
tern mining. In Proceedings of the International Confer-
ence on Data Mining.
Yan, X., & Han, J. (2003). CloseGraph: Mining closed
frequent graph patterns. In Proceedings of the Ninth
TEAM LinG
Graph-Based Data Mining
International Conference on Knowledge Discovery and
Data Mining.
KEY TERMS
Conceptual Graph: Graph representation described
by a precise semantics based on first-order logic.
Dynamic Graph: Graph representing a constantly
changing stream of data.
Frequent Subgraph Mining: Finding all subgraphs
within a set of graph transactions whose frequency
satisfies a user-specified level of minimum support.
Graph-Based Data Mining: Finding novel, useful, and
understandable graph-theoretic patterns in a graph rep-
resentation of data.
Graph Grammar: Grammar describing the construc-
tion of a set of graphs, where terminals and non-terminals /
represent vertices, edges or entire subgraphs.
Inductive Logic Programming: Techniques for learn-
ing a first-order logic theory to describe a set of relational
data.
Minimum Description Length (MDL) Principle: Prin-
ciple stating that the best theory describing a set of data
is the one minimizing the description length of the theory
plus the description length of the data described (or
compressed) by the theory.
Multi-Relational Data Mining: Mining patterns that
involve multiple tables in a relational database.
Supervised Graph: Graph in which each vertex and
edge can belong to multiple categories to varying de-
grees. Such a graph complicates the ability to clearly
define transactions on which to perform data mining.
545
TEAM LinG
546
Group Pattern Discovery Systems for Multiple
Data Sources
Shichao Zhang
University of Technology Sydney, Australia
Chengqi Zhang
University of Technology Sydney, Australia
INTRODUCTION
Multiple data source mining is the process of identifying
potentially useful patterns from different data sources, or
datasets (Zhang et al., 2003). Group pattern discovery
systems for mining different data sources are based on
local pattern-analysis strategy, mainly including logical
systems for information enhancing, a pattern discovery
system, and a post-pattern-analysis system.
BACKGROUND
Many large organizations have multiple data sources,
such as different branches of a multinational company.
Also, as the Web has emerged as a large distributed data
repository, it is easy nowadays to access a multitude of
data sources. Therefore, individuals and organizations
have taken into account the Internets low-cost informa-
tion and knowledge when making decisions (Lesser et al.,
2000). Although the data collected from the Internet
(called external data) brings us opportunities in improving
the quality of decisions, it generates a significant chal-
lenge: efficiently identifying quality knowledge from differ-
ent data sources (Kargupta et al., 2000; Liu et al., 2001;
Prodromidis et al., 1998; Zhong et al., 1999). Potentially,
almost every company must confront the multiple data
source (MDS) problem (Hurson et al., 1994). This problem
is difficult to solve, due to the facts that multiple data source
mining is a procedure of searching for useful patterns in
multidimensional spaces; and putting all data together from
different sources might amass a huge database for central-
ized processing and cause problems, such as data privacy
breaches, data inconsistency, and data conflict.
Recently, the authors have developed local pattern
analysis, a new multi-database mining strategy for dis-
covering some kinds of potentially useful patterns that
cannot be mined in traditional multi-database mining
techniques (Zhang et al., 2003). Local pattern analysis
delivers high-performance pattern discovery from MDSs.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
This effort provides a good insight into knowledge dis-
covery from multiple data sources.
However, there are two fundamental problems that
prevent local pattern analysis from widespread applica-
tion. First, when the data collected from the Internet is of
poor quality, the poor-quality data can disguise useful
patterns. For example, a stock investor might need to
collect information from outside data sources when mak-
ing an investment decision, such as news. If fraudulent
information collected is applied directly to investment
decisions, the investor might lose money. In particular,
much work has been built on consistent data. In the input
to the distributed data mining algorithms, it is assumed
that the data sources do not conflict with each other.
However, reality is much more inconsistent than the ideal;
the inconsistency must be resolved before any mining
algorithms can be applied. These generate a crucial re-
quirement: ontology-based data enhancement.
The second fundamental challenge is the efficiency of
mining algorithms for identifying potentially useful pat-
terns in MDSs. Over the years, there has been a great deal
of work in multiple source data mining (Aounallah et al.,
2004; Krishnaswamy et al., 2000; Li et al., 2001; Yin et al.,
2004). However, traditional multiple data source mining
still utilizes mono-database mining techniques. That is, all
the data from relevant data sources is pooled to amass a
huge dataset for discovery. These algorithms cannot
discover some useful patterns; for example, the pattern
that 70% of the branches within a company agrees that a
married customer usually has at least two cars if his or her
age is between 45 and 65. On the other hand, using our
local pattern analysis, there can be huge amounts of the
local patterns. These generate a strong requirement: the
development of efficient algorithms for identifying useful
patterns in MDSs.
This article introduces a group of pattern discovery
systems for dealing with the MDS problem, mainly (1) a
logical system for enhancing data quality, a logical sys-
tem for resolving conflicts, a data cleaning system, and a
database clustering system, for solving the first problem;
TEAM LinG
Group Pattern Discovery Systems for Multiple Data Sources
and (2) a pattern discovery system and a post-mining
system for solving the second problem.
MAIN THRUST
Group pattern discovery systems are able to (i) effectively
enhance data quality for mining MDSs and (ii) automati-
cally identify potentially useful patterns from the multi-
dimension data in MDSs.
Data Enhancement
Data enhancement includes the following:
1. The data cleaning system mainly includes these
functions: recovering incomplete data (filling the
values missed or expelling ambiguity); purifying
data (consistency of data name, consistency of data
format, correcting errors, or removing outliers); and
resolving data conflicts (using domain knowledge
or expert decision to settle discrepancy).
2. The logical system for enhancing data quality fo-
cuses on the following epistemic properties:
veridicality, introspection, and consistency.
3. The logical system for resolving conflict has the
property of obeying the weighted majority principle
in case of conflicts.
4. The fuzzy database clustering system generates
good database clusters.
Identifying Interesting Patterns
A local pattern may be a frequent itemset, an association
rule, causal rule, dependency, or some other expression.
Local pattern analysis is an in-place strategy specifically
designed for mining MDSs, providing a feasible way to
generate globally interesting models from data in multi-
dimensional spaces.
Based on our local pattern analysis, three key systems
can be developed for automatically searching for poten-
tially useful patterns from local patterns: (a) identifying
high-vote patterns; (b) finding exceptional patterns; and
(c) synthesizing patterns by weighting majority.
(a) Identifying High-Vote Patterns: Within an MDS
environment, each data source, large or small, can
have an equal power to vote for their patterns for the
decision-making of a company. Some patterns can
receive votes from most of the data sources. These
patterns are referred to as high-vote patterns. High-
vote patterns represent the commonness of the
branches. Therefore, these patterns may be far more
important in terms of decision-making within the /
company. The key problem is how to efficiently
search for high-vote patterns of interest in multi-
dimensional spaces. It can be attacked by mining
the distribution of all patterns.
(b) Finding Exceptional Patterns: Like high-vote pat-
terns, exceptional patterns also are regarded as
novel patterns in multiple data sources, which re-
flect the individuality of data sources. While high-
vote patterns are useful when a company is reaching
common decisions, headquarters also are interested in
viewing exceptional patterns used when special deci-
sions are made at only a few of the branches, perhaps
for predicting the sales of a new product. Exceptional
patterns can capture the individuality of branches.
Therefore, although an exceptional pattern receives
votes from only a few branches, it is extremely valuable
information in MDSs. The key problem is how to
construct efficient methods for measuring the inter-
estingness of exceptional patterns.
(c) Searching for Synthesizing Patterns by Weighting
Majority: Although each data source can have an
equal power to vote for patterns for making deci-
sions, data sources may be different in importance to
a company. For example, in a company, if the sale of
branch A is four times that of branch B, branch A is
certainly more important than branch B in the com-
pany. (Here, each branch in a company is viewed as
a data source in an MDS environment.) The decisions
of the company are reasonably partial to high-sale
branches. Also, local patterns may have different
supports. For example, let the supports of patterns X1
and X2 be 0.9 and 0.4 in a branch. Pattern X1 is far more
believable than pattern X2. These two examples
present the importance of branches and patterns for
decision making within a company. Therefore, syn-
thesizing patterns is very useful.
Post Pattern Analysis
In an MDS environment, a pattern (e.g., a high-vote
association rule) is attached to certain factors, including
name, vote, vsupp, and vconf. For a very large set of data
sources, a high-vote association rule may be supported
by a number of data sources. So, the sets of its support and
confidence in these data sources are too large to be
browsed by users, and thus, it is rather difficult to apply
the rule to decision making for users. Therefore, post-
pattern analysis is very important in MDS mining. The key
problem is how to construct effective partition for classi-
fying the patterns mined.
547
TEAM LinG
 Group Pattern Discovery Systems for Multiple Data Sources
FUTURE TRENDS
The data that are distributed in multiple data sources
present many complications, and associated mining tasks
and mining approaches are diverse. Therefore, many chal-
lenging research issues have emerged. Important tasks
that present themselves to multiple data source mining
researchers and multiple data source mining system and
application developers are as follows:
Establishment of a powerful representation for pat-
terns in distributed data;
Ontology-based methodologies and technologies
for data preparation/enhancement;
Development of efficient and effective mining strat-
egies and systems; and
Construction of interactive and integrated multiple
data source mining environments.
CONCLUSION
As stated previously, many companies/organizations have
utilized the Internets low-cost information when making
decisions. Potentially, almost every company must con-
front the MDS problem. Although the data from the Inter-
net assists in improving the quality of decisions, the data
is of poor quality. This leaves a large gap between the
available data and the machinery available to process the
data. It generates a crucial need: efficiently identifying
quality knowledge from MDSs. To support this need,
group pattern discovery systems have been introduced in
the context of real MDS mining systems.
REFERENCES
Aounallah, M. et al. (2004). Distributed data mining vs.
sampling techniques: A comparison. Proceedings of the
Canadian Conference on AI 2004.
Grossman, R., Bailey, S., Ramu, A., Malhi, B., & Turinsky,
A. (2000). The preliminary design of papyrus: A system for
high performance, distributed data mining over clusters.
Proceedings of Advances in Distributed and Parallel
Knowledge Discovery.
Hurson, A., Bright, M., & Pakzad, S. (1994). Multidatabase
systems: An advanced solution for global information
sharing. IEEE Computer Society Press.
Kargupta, H., Huang, W., Sivakumar, K., Park, B., & Wang,
S. (2000). Collective principal component analysis from
distributed, heterogeneous data. Principles of Data Min-
ing and Knowledge Discovery (pp. 452-457).
548
Krishnaswamy, S. et al. (2000). An architecture to sup-
port distributed data mining services in e-commerce
environments. Proceedings of the WECWIS 2000.
Lesser, V. et al. (2000). BIG: An agent for resource-
bounded information gathering and decision making.
Artificial Intelligence, 118(1-2), 197-244.
Li, B. et al. (2001). An architecture for multidatabase
systems based on CORBA and XML. Proceedings of the
12th International Workshop on Database and Expert
Systems Applications.
Liu, H., Lu, H., & Yao, J. (2001). Towards multi-database
mining: Identifying relevant databases. IEEE Transactions
on Knowledge and Data Engineering, 13(4), 541-553.
Prodromidis, A., & Stolfo, S. (1998). Pruning meta-clas-
sifiers in a distributed data mining system. Proceedings
of the First National Conference on New Information
Technologies.
Wu, X., & Zhang, S. (2003). Synthesizing high-frequency
patterns from different data sources. IEEE Transactions
on Knowledge and Data Engineering, 15(2), 353-367.
Yin, X., Han, J., Yang, J., & Yu, P. (2004). CrossMine:
Efficient classification across multiple database rela-
tions. Proceedings of the ICDE 2004.
Zhang, S., Wu, X., & Zhang, C. (2003). Multi-database
mining. IEEE Computational Intelligence Bulletin, 2(1),
5-13.
Zhang, S., Zhang, C., & Wu, X. (2004). Knowledge
discovery in multiple databases. Springer.
Zhong, N., Yao, Y., & Ohsuga, S. (1999). Peculiarity
oriented multi-database mining. Proceedings of PKDD.
KEY TERMS
Database Clustering: The process of grouping simi-
lar databases together.
Exceptional Pattern: A pattern that is strongly sup-
ported (voted for) by only a few of a group data sources.
High-Vote Pattern: A pattern that is supported (voted
for) by most of a group data sources.
Knowledge Discovery in Databases (KDD): Also re-
ferred to as data mining; the extraction of hidden predic-
tive information large databases.
Local Pattern Analysis: An in-place strategy specifi-
cally designed for generating globally interesting mod-
els from local patterns in multi-dimensional spaces.
TEAM LinG
Group Pattern Discovery Systems for Multiple Data Sources

Multi-Database Mining: The mining of potentially

useful patterns in multi-databases. /
Multiple Data Source Mining: The process of identi-
fying potentially useful patterns from different data
sources.

549

TEAM LinG
550
Heterogeneous Gene Data for Classifying
Tumors
Benny Yiu-ming Fung
The Hong Kong Polytechnic University, Hong Kong
Vincent To-yee Ng
The Hong Kong Polytechnic University, Hong Kong
INTRODUCTION
When classifying tumors using gene expression data,
mining tasks commonly make use of only a single data set.
However, classification models based on patterns ex-
tracted from a single data set are often not indicative of
an entire population and heterogeneous samples subse-
quently applied to these models may not fit, leading to
performance degradation. In short, it is not possible to
guarantee that mining results based on a single gene
expression data set will be reliable or robust (Miller et
al., 2002). This problem can be addressed using classi-
fication algorithms capable of handling multiple, het-
erogeneous gene expression data sets. Apart from im-
proving mining performance, the use of such algorithms
would make mining results less sensitive to the varia-
tions of different microarray platforms and to experi-
mental conditions embedded in heterogeneous gene
expression data sets.
BACKGROUND
Recent research into the mining of gene expression data
has operated upon multiple, heterogeneous gene ex-
pression data sets. This research has taken two broad
approaches, addressing issues related either to the theo-
retical flexibility that is required to integrate gene
expression data sets with various microarray platforms
and technologies (Lee et al., 2003), or the focus of
this chapter - issues related to tumor classification
using an integration of multiple, heterogeneous gene
expression data sets (Bloom et al., 2004; Ng, Tan, &
Sundarajan, 2003). This type of tumor classification is
made more difficult by three types of variation, varia-
tion in the available microarray technologies, experi-
mental and biological variations, and variation in the
types of cancers themselves.
The first type of variation is caused by different
probe array notations of available microarray technolo-
gies. The two most common microarray technologies
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
are photolithographically synthesized oligonucleotide
probe arrays and spotted cDNA probe arrays. These have
both been reviewed by Sebastiani, Gussoni, Kohane, and
Ramoni (2003). They differ in their criteria for measur-
ing gene expression levels. Oligonucleotide probe ar-
rays measure mRNA abundance indirectly, while spot-
ted cDNA probe arrays measure cDNA relative to hy-
bridized reference mRNA samples. With the two com-
mon microarray technologies, there exist different probe
array notations (Lee et al., 2003). For example, human
probe array notations include GeneChip (Affymetrix,
Santa Clara, CA) U133, U95, and U35 accession num-
ber sets, BMR chips (Stanford University), UniGene
clusters, cDNA clone ID and GenBank identifiers. Al-
though the notations used in different technologies
sometimes referred to the same set of genes, this does
not indicate a simple one-to-one mapping (Ramaswamy,
Ross, Lander, & Golub, 2003). Users of these notations
should be aware of the potential for duplicated acces-
sion numbers in mapped results.
Another type of variation, the statistical variation
among different gene expression data sets is unavoid-
able because experimental and biological variations are
embedded in data sets (Miller et al., 2002). First of all,
individual gene expression data sets are conducted by
different laboratories with different experimental ob-
jectives and conditions even when using the same
microarray technology. Integration of them is a painful
task. Secondly, the expression levels of genes in experi-
ments are normally measured by the ratio of the expres-
sion levels of the genes in the varying conditions of
interest to the expression levels of the genes in some
reference conditions. These reference conditions are
varied from experiment to experiment. This is not a
problem if sample sizes are large enough. Zien, Fluck,
Zimmer, and Lengauer (2003) proposed that the use of
larger sample sizes (e.g. 20 samples) can prevent mining
results of gene expression data from suffering technical
and biological variations, and produce more reliable
results. Most gene expression data sets, however, con-
tain fewer than 20 samples per class. A more flexible
TEAM LinG
Heterogeneous Gene Data for Classifying Tumors
solution would be to meta-analyze multiple, heteroge-
neous gene expression data sets, forming meta-deci-
sions from a number of individual decisions.
The last difficulty is to find common features in
various cancer types. These features can be referred as
some sets of significant genes which are most likely
expressed in most cancer types, but they may be ex-
pressed differently in varying cancer types. The study of
human cancer has recently discovered that the develop-
ment of antigen-specific cancer vaccines leads to the
discovery of immunogenic genes. This group of tumor
antigens has been introduced as the term cancer-testis
(CT) antigen (Coulie et al., 2002). Discovered CT
antigens are recently grouped into distinct subsets and
named as cancer/testis (CT) immunogenic gene fami-
lies. Some works show that most CT immunogenic
gene families are expressed in more than one cancer
type, but with various expression frequencies. Cur-
rently, researchers have reviewed and summarized that
the current discovery is 44 CT immunogenic gene fami-
lies consisting of 89 individual genes in total (Scanlan,
Simpson, & Old, 2004).
MAIN THRUST
It is possible is to make classification algorithms more
reliable and robust by combining multiple, heteroge-
neous gene expression data sets. A simple combination
method is to merge or append one data set to another.
Unfortunately, this method is inflexible because data
sets have various scales and ranges of variations. These
are required to be the same in order to have consistent
scales for comparisons after the combination.
In this chapter, we discuss two approaches to com-
bine data sets consisting of variation in the available
microarray technologies. The first, and simplest, ap-
proach is to normalize gene expression levels of genes
in the data sets with mean zero and standard deviation
one (i.e. standard normal distribution, N(0, 1)) accord-
ing to the means and standard deviations across samples
in individual data sets. While this approach is simple to
apply, it assumes that all genes have the same or similar
expression rates. However, this assumption is incor-
rect. The fact is that only a small subset of genes reflects
the existence of tumors, and that the remaining genes in
a tumor are not epidemiologically significant. It should
also be noted that the reflected genes do not all express
at the same rate. Therefore, when all genes in data sets
are normalized to have N(0, 1), the variations of the
reflected genes may be underestimated and the varia-
tions of genes which are stable and irrelevant may be
overestimated. This situation worsens as the number of
genes in data sets increases.
The second and a better approach is to select a subset
of reference genes, also known as significant genes, and 0
to use the expression levels of these genes to estimate
scaling factors which are used to rescale the expression
levels of genes in other data sets with the same set of
reference genes as in the original subset. This approach
has two advantages. The first is that it allows the effects
of outliers caused by non-significant genes to be elimi-
nated while using only a subset of significant genes. In
a gene expression data set, only a proportion of genes is
tumor-specific. Because gene expression data contains
high-dimensional data, by focusing on such tumor-spe-
cific genes in classification would reduce computa-
tional costs. The second advantage is that it improves the
quality of the normalization or re-scaling since it avoids
the underestimation of expression level of significant
genes, a problem which may arise because of the pres-
ence of large amounts of non-significant genes. We also
note that the selection algorithms are the focus of much
current research. Some works that utilize existing
features selection algorithms include Dudoit, Yang,
Callow, and Speed (2002), Bloom et al. (2004), and Lee
et al. (2003). New or enhanced algorithms have been
proposed by Park et al. (2003), Ng, Tan, and Sundarajan.
(2003), Choi, Yu, Kim, and Yoo (2003), Storey &
Tibshirani (2003), Chilingaryan, Gevorgyan, Vardanyan,
Jones, & Szabo (2002), and Golub et al. (1999).
In recent years, detection of significant genes was
mainly done using fold-change detection. This detec-
tion method is unreliable because it does not take into
account statistical variability. Currently, however, most
algorithms that are used to select significant genes
apply statistical methods. In the rest of the chapter, we
first present some recent works on the identification of
significant genes using statistical methods. We then
briefly describe our proposed measure, Impact Factors
(IFs), which can be used to carry out tumor classifica-
tion using heterogeneous gene expression data (Fung &
Ng, 2003).
Statistical Methods
The most common statistical method for identifying
significant genes is the two-sample t-test (Cui &
Churchill, 2003). The advantage of this test is that,
because it requires only one gene to be studied for each
t-test, it is insensitive to heterogeneity in variance
across a couple of genes. However, while reliable t-
values require large sample sizes, gene expression data
sets normally consist of small sample sizes. This prob-
lem of small sample sizes can be overcome using global
t-tests, but it assumes that the variance is homogeneous
between different genes (Tusher, Tibshirani, & Chu,
2001). Tusher, Tibshirani, and Chu (2001) proposed a
551
TEAM LinG

modified t-test that they called significance analysis of
microarray (SAM). SAM identifies significant genes in
microarray experiments by measuring fluctuations of the
expression levels of genes across a number of microarray
experiments. These fluctuations are estimated using per-
mutation tests and, to avoid higher t-values, they are
expressed as a constant in the denominators of the two-
sample t-test. Tibshirani, Hastie, Narasimhan, and Chu
(2002) proposed a modified nearest-centroid classifica-
tion. For all genes, it uses a t-test to calculate a centroid
distance, defined as the distance from class centroids to
overall centroids among classes of the genes. The centroid
distance is then used to shrink the class centroids towards
the overall centroid to order to reduce overfitting.
Correlation analysis is another common statistical
method used to rank the significance of genes. Kuo,
Jenssen, Butte, Ohno-Machado, and Kohane (2002) ap-
plied the Pearson linear and Spearman rank-order corre-
lation coefficients to study the flexibility of cross-
platform utilization of data from multiple gene expres-
sion data sets. Lee et al. (2003) also used the Pearson and
Spearman correlation coefficients to study the correla-
tion among NCI-60 cancer data sets consisting of differ-
ent cancer types. They, however, proposed a measure to
rank the correlation of correlation among various data
sets. Later studies of correlation focused on multi-
platform, multi-type tumor data sets. Bloom et al. (2004)
used the Kruskal-Wallis H-test to identify significant
genes within multi-type, multi-platform tumor data sets
consisting of 21 data sets and 15 different cancer types.
Impact Factors
Recently, we proposed a dissimilarity measure called
Impact Factors (IFs) that measure the inter-experimen-
tal variations between individual classes in training
samples and heterogeneous testing samples (Fung & Ng,
2003). The calculation of IFs takes place in two stages of
selections, selection for re-scaling and selection for
classification. In the first stage, we first use SAM to
select a set of significant genes. From these, we calcu-
late individual reference points corresponding to differ-
ent classes in training set. These reference points are
then used to calculate their own scaling factors for the
corresponding classes. The factors are used to rescale
the expression levels of all genes in testing samples.
There are two advantages to using individual scaling
factors corresponding to different classes: they ensure
that the different gene expression levels of one class are
not underestimated or overestimated because of unbal-
anced sample sizes between classes, and they allow
individual testing samples to be compared with indi-
vidual classes.
552
Heterogeneous Gene Data for Classifying Tumors
The second stage of selection is selection for classi-
fication. This is done by calculating the differences
between rescaled testing samples (since there are two
scaling factors, there are two rescaled samples in binary-
class tumor classification) and individual classes in training
set. It should be noted that, to improve the discriminative
power of the IFs, only those genes with higher differences
are selected and used in classification.
IFs have been integrated into classifiers to perform
meta-classification of heterogeneous cancer gene ex-
pression data (Fung & Ng, 2003). For most classifiers
using either similarity or dissimilarity measures for
making classification decisions, IFs can be integrated
into classifiers by multiplying the IFs directly with the
original measures. The actual multiplication to be car-
ried out depends on whether it is considered as dissimi-
larity or similarity measures used by classifiers for
making decisions. If it is being applied to dissimilarity
measures, the IF of a class is multiplied by the measure
having the same class as the corresponding IF. In con-
trast, if it is being applied to similarity measures, the IF
of a class is multiplied by the measure having another
class as the corresponding IF.
FUTURE TRENDS
Although DNA microarrays can be used to predict
patients responses to medical treatment as well as
clinical outcomes, tumor classification using gene
expression data is as yet unreliable. This unreliability
has multiple causes. First of all, while some interna-
tional organizations such as the European
Bioinformatics Institute (EBI) and the National Cen-
ter for Biotechnology Information (NCBI) have cre-
ated their own gene expression data repositories, there
still exists no international benchmark data reposito-
ries. This makes it difficult for researchers to validate
findings. Certainly, there is a need for integrated gene
expression databases which can help researchers to
validate their findings with data conducted by different
laboratories around the world, and hence efficiency
and effectiveness of different proposed mining algo-
rithms can be compared objectively.
Unreliability could also be said to arise from inad-
equate interdisciplinary communication between pro-
fessionals and researchers in relevant fields. Recently,
a number of promising mining algorithms have been
proposed but it still remains for much work of this kind
to be analyzed and validated in molecular and biological
terms (Sevenet & Cussenot, 2003). The developers of
these algorithms would welcome such input, as it would
be an invaluable assistance to them in constructing
TEAM LinG
Heterogeneous Gene Data for Classifying Tumors
more general frameworks for defining different mining
tasks in bioinformatics.
Lastly, in terms of clinical studies, findings that have
been made using tumor classification based on gene
expression data is still not sufficiently reliable to draw
conclusions. This is because there the performance of
tumor classification may be influenced by many as yet
unidentified factors. Such factors may include the effects
of tumor heterogeneity and inflammatory cells.
Unreliability also arises from the narrow use of gene
expression data. Data mining algorithms mainly focus on
a set of genes, which are used as diagnostic markers in
clinical diagnosis. It could be more interesting to look for
algorithms which use as diagnostic markers not only the
expression levels of genes, but also other categories of
data, such as histopathology and patients information.
CONCLUSION
DNA microarray techniques are nowadays important in
cancer-related studies. These techniques operate on
large, publicly available gene expression data sets, con-
trolled by dispersed laboratories operating under vari-
ous experimental conditions and using a variety of
microarray technologies and platforms. A realistic re-
sponse to these facts is to mine data using integrated,
multiple, heterogeneous gene expression data. The ro-
bustness and reliability of mining algorithms developed
to handle such data can be validated on a new set of
heterogeneous gene expression data in addition to cross-
validation within a single data set. Algorithms that pass
such validation are less sensitive to variations induced
by experimental conditions and to different microarray
platforms, and thus the robustness and reliability of
mining results can be guaranteed.
In this chapter, we have discussed the difficulties of
three types of variation in tumor classification of mul-
tiple, heterogeneous gene expression data: variation in
the available microarray technologies, experimental and
biological variation, and variation in the types of can-
cers themselves. We further point out that identifica-
tion of significant genes is an important process in
tumor classification using gene expression data, espe-
cially for multiple, heterogeneous data. We have re-
viewed some recent works on the identification of sig-
nificant genes. Finally, we have presented the concept
of Impact Factors (IFs), which are used to combine and
classify heterogeneous gene expression data
REFERENCES
0
Bloom, G., Yang, I.V., Boulware, D., Kwong, K.Y., Coppola,
D., Eschrich, S. et al. (2004). Multi-platform, multi-site,
microarray-based human tumor classification. American
Journal of Pathology, 164(1), 9-16.
Chilingaryan, A., Gevorgyan, N., Vardanyan, A., Jones,
D., & Szabo, A. (2002). Multivariate approach for se-
lecting sets of differentially expressed genes. Math-
ematical Biosciences, 176(1), 59-69.
Cui, X., & Churchill, G.A. (2003). Statistical tests for
differential expression in cDNA microarray experi-
ments. Genome Biology, 4(4), 210.
Choi, J.K., Yu, U., Kim, S., & Yoo, O.J. (2003). Combin-
ing multiple microarray studies and modeling interstudy
variation. Bioinformatics, 19(Suppl. 1), i84-i90.
Coulie, P.G., Karanikas, V., Lurquin, C., Colau, D.,
Connerotte, T., Hanagiri, T. et al. (2002). Cytolytic T-cell
responses of cancer patients vaccinated with a MAGE
antigen. Immunological Reviews, 188(1), 33-42.
Dudoit, S., Yang, Y.H., Callow, M.J., & Speed T.P.
(2002). Statistical methods for identifying differen-
tially expressed genes in replicated cDNA microarray
experiments. Statistica Sinica, 12, 111-139.
Fung, B.Y.M., & Ng, V.T.Y. (2003). Classification of het-
erogeneous gene expression data. SIGKDD Special
Issue on Microarray Data Mining, 5(2), 69-78.
Golub, T.R., Slonim, D.K., Tamayo, P., Huard, C.,
GaasenBeek, M., Mesirov, J.P., et al. (1999). Molecular
classification of cancer: Class discovery and class pre-
diction by gene-expression monitoring. Science, 286,
531-537.
Kuo, W.P., Jenssen, T.K., Butte, A.J., Ohno-Machado,
L., & Kohane, I.S. (2002). Analysis of matched mRNA
measurements from two different microarray technolo-
gies. Bioinformatics, 18(3), 405-412.
Lee, J.K., Bussey, K.J., Gwadry, F.G., Reinhold, W.,
Riddick, G., Pelletier, S.L. et al. (2003). Comparing cDNA
and oligonucleotide array data: concordance of gene
expression across platforms for the NCI-60 cancer cells.
Genome Biology, 4(12), R82.
Miller, L.D., Long, P.M., Wong, L., Mukherjee, S., McShane,
L.M., & Liu, E.T. (2002). Optimal gene expression analysis
by microarrays. Cancer Cell, 2(5), 353-361.
553
TEAM LinG

Ng, S.K., Tan, S.H., & Sundarajan, V.S. (2003). On combin-
ing multiple microarray studies for improved functional
classification by whole-dataset feature selection. Ge-
nome Informatics, 14, 44-53.
Park, T., Yi, S.G., Lee, S., Lee, S.Y., Yoo, D.H., Ahn, J.I. et
al. (2003). Statistical tests for identifying differentially
expressed genes in time-course microarray experiments.
Bioinformatics, 19(6), 694-703.
Ramaswamy, S., Ross, K.N., Lander, E.S., & Golub, T.R.
(2003). Evidence for a molecular signature of metastasis
in primary solid tumors. Nature Genetics, 33, 49-54.
Scanlan, M.J., Simpson, A.J.G., & Old, L.J. (2004). The
cancer/testis genes: Review, standardization, and com-
mentary. Cancer Immunity, 4, 1.
Sebastiani, P., Gussoni, E., Kohane, I.S., & Ramoni,
M.F. (2003). Statistical challenges in functional
genomics. Statistical Science, 18(1), 33-70.
Sevenet, N., & Cussenot, O. (2003). DNA microarrays
in clinical practice: Past, present, and future. Clinical
and Experimental Medicine, 3(1), 1-3.
Storey, J.D., & Tibshirani, R.J. (2003). Statistical signifi-
cance for genomewide studies. Proceedings of the Na-
tional Academy of Sciences of the United States of America,
100(16), 9440-9445.
Tibshirani, R., Hastie, T., Narasimhan, B., & Chu, G. (2002).
Diagnosis of multiple cancer types by shrunken centroids
of gene expression. Proceedings of the National Acad-
emy of Sciences of the United States of America, 99(10),
6567-6572.
Tusher, V.G., Tibshirani, R., & Chu, G. (2001). Significance
analysis of microarrays applied to the ionizing radiation
response. Proceedings of the National Academy of Sci-
ences of the United States of America, 98(9), 5116-5121.
554
Heterogeneous Gene Data for Classifying Tumors
Zien, A., Fluck, J., Zimmer, R., & Lengauer, T. (2003).
Microarrays: How many do you need? Journal of Compu-
tational Biology, 10(3-4), 653-667.
KEY TERMS
Bioinformatics: It is an integration of mathematical,
statistical and computational methods to analyze and
handle biological, biomedical, biochemical, and biophysi-
cal information.
Cancer-testis (CT) Antigen: It is immunogenic in
cancer patients, which exhibit highly tissue-restricted
expression, and are considered promising target mol-
ecules for cancer vaccines.
Classification: It is the process of distributing things
into classes or categories of the same type by a learnt
mapping function.
Gene Expression: It describes how the information of
transcription and translation encoded in a segment of
DNA is converted into proteins in a cell.
Microarrays: It is the technology for biological explo-
ration which allows to simultaneously measure the amount
of mRNA in up to tens of thousand of genes in a single
experiment.
Normalization: In terms of gene expression data, it is
a pre-processing to minimize systematic bias and remove
the impact of non-biological influences before data analy-
sis is performed.
Probe Arrays: They are a list of labeled, single-
stranded DNA or RNA molecules in specific nucleotide
sequences, which are used to detect the complementary
base sequence by hybridization.
TEAM LinG

Hierarchical Document Clustering
Benjamin C. M. Fung
Simon Fraser University, Canada
Ke Wang
Simon Fraser University, Canada
Martin Ester
Simon Fraser University, Canada
INTRODUCTION
Document clustering is an automatic grouping of text
documents into clusters so that documents within a
cluster have high similarity in comparison to one an-
other, but are dissimilar to documents in other clusters.
Unlike document classification (Wang, Zhou, & He,
2001), no labeled documents are provided in clustering;
hence, clustering is also known as unsupervised learn-
ing. Hierarchical document clustering organizes clus-
ters into a tree or a hierarchy that facilitates browsing.
The parent-child relationship among the nodes in the
tree can be viewed as a topic-subtopic relationship in a
subject hierarchy such as the Yahoo! directory.
This chapter discusses several special challenges in
hierarchical document clustering: high dimensionality,
high volume of data, ease of browsing, and meaningful
cluster labels. State-of-the-art document clustering al-
gorithms are reviewed: the partitioning method
(Steinbach, Karypis, & Kumar, 2000), agglomerative and
divisive hierarchical clustering (Kaufman & Rousseeuw,
1990), and frequent itemset-based hierarchical cluster-
ing (Fung, Wang, & Ester, 2003). The last one, which was
recently developed by the authors, is further elaborated
since it has been specially designed to address the hierar-
chical document clustering problem.
BACKGROUND
Document clustering is widely applicable in areas such
as search engines, web mining, information retrieval,
and topological analysis. Most document clustering
methods perform several preprocessing steps including
stop words removal and stemming on the document set.
Each document is represented by a vector of frequen-
cies of remaining terms within the document. Some
document clustering algorithms employ an extra pre-
processing step that divides the actual term frequency
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
555
0
by the overall frequency of the term in the entire document
set. The idea is that if a term is too common across different
documents, it has little discriminating power (Rijsbergen,
1979). Although many clustering algorithms have been
proposed in the literature, most of them do not satisfy the
special requirements for clustering documents:
High Dimensionality: The number of relevant terms
in a document set is typically in the order of thou-
sands, if not tens of thousands. Each of these terms
constitutes a dimension in a document vector. Natu-
ral clusters usually do not exist in the full dimen-
sional space, but in the subspace formed by a set of
correlated dimensions. Locating clusters in sub-
spaces can be challenging.
Scalability: Real world data sets may contain
hundreds of thousands of documents. Many clus-
tering algorithms work fine on small data sets, but
fail to handle large data sets efficiently.
Accuracy: A good clustering solution should have
high intra-cluster similarity and low inter-cluster
similarity, i.e., documents within the same cluster
should be similar but are dissimilar to documents
in other clusters. An external evaluation method,
the F-measure (Rijsbergen, 1979), is commonly
used for examining the accuracy of a clustering
algorithm.
Easy to Browse with Meaningful Cluster Descrip-
tion: The resulting topic hierarchy should provide
a sensible structure, together with meaningful clus-
ter descriptions, to support interactive browsing.
Prior Domain Knowledge: Many clustering algo-
rithms require the user to specify some input param-
eters, e.g., the number of clusters. However, the
user often does not have such prior domain knowl-
edge. Clustering accuracy may degrade drastically
if an algorithm is too sensitive to these input param-
eters.
TEAM LinG

DOCUMENT CLUSTERING METHODS
Hierarchical Clustering Methods
One popular approach in document clustering is
agglomerative hierarchical clustering (Kaufman &
Rousseeuw, 1990). Algorithms in this family build the
hierarchy bottom-up by iteratively computing the simi-
larity between all pairs of clusters and then merging the
most similar pair. Different variations may employ
different similarity measuring schemes (Karypis, 2003;
Zhao & Karypis, 2001). Steinbach (2000) shows that
Unweighted Pair Group Method with Arithmatic Mean
(UPGMA) (Kaufman & Rousseeuw, 1990) is the most
accurate one in its category. The hierarchy can also be
built top-down which is known as the divisive approach.
It starts with all the data objects in the same cluster and
iteratively splits a cluster into smaller clusters until a
certain termination condition is fulfilled.
Methods in this category usually suffer from their
inability to perform adjustment once a merge or split
has been performed. This inflexibility often lowers the
clustering accuracy. Furthermore, due to the complex-
ity of computing the similarity between every pair of
clusters, UPGMA is not scalable for handling large data
sets in document clustering as experimentally demon-
strated in (Fung, Wang, & Ester, 2003).
Partitioning Clustering Methods
K-means and its variants (Cutting, Karger, Pedersen, &
Tukey, 1992; Kaufman & Rousseeuw, 1990; Larsen &
Aone, 1999) represent the category of partitioning clus-
tering algorithms that create a flat, non-hierarchical
clustering consisting of k clusters. The k-means algo-
rithm iteratively refines a randomly chosen set of k
initial centroids, minimizing the average distance (i.e.,
maximizing the similarity) of documents to their clos-
est (most similar) centroid. The bisecting k-means al-
gorithm first selects a cluster to split, and then employs
basic k-means to create two sub-clusters, repeating
these two steps until the desired number k of clusters is
reached. Steinbach (2000) shows that the bisecting k-
means algorithm outperforms basic k-means as well as
agglomerative hierarchical clustering in terms of accu-
racy and efficiency (Zhao & Karypis, 2002).
Both the basic and the bisecting k-means algorithms
are relatively efficient and scalable, and their complex-
ity is linear to the number of documents. As they are
easy to implement, they are widely used in different
clustering applications. A major disadvantage of k-
means, however, is that an incorrect estimation of the
input parameter, the number of clusters, may lead to
poor clustering accuracy. Also, the k-means algorithm
556
Hierarchical Document Clustering
is not suitable for discovering clusters of largely vary-
ing sizes, a common scenario in document clustering.
Furthermore, it is sensitive to noise that may have a
significant influence on the cluster centroid, which in
turn lowers the clustering accuracy. The k-medoids
algorithm (Kaufman & Rousseeuw, 1990; Krishnapuram,
Joshi, & Yi, 1999) was proposed to address the noise
problem, but this algorithm is computationally much
more expensive and does not scale well to large docu-
ment sets.
Frequent Itemset-Based Methods
Wang et al. (1999) introduced a new criterion for clus-
tering transactions using frequent itemsets. The intu-
ition of this criterion is that many frequent items should
be shared within a cluster while different clusters should
have more or less different frequent items. By treating
a document as a transaction and a term as an item, this
method can be applied to document clustering; however,
the method does not create a hierarchy of clusters.
The Hierarchical Frequent Term-based Clustering
(HFTC) method proposed by (Beil, Ester, & Xu, 2002)
attempts to address the special requirements in docu-
ment clustering using the notion of frequent itemsets.
HFTC greedily selects the next frequent itemset, which
represents the next cluster, minimizing the overlap of
clusters in terms of shared documents. The clustering
result depends on the order of selected itemsets, which
in turn depends on the greedy heuristic used. Although
HFTC is comparable to bisecting k-means in terms of
clustering accuracy, experiments show that HFTC is not
scalable (Fung, Wang, & Ester, 2003).
A Scalable Algorithm for Hierarchical
Document Clustering: FIHC
A scalable document clustering algorithm, Frequent
Itemset-based Hierarchical Clustering (FIHC) (Fung,
Wang, & Ester, 2003), is discussed in greater detail
because this method satisfies all of the requirements of
document clustering mentioned above. We use item
and term as synonyms below. In classical hierarchical
and partitioning methods, the pairwise similarity be-
tween documents plays a central role in constructing a
cluster; hence, those methods are document-centered.
FIHC is cluster-centered in that it measures the cohe-
siveness of a cluster directly using frequent itemsets:
documents in the same cluster are expected to share
more common itemsets than those in different clusters.
A frequent itemset is a set of terms that occur
together in some minimum fraction of documents. To
illustrate the usefulness of this notion for the task of
clustering, let us consider two frequent items, win-
TEAM LinG
Hierarchical Document Clustering
dows and apple. Documents that contain the word
windows may relate to renovation. Documents that
contain the word apple may relate to fruits. However,
if both words occur together in many documents, then
another topic that talks about operating systems should
be identified. By precisely discovering these hidden
topics as the first step and then clustering documents
based on them, the quality of the clustering solution can
be improved. This approach is very different from HFTC
where the clustering solution greatly depends on the
order of selected itemsets. Instead, FIHC assigns docu-
ments to the best cluster from among all available clus-
ters (frequent itemsets). The intuition of the clustering
criterion is that there are some frequent itemsets for
each cluster in the document set, but different clusters
share few frequent itemsets. FIHC uses frequent itemsets
to construct clusters and to organize clusters into a topic
hierarchy.
The following definitions are introduced in (Fung,
Wang, & Ester, 2003): A global frequent itemset is a set
of items that appear together in more than a minimum
fraction of the whole document set. A global frequent
item refers to an item that belongs to some global fre-
quent itemset. A global frequent itemset containing k
items is called a global frequent k-itemset. A global
frequent item is cluster frequent in a cluster Ci if the
item is contained in some minimum fraction of docu-
ments in Ci. FIHC uses only the global frequent items in
document vectors; thus, the dimensionality is signifi-
cantly reduced.
The FIHC algorithm can be summarized in three
phases: First, construct initial clusters. Second, build a
cluster (topic) tree. Finally, prune the cluster tree in
case there are too many clusters.
Constructing Clusters
For each global frequent itemset, an initial cluster is
constructed to include all the documents containing this
itemset. Initial clusters are overlapping because one
document may contain multiple global frequent itemsets.
FIHC utilities this global frequent itemset as the cluster
label to identify the cluster. For each document, the
best initial cluster is identified and the document is
assigned only to the best matching initial cluster. The
goodness of a cluster Ci for a document docj is measured
by some score function using cluster frequent items of
initial clusters. After this step, each document belongs
to exactly one cluster. The set of clusters can be viewed
as a set of topics in the document set.
Example: Figure 1 depicts a set of initial clusters.
Each of them is labeled with a global frequent
Figure 1. Initial Clusters
0
{Sports, Tennis} {Sports, Tennis, Ball} {Sports, Tennis, Racket}
{Sports, Ball}
Doci
Sports, Tennis, Ball
{Sports}
itemset. A document Doc1 containing global fre-
quent items Sports, Tennis, and Ball is as-
signed to clusters {Sports}, {Sports, Ball}, {Sports,
Tennis} and {Sports, Tennis, Ball}. Suppose
{Sports, Tennis, Ball} is the best cluster for Doc1
measured by some score function. Doc1 is then
removed from {Sports}, {Sports, Ball}, and {Sports,
Tennis}.
Building Cluster Tree
In the cluster tree, each cluster (except the root node)
has exactly one parent. The topic of a parent cluster is
more general than the topic of a child cluster and they
are similar to a certain degree (see Figure 2 for an
example). Each cluster uses a global frequent k-itemset
as its cluster label. A cluster with a k-itemset cluster
label appears at level k in the tree. The cluster tree is
built bottom up by choosing the best parent at level k-
1 for each cluster at level k. The parents cluster label
must be a subset of the childs cluster label. By treating
all documents in the child cluster as a single document,
the criterion for selecting the best parent is similar to
the one for choosing the best cluster for a document.
Example: Cluster {Sports, Tennis, Ball} has a
global frequent 3-itemset label. Its potential par-
ents are {Sports, Ball} and {Sports, Tennis}. Sup-
pose {Sports, Tennis} has a higher score. It be-
comes the parent cluster of {Sports, Tennis, Ball}.
Pruning Cluster Tree
The cluster tree can be broad and deep, which becomes
not suitable for browsing. The goal of tree pruning is to
efficiently remove the overly specific clusters based
on the notion of inter-cluster similarity. The idea is
that if two sibling clusters are very similar, they should
be merged into one cluster. If a child cluster is very
similar to its parent (high inter-cluster similarity),
then replace the child cluster with its parent cluster.
The parent cluster will then also include all documents
of the child cluster.
557
TEAM LinG

Figure 2. Sample cluster tree
{Sports} Cluster label
{Sports, Ball} {Sports, Tennis}
{Sports, Tennis, Ball} {Sports, Tennis, Racket}
Example: Suppose the cluster {Sports, Tennis, Ball}
is very similar to its parent {Sports, Tennis} in
Figure 2. {Sports, Tennis, Ball} is pruned and its
documents, e.g., Doc1, are moved up into cluster
{Sports, Tennis}.
Evaluation of FIHC
The FIHC algorithm was experimentally evaluated and
compared to state-of-the-art document clustering meth-
ods. See (Fung, Wang, & Ester, 2003) for more details.
FIHC uses only the global frequent items in document
vectors, drastically reducing the dimensionality of the
document set. Experiments show that clustering with
reduced dimensionality is significantly more efficient
and scalable. FIHC can cluster 100K documents within
several minutes while HFTC and UPGMA cannot even
produce a clustering solution. FIHC is not only scalable,
but also accurate. The clustering accuracy of FIHC
consistently outperforms other methods. FIHC allows
the user to specify an optional parameter, the desired
number of clusters in the solution. However, close-to-
optimal accuracy can still be achieved even if the user
does not specify this parameter.
The cluster tree provides a logical organization of
clusters which facilitates browsing documents. Each clus-
ter is attached with a cluster label that summarizes the
documents in the cluster. Different from other clustering
methods, no separate post-processing is required for
generating these meaningful cluster descriptions.
RELATED LINKS
The followings are some clustering tools on the Internet:
Tools: FIHC implements Frequent Itemset-based
Hierarchical Clustering.
Website: http://www.cs.sfu.ca/~ddm/
Tools: CLUTO implements Basic/Bisecting K-means
and Agglomerative methods.
558
Hierarchical Document Clustering
Website: http://www-users.cs.umn.edu/~karypis/
cluto/
Tools: Vivsimo is a clustering search engine.
Website: http://vivisimo.com/
FUTURE TRENDS
Incrementally Updating the Cluster
Tree
One potential research direction is to incrementally
update the cluster tree (Guha, Mishra, Motwani &
OCallaghan, 2000). In many cases, the number of docu-
ments is growing continuously in a document set, and it
is infeasible to rebuild the cluster tree upon every
arrival of a new document. Using FIHC, one can simply
assign the new document to the most similar existing
cluster. But the clustering accuracy may degrade in the
course of the time, since the original global frequent
itemsets may no longer reflect the current state of the
overall document set. Incremental clustering is closely
related to some of the recent research on data mining in
stream data (Ordonez, 2003).
CONCLUSION
Most traditional clustering methods do not completely
satisfy special requirements for hierarchical document
clustering, such as high dimensionality, high volume,
and ease of browsing. In this chapter, we review several
document clustering methods in the context of these
requirements, and a new document clustering method,
FIHC, is discussed a bit more detail. Due to massive
volumes of unstructured data generated in the globally
networked environment, the importance of document
clustering will continue to grow.
REFERENCES
Beil, F., Ester, M., & Xu, X. (2002). Frequent term-
based text clustering. International Conference on
Knowledge Discovery and Data Mining, KDD02 (pp.
436-442), Edmonton, Alberta, Canada.
Cutting, D.R., Karger, D.R., Pedersen, J.O., & Tukey,
J.W. (1992). Scatter/gather: A cluster-based approach
to browsing large document collections. International
Conference on Research and Development in Infor-
TEAM LinG
Hierarchical Document Clustering
mation Retrieval, SIGIR92 (pp. 318-329), Copenhagen,
Denmark.
Fung, B., Wang, K., & Ester, M. (2003, May). Hierarchi-
cal document clustering using frequent itemsets. SIAM
International Conference on Data Mining, SDM03 (pp.
59-70), San Francisco, CA, United States..
Guha, S., Mishra, N., Motwani, R., & OCallaghan, L.
(2000). Clustering data streams. Symposium on Foun-
dations of Computing Science (pp. 359-366).
Karypis, G. (2003). Cluto 2.1.1: A software package for
clustering high dimensional datasets. Retrieved from
http://www-users.cs.umn.edu/~karypis/cluto/
Kaufman, L., & Rousseeuw, P.J. (1990, March). Finding
groups in data: An introduction to cluster analysis. New
York: John Wiley & Sons, Inc.
Krishnapuram, R., Joshi, A., & Yi, L. (1999, August). A
fuzzy relative of the k-medoids algorithm with application
to document and snippet clustering. IEEE International
Conference - Fuzzy Systems, FUZZIEEE 99, Korea.
Larsen, B., & Aone, C. (1999). Fast and effective text
mining using linear-time document clustering. Interna-
tional Conference on Knowledge Discovery and Data
Mining, KDD99 (pp. 16-22), San Diego, California, United
States.
Ordonez, C. (2003). Clustering binary data streams with
K-means. Workshop on Research issues in data min-
ing and knowledge discovery, SIGMOD03 (pp. 12-19),
San Diego, California, United States.
Steinbach, M., Karypis, G., & Kumar, V. (2000). A
comparison of document clustering techniques. Work-
shop on Text Mining, SIGKDD00.
van Rijsbergen, C.J. (1979). Information retrieval (2nd
ed.). London: Butterworth Ltd.
Wang, K., Xu, C., & Liu, B. (1999). Clustering transac-
tions using large items. International Conference on
Information and Knowledge Management, CIKM99 (pp.
483-490), Kansas City, Missouri, United States.
Wang, K., Zhou, S., & He Y. (2001, April). Hierarchical
classification of real life documents. SIAM Interna-
tional Conference on Data Mining, SDM01, Chicago,
United States.
Zhao, Y., & Karypis, G. (2001). Criterion functions for
document clustering: Experiments and analysis. Techni- 0
cal report, Department of Computer Science, University of
Minnesota.
Zhao, Y., & Karypis, G. (2002, November). Evaluation of
hierarchical clustering algorithms for document datasets.
International Conference on Information and Knowl-
edge Management (pp. 515-524), McLean, Virginia, United
States.
KEY TERMS
Cluster Frequent Item: A global frequent item is
cluster frequent in a cluster Ci if the item is contained in
some minimum fraction of documents in Ci.
Document Clustering: The automatic organization
of documents into clusters or group so that documents
within a cluster have high similarity in comparison to
one another, but are very dissimilar to documents in
other clusters.
Document Vector: Each document is represented
by a vector of frequencies of remaining items after
preprocessing within the document.
Global Frequent Itemset: A set of words that
occur together in some minimum fraction of the whole
document set.
Inter-Cluster Similarity: The overall similarity among
documents from two different clusters.
Intra-Cluster Similarity: The overall similarity among
documents within a cluster.
Medoid: The most centrally located object in a cluster.
Stemming: For text mining purposes, morphological
variants of words that have the same or similar semantic
interpretations can be considered as equivalent. For ex-
ample, the words computation and compute can be
stemmed into comput.
Stop Words Removal: A preprocessing step for text
mining. Stop words, like the and this which rarely help
the mining process, are removed from input data.
559
TEAM LinG
560

High Frequency Patterns in Data Mining

Tsau Young Lin
San Jose State University, USA

INTRODUCTION 1. A relational table is a bag relation, that is, repeti-

tions of tuples are permissible (Garcia-Monila et al.
The principal focus is to examine the foundation of 2002)
association (rule) mining (AM) via granular computing 2. An item is an attribute value,
(GrC). The main results is: The set of all high frequency 3. A q-itemset is a subtuple of length q,
patterns can be found by sloving linear inequalities 4. A high frequency pattern of length q is a q-subtuple
within a polynomial time. if its number of occurrences is greater than or
equal to a given threshold.

BACKGROUND Emerging Data Mining Method -

Some Foundation Issues in Data Mining
What is data mining? The following informal paraphrase
of Fayad et al. (1996)s definition seems quite universal:
Deriving useful patterns from data.
The keys are data, patterns, derivation system, and
useful-ness. We will examine critically the current
practices of AM.
Some Basic Terms in Association
Mining (AM)
In AM, two measures, support and confidence, are the
main criteria. It is well known among researchers the
support is the main hurdle, in other words, high fre-
quency patterns are the main focus. AM is originated
from the market basket data (Agrawal, 1993). However,
we will be interested in AM for relational tables. For
definitive, we assert:
Granular Computing
Bitmap index is a common notion in database theory.
The advantage of bitmap representation is
computationally efficient (Louis & Lin, 2000), and the
drawback is the order of the table has to be fixed
(Garcia-Molina, 2002). Based on granular computing,
we propose a new method, called granular representa-
tions, that avoids this drawback. We will illustrate the
idea by examples. The following example is modified
from the text cited above (p. 702). A relational table K
is viewed as a knowledge representation of a set V,
called the universe, of real world entities by tuples of
data; see Table 1.
A bitmap index for an attribute is a collection of bit-
vectors, one for each possible value that may appear in
the attribute. For the first attribute, BusinesSize (the
amount of business in millions), the bitmap index would
have nine bit-vectors. The first bit-vector, for value
TWENTY, is 100011100, because the first, fifth, sixth,
and seventh tuple have BusinesSize = TWENTY. The
other two, for values TEN and THIRTY, are 011100000 and
000000011 respectively; Table 1 shows both the original

Table 1. K and B are isomorphic

V BusinesSize Bmonth City BusinesSize Bmonth City

v1 TWENTY MAR NY 100011100 110011000 101000000
v2 TEN MAR SJ 011100000 110011000 010011100
v3 TEN FEB NY 011100000 001100000 101000000
v4 K TEN FEB LA 011100000 001100000 000100011
v5 TWENTY MAR SJ 100011100 110011000 010011100
v6 TWENTY MAR SJ 100011100 110011000 010011100
v7 TWENTY APR SJ 100011100 000000100 010011100
v8 THIRTY JAN LA 000000011 000000011 000100011
v9 THIRTY JAN LA 000000011 000000011 000100011
Relational Table K Bitmap Table B
TABLE 1. K AND B ARE ISOMORPHIC

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
High Frequency Patterns in Data Mining

Table 2a. Granular data model (GDM) for BusinesSize attribute

BusinesSize Granular Representation Bitmap Representation

0
TWENTY ={v1, v5, v6, v7} =100011100
TEN ={v2, v3, v4} =011100000
THIRTY ={v8, v9} =000000011
GDM in Granules GDM in Bitmaps

Table 2b. Granular data model (GDM) for Bmonth attribute

Bmonth Granular Representation Bitmap Representation

Jan ={v8, v9} =000000011
Feb ={v3, v4} =001100000
Mar ={v1, v2, v5, v6} =110011000
APR ={v7} =000000100
GDM in Granules GDM in Bitmaps

Table 2c. Granular data model (GDM) for CITY attribute

City Granular Representation Bitmap Representation

LA ={v4, v8, v9} =000100011
NY ={v1, v3} ={v1, v3}
SJ ={v2, v5, v6, v7} =010011100
GDM in Granules GDM in Bitmaps

Table 2. K and G are isomorphic

V BusinesSize Bmonth City BusinesSize Bmonth City

v1 TWENTY MAR NY {v1,v5,v6,v7} {v1,v2,v5,v6} {v1,v3}
v2 TEN MAR SJ {v2,v3,v4} {v1,v2,v5,v6} {v2,v5,v6,v7}
v3 TEN FEB NY {v2,v3,v4} {v3,v4} {v1,v3}
v4 K TEN FEB LA {v2,v3,v4} {v3,v4} {v4,v8,v9}
v5 TWENTY MAR SJ {v1,v5,v6,v7} {V1,V2,V5,V6} {V2,V5,V6,V7}
v6 TWENTY MAR SJ {V1,V5,V6,V7} {V1,V2,V5,V6} {V2,V5,V6,V7}
v7 TWENTY APR SJ {V1,V5,V6,V7} {v7} {V2,V5,V6,V7}
v8 THIRTY JAN LA {V8, V9} {v8,v9} {V4,V8,V9}
v9 THIRTY JAN LA {V8, V9} {v8,v9} {V4,V8,V9}
Bag Relation K GRANULR TABLE G

table and bitmap table. Bmonth means Birth month; City
means the location of the entities.
Next, we will interpret the bit-vectors in terms of set
theory. A bit-vector can be viewed as a representation of
a subset of V. For example, the bit-vector, 100011100,
of BusinesSize = TWENTY says that the first, fifth,
sixth, and seventh entities have been selected, in other
words, the bit-vector represents the subset {v1, v5, v6, v 7}.
The other two bi-vectors, for values TEN and THIRTY,
represent the subsets {v2, v3, v4} and {v 8, v9} respectively.
We summarize such translations in Table 2a,b,c. and
refer to these subsets as elementary granules.
Some easy observations:
1. The collection of elementary granules of an at-
tribute (column) forms a partition, that is, all
granules of this attribute are pairwise disjoint.
This fact was observed by Pawlak (1982) and Tony
Lee (1983).
2. From Tables 1 and 2, one can easily conclude that
the relational table K, the bitmap table B and
granular table G are isomorphic. Two tables are
isomorphic if one can transform a table to the
other by renaming all attribute values in a one-to-
one fashion.
561

TEAM LinG
 High Frequency Patterns in Data Mining

Granular Data Model (GDM)
Uninterpreted Relational Table in Free
Format
The middle columns of Table 2a, 2b and 2c define 3
partitions. The universe and such 3 partions, denoted by
(V, {E BusinesSize,E Bmonth,ECity}), determines the granular
table G and vice versa. More generally, 3-tuple (V, E, C) is
called a GDM, where E is a set of finite family of partitions,
and C consists of the names of all elementary granules. A
partition (equivalence relation) of V that is not in the given
E is referred to as an uninterpreted attribute of GDM, and
its elementary granules are un-interpreted attribute val-
ues.
GDM Theorem: The granular table G. determines
GDM and vice versa.
In view of Isomorphic theorem below, it is sufficient to
do AM in GDM.
MAIN THRUST
Analysis of Association Mining (AM)
To understand the mathematical mechanics of AM, let us
examine how the information has been created and pro-
cessed. We will take the deductive data mining approach.
First, let us set up some terminology. A symbol is a
string of bits and bytes that represents a slice of real
world, however, such a real world meaning does not
participate in the formal processing or computing. We
term such a processing computing with symbols. In AI,
such a symbol is termed a semantic primitive.
(Feigenbaum,1981). A symbol is termed a word, if the
intended real world meaning participates in the formal
processing or computing. We term such a processing
computing with words. Note that mathematicians use
words (in group theory) as symbols; their words are our
symbols.
Data Processing and Computing with
Words
In traditional data processing (TDP), a relational table
is a knowledge representation of a slice of real world.
So each symbol of the table represents (to human) a
piece of the real world; however, such a representation
is not implemented in the system. Nevertheless, DBMS,
under human commands, does process the data, for
examples, Bmonth (attribute), April, March (attribute
values) with human-perceived semantics. So in TDP the
relational table is a table of words; TDP is human
directed computing with words.
Data Mining and Computing with
Symbols
In (automated) AM we use the table created in TDP.
However, AM algorithms regard the TDP data as sym-
bols; no real world meaning of each word participates
in the process of AM. High frequency patterns are
completely deduced from the counting of the symbols.
AM is computing with symbols. The input data of AM is
a relational table of symbols, whose real wolrd mean-
ing does not participate in formal computing.
Under such a circumstance, if we replace the given
set of symbols by a new set, then we can derive new
patterns by simply replacing the symbols in old pat-
terns. Formally, we have (Lin, 2002)
Isomorphic Theorem: Isomorphic relational tables
have isomorphic patterns.
This theorem implies that the theory of AM is a
syntactic theory.

Table 3. The isomorphim of Table K and K

V BusinesSize Bmonth CITY U WT NAME Material

v1 TWENTY MAR NY u1 20 SCREW STEEL
v2 TEN MAR SJ u2 10 SCREW BRASS
v3 TEN FEB NY u3 10 NAIL STEEL
v4 K TEN FEB LA u4 K 10 NAIL ALLOY
v5 TWENTY MAR SJ u5 20 SCREW BRASS
v6 TWENTY MAR SJ u6 20 SCREW BRASS
v7 TWENTY APR SJ u7 20 PIN BRASS
v8 THIRTY JAN LA u8 30 HAMMER ALLOY
v9 THIRTY JAN LA u9 30 HAMMER ALLOY
Bag Relation K Bag Relation K

562

TEAM LinG
High Frequency Patterns in Data Mining
Table 4. Three isomorphic 2-patterns; support=cardinality of granules
K K
(TWENTY, MAR) (20, SCREW)
(MAR, SJ) (SCREW, BRASS)
(TWENTY, SJ) (20, BRASS)
Example: From Table 3, it should be clear that the
one-to-one correspondences between K and K
induces consistently a one-to-one correspondence
between the two sets of distinct attribute values.
We describe such a phenomenon by the statement:
K and K are isomorphic.
In Table 4, we display the high frequency patterns of
length 2 from Table K, K and GDM; the three sets of
patterns are isomorphic to each other. So for AM, we
can use any one of the three tables. An observation: In
using K or K for AM, one needs to scan the table to get
the support, while in using GDM, the support can be
read from the cardinality of the granules, no database
scan is required one strength of GDM. Another obser-
vation: From the definition of elementary granules, it
should be obvious that subtuples are mapped to the
intersections of elementary granules.
Patterns and Granular Formulas
Implicitly AM has assumed high frequency patterns are
expressions of the input symbols (elements of the
input relational table.) Such assumptions are not made
in other techniques. In neural network techniques, the
input data are numerical, its patterns are not numerical
expressions. They are essentially functions that are
derived from activation functions (Lin, 1996; Park &
Sanders, 1989).
Let us back to AM, the implicit assumption simpli-
fies the problem. What are the possible expressions
of the input symbols? There are two possible formal-
isms, logic formula and set theoretical algebraic ex-
pression. In logic form, we have several choices, deduc-
tive database systems, datalog, or decision logic among
others (Pawlak, 1991; Ullman, 1988-89); we choose
decision logic because it is simpler. In set theoretical
form, we use GDM (Lin, 2000).
Expressions of High Frequency
Patterns
1. Logic Based: A high frequency pattern in deci-
sion logic is a logic formula, whose meaning set
(support) has cardinality greater than or equal to the threshold.
GDM in Granules Support
0
={v1, v5, v6, v7}{v1, v2, v5, v6} 3
={v1, v2, v5, v6}{v2, v5, v6, v7} 3
={v1, v5, v6, v7}{v2, v5, v6, v7} 3
2. Set Based: A high frequency pattern in GDM is a
granular expression, which is a set theoretical
algebriac expression of elementary granules; when
the expression is evaluated set theoretically, the
cardinailty of the resultant set is greater than or
equal to the threshold; we will call the algebraic
expressions granular pattern. Note that several
distinct algebraic expressions of elementary gran-
ules may have the same resultant set.
Informally, a logical formula of granular pattern is
the logic formula of the names of elementary granules
(Lin, 2000); more pricisely we translate elementary
granules, and into their names, or and and
respectively. Next, we note that there are only finitely
many distinct subsets that can be generated by the inter-
sections and unions of elementary granules in GDM. If
we only consider the disjunct normal form, the total
possible high frequency patterns in AM is finite.
Finding High Frequency Patterns by
Solving a Set of Linear Inequalities
Let B be the Boolean algebra generated by the elemen-
tary granules; the partial order is the set theoretical
inclusion . Then B is the set of all granular expres-
sions. Let O be the smallest element (it is not necessary
an empty set) and I is the greatest element (I is the
universe V). An element p is an atom, if p O, and there
is no element x such that p x O. Each atom p is an
intersection of some elementary granules. Let S(b) be
the set of all atom pj such that pj b and s(b) be its
cardinality. From (Birkoff & MacLane, 1977, Chapter
11), we have
Proposition: Every b B can be expressed in the
form b = p1 . . . ps(b).
For convenience, let us define an operation of a
binary number x and a set S. We write S*x to mean the
following:
S*x = S, if x=1 and S
S*x = , if x=0 or S =
563
TEAM LinG

Let p1, p2, . . ,pm be the set of all atoms in B. Then a
granular expression b can be expressed as
b=p1*x 1 . . . p m* xm .
and its cardinality can be expressed as
|b| = | p i |*xi
where | | is the cardinality of .
Main Theorem: Let s be the threshold. Then b is a
high frequency pattern, if
|b| = | pi |*xi s (*)
In applications, pis are readily computable; it is the
elementary granules of the intersection of all partitions
(defined by attributes); see the Table 1 and 2. So we only
need to find all binary solutions of xi. The generators of the
solution can be enumerated along the hyperplanes of the
inequalities of the constraints.
Observations
Theoretically, this is a remarkable theorem. It says all
possible high frequency patterns can be found by solving
linear inequalities. However, the practicality of the
main theorem is highly depended on the complexity of
the problem. If both | pi | and s are small, then the number
of solutions will be out of hands, simply due to the size of
the number. We would like to stress that the difficulty is
simply due to the size of possible solutions, not the
methodology. The result implies that the notion of high
frequency patterns may not be tight enough. At this
moment, (*) is useful only if the number of attributes under
considerations is small.
FUTURE TRENDS
Tighter Notion of Patterns
Let us consider the real world meaning of the patterns of
length 2, namely, (TWENTY, MAR) and (20, SCREW).
What does this subtuple (TWENTY, MAR) mean? 20
million dollar business on March? The last statement is not
the original meaning of the schema: Originally it means v1,
v5, v6 have 20 million dollar business and they were born in
March. This subtuple has no meaning on its own. On the
other hand, (20, SCREW) from K is a valid pattern (most of
screws have weight 20). In summary, we have
564
High Frequency Patterns in Data Mining
(TWENTY, MAR) from K has no meaning on its own,
(20, SCREW) from K has a valid meaning.
Let RW(K) be the Real World that K is representing.
The summary implies that the subtuple (TWENTY, MAR),
even though occurs very frequently in the table, there is
no real world event correspond to it. The data implies
that three entities v 1, v5, v6 have common properties
encoded by Twenty and Mar. In the table K, they are
naively summarized into one concept (TWENTY,
MAR). Unfortunately, in the real world RW(K), the
three occurrences of Twenty and Mar (from three
entities, v1, v 5, v 6) do not integrate into an appropriate
new concept (TWENTY, MAR). Such error occurs,
because high frequency is an inadequate or inaccurate
criterion. We need a tighter notion of patterns.
Semantic Oriented Data Mining
If we do know how to compute the semantics, then the
computation should tell us that the repeated two words
TWENTY and MAR can not be combined into a new
concept regardless of high repetitions, and should be
dropped out. So semantic oriented data mining is needed
(Lin & Louie. 2001, 2002). As ontology, semantic web,
and computing with words (semantic computing) are
heating up, it could be a right time to move onto seman-
tic oriented data mining.
New Notions of Patterns and
Algorithmic Information Theory
In (Lin, 1993), based on algorithmic information theory
or Kolmogorov complexity theory, we proposed that a
non-random (compressible string) is a string with pat-
terns and the shortest Turing machine that generates this
string is the pattern. We concluded, then, that a finite
sequence (a relational table is a finite sequence) with
long constant subsequences (the length of such constant
sequence is the support) is trivially compressible (hav-
ing a pattern). High frequency patterns are such patterns.
Taking the same thought, what would be the next less
trivial compressible finite sequences?
CONCLUSION
Our analysis on association mining seems fruitful: (1)
High frequency patterns are natural generalizations of
association rules. (2) All high frequency patterns (gener-
alized associations) can be found by solving linear in-
equalities. (3) High frequency patterns are rather lean in
semantics (Isomorphic Theorem). So semantic oriented
AM or new notion of patterns may be needed.
TEAM LinG
High Frequency Patterns in Data Mining
REFERENCES
Agrawal, R,. Imielinski, T., & Swami, A. (1993, June).
Mining association rules between sets of items in large
databases. In Proceeding of ACM-SIGMOD interna-
tional Conference on Management of Data (pp. 207-216),
Washington, DC.
Barr, A., & Feigenbaum, E.A. (1981). The handbook of
artificial intelligence. Willam Kaufmann.
Fayad, U.M., Piatetsky-Sjapiro, G., & Smyth, P. (1996).
From data mining to knowledge discovery: An overview.
In Fayard, Piatetsky-Sjapiro, Smyth, & Uthurusamy (Eds.),
Knowledge Discovery in Databases. AAAI/MIT Press.
Garcia-Molina, H., Ullman, J. D., & Widom, J. (2002).
Database systems-The complete book. Prentice Hall.
Lee, T.T. (1983). Algebraic theory of relational data-
bases. The Bell System Technical Journal, 62(10),3159-
3204.
Lin, T.Y. (1993). Rough patterns in data - Rough sets and
foundation of intrusion detection systems. Journal of
Foundation of Computer Science and Decision Sup-
port, 18(3-4), 225-241.
Lin, T.Y. (1996, July). The power and limit of neural
networks. In Proceedings of the 1996 Engineering
Systems Design and Analysis Conference, 7 (pp. 49-53),
Montpellier, France.
Lin, T.T. (2000). Data mining and machine oriented mod-
eling: A granular computing approach. Journal of Ap-
plied Intelligence, 13(2), 113-124.
Lin, T.Y., & Louie, E. (2001). Semantics oriented asso-
ciation rules. In 2002 World Congress of Computa-
tional Intelligence (pp. 956-961), Honolulu, Hawaii, May
12-17, (paper # 5754).
Louie, E., & Lin, T.Y. (2000, October). Finding associa-
tion rules using fast bit computation: Machine-oriented
modeling. In Z. Ras, & S. Ohsuga (Eds.), Foundations
of intelligent systems, Lecture Notes in Artificial Intelli-
gence #1932, 486-494, Springer-Verlag. 12th Interna-
tional Symposium on Methodologies for Intelligent Sys-
tems, Charlotte, NC.
Park, J., & Sandberg, I.W. (1991). Universal approximation
using radial-basis-function networks. Neural Computa-
tion, 3, 246-257.
Pawlak, Z. (1982). Rough sets. International Journal
of Computer and Information Sciences, 11, 341-356.
Pawlak Z. (1991). Rough sets. Theoretical aspects of
reasoning about data. Kluwer Academic Publishers. 0
Ullman, J. (1988). Principles of database and knowledge-
base systems,vol. I. Computer Science Press.
Ullman, J. (1989). Principles of database and knowledge-
base systems, vol. II. Computer Science Press.
KEY TERMS
Algorithmic Information Theory: Absolute infor-
mation theory based on Kolmogorov complexity theory.
Association (Undirected Association Rule): A
subtuple of a bag relation whose support is greater than
a given threshold.
Bag Relation: A relation that permits repetition of
tuples
Computing with Symbols: The interpretations of
the symbols are not participating in the formal data
processing or computing,
Computing with Words: In this article, computing
with words means one form of formal data processing or
computing, in which the interpretations of the symbols
do participate. L.A .Zadeh uses this term in a much
deeper way.
Deductive Data Mining: A data mining methodol-
ogy that requires to list explicitly the input data and
background knowledge. Roughly it treats data mining as
deductive science (axiomatic method)
Granulation and Partition: Partition is a decomposi-
tion of a set into a collection of mutually disjoint subsets.
Granulation is defined similarly, but allows the subsets to
be generalized subsets, such as fuzzy sets, and permits
the overlapping.
Kolmogorov Complexity of a String: The length
of shortest program that can generate the given string.
Semantic Primitive: It is a symbol that has interpre-
tation, but the interpretation is not implemented in the
system. So in automated computing, semantic primitive is
treated as symbols. However, in interactive computing; it
may be treated as a word (not necessary).
Support: Support is the percentage of the tuples in
which that subtuple occurs.
565
TEAM LinG
566
Homeland Security Data Mining and Link
Analysis
Bhavani Thuraisingham
The MITRE Corporation, USA
INTRODUCTION
Data mining is the process of posing queries to large
quantities of data and extracting information often pre-
viously unknown using mathematical, statistical, and
machine-learning techniques. Data mining has many
applications in a number of areas, including marketing
and sales, medicine, law, manufacturing, and, more re-
cently, homeland security. Using data mining, one can
uncover hidden dependencies between terrorist groups
as well as possibly predict terrorist events based on past
experience. One particular data-mining technique that is
being investigated a great deal for homeland security is
link analysis, where links are drawn between various
nodes, possibly detecting some hidden links.
This article provides an overview of the various
developments in data-mining applications in homeland
security. The organization of this article is as follows.
First, we provide some background on data mining and
the various threats. Then, we discuss the applications of
data mining and link analysis for homeland security.
Privacy considerations are discussed next as part of
future trends. The article is then concluded.
BACKGROUND
We provide background information on both data min-
ing and security threats.
Data Mining
Data mining is the process of posing various queries and
extracting useful information, patterns, and trends often
previously unknown from large quantities of data possi-
bly stored in databases. Essentially, for many organiza-
tions, the goals of data mining include improving mar-
keting capabilities, detecting abnormal patterns, and
predicting the future, based on past experiences and
current trends. There is clearly a need for this technol-
ogy. There are large amounts of current and historical
data being stored. Therefore, as databases become larger,
it becomes increasingly difficult to support decision
making. In addition, the data could be from multiple
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
sources and multiple domains. There is a clear need to
analyze the data to support planning and other functions
of an enterprise.
Some of the data-mining techniques include those
based on statistical reasoning techniques, inductive logic
programming, machine learning, fuzzy sets, and neural
networks, among others. The data-mining problems in-
clude classification (finding rules to partition data into
groups), association (finding rules to make associations
between data), and sequencing (finding rules to order
data). Essentially, one arrives at some hypotheses, which
is the information extracted from examples and patterns
observed. These patterns are observed from posing a
series of queries; each query may depend on the re-
sponses obtained to the previous queries posed.
Data mining is an integration of multiple technolo-
gies. These include data management, such as database
management, data warehousing, statistics, machine learn-
ing, decision support, and others, such as visualization
and parallel computing. There is a series of steps involved
in data mining. These include getting the data organized
for mining, determining the desired outcomes to mining,
selecting tools for mining, carrying out the mining, prun-
ing the results so that only the useful ones are considered
further, taking actions from the mining, and evaluating the
actions to determine benefits. There are various types of
data mining. By this we do not mean the actual techniques
used to mine the data, but what the outcomes will be.
These outcomes also have been referred to as data-
mining tasks. These include clustering, classification
anomaly detection, and forming associations.
While several developments have been made, there
also are many challenges. For example, due to the large
volumes of data, how can the algorithms determine
which technique to select and what type of data mining
to do? Furthermore, the data may be incomplete and/or
inaccurate. At times, there may be redundant informa-
tion, and at times, there may not be sufficient informa-
tion. It is also desirable to have data-mining tools that
can switch to multiple techniques and support multiple
outcomes. Some of the current trends in data mining
include mining Web data, mining distributed and hetero-
geneous databases, and privacy-preserving data mining,
where one ensures that one can get useful results from
mining and at the same time maintain the privacy of
TEAM LinG
Homeland Security Data Mining and Link Analysis
individuals (Berry & Linoff; Han & Kamber, 2000;
Thuraisingham, 1998).
Security Threats
Security threats have been grouped into many catego-
ries (Thuraisingham, 2003). These include informa-
tion-related threats, where information technologies
are used to sabotage critical infrastructures, and non-
information-related threats, such as bombing buildings.
Threats also may be real-time threats and non-real-time
threats. Real-time threats are threats where attacks have
timing constraints associated with them, such as build-
ing X will be attacked within three days. Non-real-time
threats are those threats that do not have timing con-
straints associated with them. Note that non-real-time
threats could become real-time threats over time.
Threats also include bioterrorism, where biological
and possibly chemical weapons are used to attack, and
cyberterrorism, where computers and networks are at-
tacked. Bioterrorism could cost millions of lives, and
cyberterrorism, such as attacks on banking systems, could
cost millions of dollars. Some details on the threats and
countermeasures are discussed in various texts (Bolz,
2001). The challenge is to come up with techniques to
handle such threats. In this article, we discuss data-
mining techniques for security applications.
MAIN THRUST
First, we will discuss data mining for homeland secu-
rity. Then, we will focus on a specific data-mining
technique called link analysis for homeland security.
An aspect of homeland security is cyber security. There-
fore, we also will discuss data mining for cyber security.
Applications of Data Mining for
Homeland Security
Data-mining techniques are being examined extensively
for homeland security applications. The idea is to gather
information about various groups of people and study
their activities and determine if they are potential ter-
rorists. As we have stated earlier, data-mining outcomes
include making associations, linking analyses, forming
clusters, classification, and anomaly detection. The tech-
niques that result in these outcomes are techniques
based on neural networks, decisions trees, market-bas-
ket analysis techniques, inductive logic programming,
rough sets, link analysis based on graph theory, and
nearest-neighbor techniques. The methods used for data
mining include top-down reasoning, where we start with
a hypothesis and then determine whether the hypothesis
is true, or bottom-up reasoning, where we start with 0
examples and then come up with a hypothesis
(Thuraisingham, 1998). In the following, we will exam-
ine how data-mining techniques may be applied for
homeland security applications. Later, we will examine
a particular data-mining technique called link analysis
(Thuraisingham, 2003).
Data-mining techniques include techniques for mak-
ing associations, clustering, anomaly detection, predic-
tion, estimation, classification, and summarization. Es-
sentially, these are the techniques used to obtain the
various data-mining outcomes. We will examine a few
of these techniques and show how they can be applied to
homeland security. First, consider association rule min-
ing techniques. These techniques produce results, such
as John and James travel together or Jane and Mary
travel to England six times a year and to France three
times a year. Essentially, they form associations be-
tween people, events, and entities. Such associations
also can be used to form connections between different
terrorist groups. For example, members from Group A
and Group B have no associations, but Groups A and B
have associations with Group C. Does this mean that
there is an indirect association between A and C?
Next, let us consider clustering techniques. Clusters
essentially partition the population based on a charac-
teristic such as spending patterns. For example, those
living in the Manhattan region form a cluster, as they
spend over $3,000 on rent. Those living in the Bronx
from another cluster, as they spend around $2,000 on
rent. Similarly, clusters can be formed based on terror-
ist activities. For example, those living in region X
bomb buildings, and those living in region Y bomb
planes.
Finally, we will consider anomaly detection tech-
niques. A good example here is learning to fly an air-
plane without wanting to learn to take off or land. The
general pattern is that people want to get a complete
training course in flying. However, there are now some
individuals who want to learn to fly but do not care about
take off or landing. This is an anomaly. Another example
is John always goes to the grocery store on Saturdays.
But on Saturday, October 26, 2002, he went to a fire-
arms store and bought a rifle. This is an anomaly and may
need some further analysis as to why he is going to a
firearms store when he has never done so before. Some
details on data mining for security applications have
been reported recently (Chen, 2003).
Applications of Link Analysis
Link analysis is being examined extensively for applica-
tions in homeland security. For example, how do we
567
TEAM LinG

connect the dots describing the various events and make
links and connections between people and events? One
challenge to using link analysis for counterterrorism is
reasoning with partial information. For example, agency
A may have a partial graph, agency B another partial
graph, and agency C a third partial graph. The question is
how do you find the associations between the graphs,
when no agency has the complete picture? One would
ague that we need a data miner that would reason under
uncertainty and be able to figure out the links between
the three graphs. This would be the ideal solution, and the
research challenge is to develop such a data miner. The
other approach is to have an organization above the three
agencies that will have access to the three graphs and
make the links.
The strength behind link analyses is that by visualiz-
ing the connections and associations, one can have a
better understanding of the associations among the vari-
ous groups. Associations such as A and B, B and C, D and
A, C and E, E and D, F and B, and so forth can be very
difficult to manage, if we assert them as rules. However,
by using nodes and links of a graph, one can visualize the
connections and perhaps draw new connections among
different nodes. Now, in the real world, there would be
thousands of nodes and links connecting people, groups,
events, and entities from different countries and conti-
nents as well as from different states within a country.
Therefore, we need link analysis techniques to deter-
mine the unusual connection, such as a connection be-
tween G and P, for example, which is not obvious with
simple reasoning strategies or by human analysis.
Link analysis is one of the data-mining techniques
that is still in its infancy. That is, while much has been
written about techniques such as association rule min-
ing, automatic clustering, classification, and anomaly
detection, very little material has been published on link
analysis. We need interdisciplinary researchers such as
mathematicians, computational scientists, computer sci-
entists, machine-learning researchers, and statisticians
working together to develop better link analysis tools.
Applications of Data Mining for Cyber
Security
Data mining also has applications in cyber security,
which is an aspect of homeland security. The most promi-
nent application is in intrusion detection. For example,
our computers and networks are being intruded by unau-
thorized individuals. Data-mining techniques, such as
those for classification and anomaly detection, are being
used extensively to detect such unauthorized intrusions.
For example, data about normal behavior is gathered, and
when something occurs out of the ordinary, it is flagged
as an unauthorized intrusion. Normal behavior could be
568
Homeland Security Data Mining and Link Analysis
that Johns computer is never used between 2:00 A .M.
and 5:00 A.M.. When Johns computer is in use at 3:00
A .M ., for example, then this is flagged as an unusual
pattern.
Data mining is also being applied to other applica-
tions in cyber security, such a auditing. Here again, data
on normal database access is gathered, and when some-
thing unusual happens, then this is flagged as a possible
access violation. Digital forensics is another area where
data mining is being applied. Here again, by mining the
vast quantities of data, one could detect the violations
that have occurred. Finally, data mining is being used
for biometrics. Here, pattern recognition and other
machine-learning techniques are being used to learn
the features of a person and then to authenticate the
person, based on the features.
FUTURE TRENDS
While data mining has many applications in homeland
security, it also causes privacy concerns. This is be-
cause we need to collect all kinds of information about
people, which causes private information to be di-
vulged. Privacy and data mining have been the subject of
much debate during the past few years, although some
early discussions also have been reported
(Thuraisingham, 1996).
One promising direction is privacy-preserving data
mining. The challenge here is to carry out data mining but
at the same time ensure privacy. For example, one could
use randomization as a technique and give out approxi-
mate values instead of the actual values. The challenge is
to ensure that the approximate values are still useful.
Many papers on privacy-preserving data mining have
been published recently (Agrawal & Srikant, 2000).
CONCLUSION
This article has discussed data-mining applications in
homeland security. Applications in national security
and cyber security both are discussed. We first pro-
vided an overview of data mining and security threats
and then discussed data-mining applications. We also
emphasized a particular data-mining techniquelink
analysis. Finally, we discussed privacy-preserving data
mining.
It is only during the past three years that data mining
for security applications has received a lot of attention.
Although a lot of progress has been made, there is also
a lot of work that needs to be done. First, we need to
have a better understanding of the various threats. We
need to determine which data-mining techniques are
TEAM LinG
Homeland Security Data Mining and Link Analysis
applicable to which threats. Much research is also needed
on link analysis. To develop effective solutions, data-
mining specialists have to work with counter-terrorism
experts. We also need to motivate the tool vendors to
develop tools to handle terrorism.
NOTE
The views and conclusions expressed in this article are those
of the author and do not reflect the policies of the MITRE
Corporation or of the National Science Foundation.
REFERENCES
Agrawal, R, & Srikant, R. (2000). Privacy-preserving
data mining. Proceedings of the ACM SIGMOD Con-
ference, Dallas, Texas.
Berry, M., & Linoff, G. (1997). Data mining tech-
niques for marketing, sales, and customer support.
New York: John Wiley.
Bolz, F. (2001). The counterterrorism handbook: Tac-
tics, procedures, and techniques. CRC Press.
Chen, H. (Ed.). (2003). Proceedings of the 1st Confer-
ence on Security Informatics, Tucson, Arizona.
Han, J., & Kamber, M. (2000). Data mining, concepts
and techniques. CA: Morgan Kaufman.
Thuraisingham, B. (1996). Data warehousing, data min-
ing and security. Proceedings of the IFIP Database
Security Conference, Como, Italy.
Thuraisingham, B. (1998). Data mining: Technologies,
techniques, tools and trends, FL: CRC Press.
Thuraisingham, B. (2003). Web data mining technolo-
gies and their applications in business intelligence 0
and counter-terrorism. FL: CRC Press.
KEY TERMS
Cyber Security: Techniques used to protect the
computer and networks from the threats.
Data Management: Techniques used to organize,
structure, and manage the data, including database man-
agement and data administration.
Digital Forensics: Techniques to determine the
root causes of security violations that have occurred in
a computer or a network.
Homeland Security: Techniques used to protect a
building or an organization from threats.
Intrusion Detection: Techniques used to protect
the computer system or a network from a specific
threat, which is unauthorized access.
Link Analysis: A data-mining technique that uses
concepts and techniques from graph theory to make
associations.
Privacy: Process of ensuring that information
deemed personal to an individual is protected.
Privacy-Preserving Data Mining: Data-mining
techniques that extract useful information but at the
same time ensure the privacy of individuals.
Threats: Events that disrupt the normal operation of
a building, organization, or network of computers.
569
TEAM LinG
570
Humanities Data Warehousing
Janet Delve
University of Portsmouth, UK
INTRODUCTION
Data Warehousing is now a well-established part of the
business and scientific worlds. However, up until re-
cently, data warehouses were restricted to modeling
essentially numerical data examples being sales fig-
ures in the business arena (e.g. Wal-Marts data ware-
house) and astronomical data (e.g. SKICAT) in scien-
tific research, with textual data providing a descriptive
rather than a central role. The lack of ability of data
warehouses to cope with mainly non-numeric data is
particularly problematic for humanities1 research uti-
lizing material such as memoirs and trade directories.
Recent innovations have opened up possibilities for
non-numeric data warehouses, making them widely ac-
cessible to humanities research for the first time. Due
to its irregular and complex nature, humanities research
data is often difficult to model and manipulating time
shifts in a relational database is problematic as is fitting
such data into a normalized data model. History and
linguistics are exemplars of areas where relational data-
bases are cumbersome and which would benefit from
the greater freedom afforded by data warehouse dimen-
sional modeling.
BACKGROUND
Hudson (2001, p. 240) declared relational databases to be
the predominant software used in recent, historical re-
search involving computing. Historical databases have
been created using different types of data from diverse
countries and time periods. Some databases are modest
and independent, others part of a larger conglomerate
like the North Atlantic Population Project (NAPP)
project that entails integrating international census data.
One issue that is essential to good database creation is
data modeling; which has been contentiously debated
recently in historical circles.
When reviewing relational modeling in historical
research, (Bradley, 1994) contrasted straightforward
business data with incomplete, irregular, complex- or
semi-structured historical data. He noted that the rela-
tional model worked well for simply-structured busi-
ness data, but could be tortuous to use for historical
data. (Breure, 1995) pointed out the advantages of in-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
putting data into a model that matches it closely, some-
thing that is very hard to achieve with the relational
model. (Burt2 & James, 1996) considered the relative
freedom of using source-oriented data modeling
(Denley, 1994) as compared to relational modeling
with its restrictions due to normalization (which splits
data into many separate tables), and highlighted the
possibilities of data warehouses. Normalization is not
the only hurdle historians encounter when using the
relational model.
Date and time fields provide particular difficulties:
historical dating systems encompass a number of dif-
ferent calendars, including the Western, Islamic, Revo-
lutionary and Byzantine. Historical data may refer to
the first Sunday after Michaelmas, requiring calcula-
tion before a date may be entered into a database. Unfor-
tunately, some databases and spreadsheets cannot handle
dates falling outside the late 20th century. Similarly, for
researchers in historical geography, it might be neces-
sary to calculate dates based on the local introduction of
the Gregorian calendar, for example. These difficulties
can be time-consuming and arduous for researchers.
Awkward and irregular data with abstruse dating systems
thus do not fit easily into a relational model that does not
lend itself to hierarchical data. Many of these problems
also occur in linguistics computing.
Linguistics is a data-rich field, with multifarious
forms for words, multitudinous rules for coding sounds,
words and phrases, and also numerous other parameters
- geography, educational and social status. Databases
are used for housing many types of linguistic data from
a variety of research domains - phonetics, phonology,
morphology, syntax, lexicography, computer-assisted
learning (CAL), historical linguistics and dialectology.
Data integrity and consistency are of utmost importance
in this field. Relational DataBase Management Systems
(RDBMSs) are able to provide this, together with pow-
erful and flexible search facilities (Nerbonne, 1998).
Bliss and Ritter (2001) discussed the constraints
imposed on them when using the rigid coding structure
of the database developed to house pronoun systems
from 109 languages. They observed that coding intro-
duced interpretation of data and concluded that design-
ing a typological database is not unlike trying to fit a
square object into a round hole. Linguistic data is highly
variable, database structures are highly rigid, and the
two do not always fit. Brown (2001) outlined the fact
TEAM LinG
Humanities Data Warehousing
that different database structures may reflect a particular
linguistic theory, and also mentioned the trade-off be-
tween quality and quantity in terms of coverage.
The choice of data model thus has a profound effect
on the problems that can be tackled and the data that can
be interrogated. For both historical and linguistic re-
search, relational data modeling using normalization
often appears to impose data structures which do not fit
naturally with the data and which constrain subsequent
analysis. Coping with complicated dating systems can
also be very problematic. Surprisingly, similar difficul-
ties have already arisen in the business community, and
have been addressed by data warehousing.
MAIN THRUST
Data Warehousing in the Business
Context
Data warehouses came into being as a response to the
problems caused by large, centralized databases which
users found unwieldy to query. Instead, they extracted
portions of the databases which they could then control,
resulting in the spider-web problem where each de-
partment produces queries from its own, uncoordinated
extract database (Inmon, 2001, 99. 6-14). The need was
thus recognized for a single, integrated source of clean
data to serve the analytical needs of a company.
A data warehouse can provide answers to a com-
pletely different range of queries than those aimed at a
traditional database. Using an estate agency as a typical
business, the type of question their local databases
should be able to answer might be How many three-
bedroomed properties are there in the Botley area up to
the value of 150,000? The type of over-arching ques-
tion a business analyst (and CEOs) would be interested
in might be of the general form Which type of property
sells for prices above the average selling price for
properties in the main cities of Great Britain and how
does this correlate to demographic data? (Begg &
Connolly, 2004, p. 1154). To trawl through each local
estate agency database and corresponding local county
council database, then amalgamate the results into a
report would take a long time and a lot of resources. The
data warehouse was created to answer this type of need.
Basic Components of a Data
Warehouse
Inmon (2002, p. 31), the father of data warehousing,
defined a data warehouse as being subject-oriented,
integrated, non-volatile and time-variant. Emphasis
is placed on choosing the right subjects to model as
opposed to being constrained to model around applica- 0
tions. Data warehouses do not replace databases as such
- they co-exist alongside them in a symbiotic fashion.
Databases are needed both to serve the clerical commu-
nity who answer day-to-day queries such as what is A.R.
Smiths current overdraft? and also to feed a data
warehouse. To do this, snapshots of data are extracted
from a database on a regular basis (daily, hourly and in
the case of some mobile phone companies almost real-
time). The data is then transformed (cleansed to ensure
consistency) and loaded into a data warehouse. In addi-
tion, a data warehouse can cope with diverse data sources,
including external data in a variety of formats and sum-
marized data from a database. The myriad types of data
of different provenance create an exceedingly rich and
varied integrated data source opening up possibilities
not available in databases. Thus all the data in a data
warehouse is integrated. Crucially, data in a warehouse
is not updated - it is only added to, thus making it non-
volatile, which has a profound effect on data modeling,
as the main function of normalization is to obviate
update anomalies. Finally, a data warehouse has a time
horizon (that is contains data over a period) of five to ten
years, whereas a database typically holds data that is
current for two to three months.
Data Modeling in a Data Warehouse
Dimensional Modeling
There is a fundamental split in the data warehouse com-
munity as to whether to construct a data warehouse from
scratch, or to build them via data marts. A data mart is
essentially a cut-down data warehouse that is restricted
to one department or one business process. Inmon (2001,
p. 142) recommended building the data warehouse first,
then extracting the data from it to fill up several data marts.
The data warehouse modeling expert Kimball (2002)
advised the incremental building of several data marts
that are then carefully integrated into a data warehouse.
Whichever way is chosen, the data is normally modeled
via dimensional modeling. Dimensional models need to
be linked to the companys corporate ERD (Entity Re-
lationship Diagram) as the data is actually taken from
this (and other) source(s). Dimensional models are
somewhat different from ERDs, the typical star model
having a central fact table surrounded by dimension
tables. Kimball (2002, pp. 16-18) defined a fact table as
the primary table in a dimensional model where the
numerical performance measurements of the business
are storedSince measurement data is overwhelmingly
the largest part of any data mart, we avoid duplicating it
in multiple places around the enterprise. Thus the fact
table contains dynamic numerical data such as sales
571
TEAM LinG

quantities and sales and profit figures. It also contains
key data in order to link to the dimension tables. Dimen-
sion tables contain the textual descriptors of the busi-
ness process being modeled and their depth and breadth
define the analytical usefulness of the data warehouse.
As they contain descriptive data, it is assumed they will
not change at the same rapid rate as the numerical data in
the fact table that will certainly change every time the
data warehouse is refreshed. Dimension tables typically
have 50-100 attributes (sometimes several hundreds)
and these are not usually normalized. The data is often
hierarchical in the tables and can be an accurate reflec-
tion of how data actually appears in its raw state (Kimball
2002, pp. 19-21). There is not the need to normalize as data
is not updated in the data warehouse, although there are
variations on the star model such as the snowflake and
starflake models which allow varying degrees of normal-
ization in some or all of their dimension tables. Coding is
disparaged due to the long-term view that definitions may
be lost and that the dimension tables should contain the
fullest, most comprehensible descriptions possible
(Kimball 2002, p. 49). The restriction of data in the fact table
to numerical data has been a hindrance to academic com-
puting. However, Kimball has recently developed factless
fact tables (Kimball, 2002) which do not contain measure-
ments, thus opening the door to a much broader spectrum
of possible data warehouses.
Applying the Data Warehouse
Architecture to Historical and Linguistic
Research
One of the major advantages of data warehousing is the
enormous flexibility in modeling data. Normalization is
no longer an automatic straightjacket and hierarchies can
be represented in dimension tables. The expansive time
dimension (Kimball, 2002, p. 39) is a welcome by-product
of this modeling freedom, allowing country-specific calen-
dars, synchronization across multiple time zones and the
inclusion of multifarious time periods. It is possible to add
external data from diverse sources and summarized data
from the source database(s). The data warehouse is built
for analysis which immediately makes it attractive to hu-
manities researchers. It is designed to continuously re-
ceive huge volumes (terabytes) of data, but is sensitive
enough to cope with the idiosyncrasies of geographic
location dimensions within GISs (Kimball, 2002, p. 227).
Additionally a data warehouse has advanced indexing
facilities that make it desirable for those controlling vast
quantities of data. With a data warehouse it is theoretically
possible to publish the right data that has been collected
from a variety of sources and edited for quality and con-
sistency. In a data warehouse all data is collated so a
variety of different subsets can be analyzed whenever
572
Humanities Data Warehousing
required. It is comparatively easy to extend a data ware-
house and add material from a new source. The data
cleansing techniques developed for data warehousing
are of interest to researchers, as is the tracking facility
afforded by the meta data manager (Begg & Connolly,
2004, p. 1169-1170).
In terms of using data warehouses off the shelf,
some humanities research might fit into the numerical
fact topology, but some might not. The factless fact
table has been used to create several American univer-
sity data warehouses, but expertise in this area would
not be as widespread as that with normal fact tables. The
whole area of data cleansing may perhaps be daunting
for humanities researchers (as it is to those in indus-
try). Ensuring vast quantities of data is clean and con-
sistent may be an unattainable goal for humanities
researchers without recourse to expensive data cleans-
ing software. The data warehouse technology is far
from easy and is based on having existing databases to
extract from, hence double the work. It is unlikely that
researchers would be taking regular snapshots of their
data, as occurs in industry, but they could equate to data
sets taken at different periods of time to data ware-
house snapshots (e.g. 1841 census, 1861 census).
Whilst many data warehouses use familiar WYSIWYGs
and can be queried with SQL-type commands, there is
undeniably a huge amount to learn in data warehousing.
Nevertheless, there are many areas in both linguistics
and historical research where data warehouses may
prove attractive.
FUTURE TRENDS
Data Warehouses and Linguistics
Research
The problems related by Bliss and Ritter (2001) con-
cerning rigid relational data structures and pre-coding
problems would be alleviated by data warehousing.
Brown (2001) outlined the dilemma arising from the
alignment of database structures with particular lin-
guistic theories, and also the conflict of quality and
quantity of data. With a data warehouse there is room
for both vast quantities of data and a plethora of detail.
No structure is forced onto the data so several theoreti-
cal approaches can be investigated using the same data
warehouse. Dalli (2001) observed that many linguistic
databases are standalone with no hope of
interoperability. His proffered solution to create an
interoperable database of Maltese linguistic data in-
volved an RDBMS and XML. Using data warehouses to
store linguistic data should ensure interoperability.
There is growing interest in corpora databases, with the
TEAM LinG
Humanities Data Warehousing
recent dedicated conference at Portsmouth, November
2003. Teich, Hansen and Fankhauser drew attention to the
multi-layered nature of corpora and speculated as to how
multi-layer corpora can be maintained, queried and ana-
lyzed in an integrated fashion. A data warehouse would
be able to cope with this complexity.
Nerbonne (1998) alluded to the importance of co-
ordinating the overwhelming amount of work being
done and yet to be done. Kretzschmar (2001) delin-
eated the challenge of preservation and display for
massive amounts of survey data. There appears to be
many linguistics databases containing data from a range
of locations/countries. For example, ALAP, the Ameri-
can Linguistic Atlas Project; ANAE, the Atlas of North
American English (part of the TELSUR Project); TDS,
the Typological Database System containing European
data; AMPER, the Multimedia Atlas of the Romance
Languages. Possible research ideas for the future may
include a broadening of horizons - instead of the empha-
sis on individual database projects, there may develop an
integrated warehouse approach with the emphasis on
larger scale, collaborative projects. These could com-
pare different languages or contain many different types
of linguistic data for a particular language, allowing for
new orders of magnitude analysis.
Data Warehouses and Historical
Research
There are inklings of historical research involving data
warehousing in Britain and Canada. A data warehouse of
current census data is underway at the University of
Guelph, Canada and the Canadian Century Research
Infrastructure aims to house census data from the last
100 years in data marts constructed using IBM software
at several sites based in universities across the country.
At the University of Portsmouth, UK, a historical data
warehouse of American mining data is under construc-
tion using Oracle Warehouse Builder (Delve, Healey, &
Fletcher, 2004). These projects give some idea of the
scale of project a data warehouse can cope with that is,
really large country-/state-wide problems. Following
these examples, it would be possible to create a data
warehouse to analyze all British censuses from 1841 to
1901 (approximately 108 bytes of data). Data from a
variety of sources over time such as hearth tax, poor
rates, trade directories, census, street directories, wills
and inventories, GIS maps for a city such as Winchester
could go into a city data warehouse. Such a project is
under active consideration for Oslo, Norway. Similarly,
a Voting data warehouse could contain voting data poll
book data and rate book data up to 1870 for the whole
country. A Port data warehouse could contain all data
from portbooks for all British ports together with yearly
trade figures. Similarly a Street directories data ware-
house would contain data from this rich source for 0
whole country for the last 100 years. Lastly, a Taxation
data warehouse could afford an overview of taxation of
different types, areas or periods. 19th century British
census data does not fit into the typical data warehouse
model as it does not have the numerical facts to go into
a fact table, but with the advent of factless fact tables a
data warehouse could now be made to house this data.
The fact that some institutions have Oracle site licenses
opens to way for humanities researchers with Oracle
databases to use Oracle Warehouse Builder as part of
the suite of programs available to them. These are
practical project suggestions which would be impos-
sible to construct using relational databases, but which,
if achieved, could grant new insights into our history.
Comparisons could be made between counties and cit-
ies and much broader analysis would be possible than
has previously been the case.
CONCLUSION
The advances made in business data warehousing are
directly applicable to many areas of historical and lin-
guistics research. Data warehouse dimensional model-
ing would allow historians and linguists to model vast
amounts of data on a countrywide basis (or larger),
incorporating data from existing databases and other
external sources. Summary data could also be included,
and this would all lead to a data warehouse containing
more data than is currently possible, plus the fact that the
data would be richer than in current databases due to the
fact that normalization is no longer obligatory. Whole
data sources could be captured, and more post-hoc analy-
sis would result. Dimension tables particularly lend them-
selves to hierarchical modeling, so data would not need
splitting into many tables thus forcing joins while query-
ing. The time dimension particularly lends itself to his-
torical research where significant difficulties have been
encountered in the past. These suggestions for historical
and linguistics research will undoubtedly resonate in
other areas of humanities research, such as historical
geography, and any literary or cultural studies involving
textual analysis (for example biographies, literary criti-
cism and dictionary compilation).
REFERENCES
Begg, C., & Connolly, T. (2004). Database systems.
Harlow: Addison-Wesley.
Bliss, & Ritter (2001). IRCS (Institute for Research
into Cognitive Science) Conference Proceedings. Re-
573
TEAM LinG

trieved from http://www.ldc.upenn.edu/annotation/da-
tabase/proceedings.html
Bradley, J. (1994). Relational database design. History
and Computing, 6(2), 71-84.
Breure, L. (1995). Interactive data entry. History and
Computing, 7(1), 30-49.
Brown. (2001). IRCS Conference Proceedings. Re-
trieved from http://www.ldc.upenn.edu/annotation/
database/proceedings.html
Burt2, J., & James, T. B. (1996). Source-Oriented Data
Processing: The triumph of the micro over the macro.
History and Computing, 8(3), 160-169.
Dalli. (2001). IRCS Conference Proceedings. Retrieved
from http://www.ldc.upenn.edu/annotation/database/
proceedings.html
Delve, J., Healey, R., & Fletcher, A. (2004, July).
Teaching data warehousing as a standalone unit using
Oracle Warehouse Builder. Proceedings of the British
Computer Society TLAD Conference, Edinburgh, UK.
Denley, P. (1994). Models, sources and users: Histori-
cal database design in the 1990s. History and Comput-
ing, 6(1), 33-43.
Hudson, P. (2000). History by numbers. An introduc-
tion to quantitative approaches. London: Arnold.
Inmon, W. H. (2002). Building the data warehouse.
New York: Wiley.
Kimball, R., & Ross, M. (2002). The data warehouse
toolkit. New York: Wiley.
Kretzschmar. (2001). IRCS Conference Proceedings.
Retrieved from http://www.ldc.upenn.edu/annotation/
database/proceedings.html
Nerbonne, J. (Ed.). (1998). Linguistic databases.
Stanford, CA: CSLI Publications.
SKICAT. Retrieved from http://www-aig.jpl.nasa.gov/
public/mls/home/fayyad/SKICAT-PR1-94.html
Teich, Hansen, & Fankhauser. (2001). IRCS Conference
Proceedings. Retrieved from http://www.ldc.upenn.edu/
annotation/database/proceedings.html
574
Humanities Data Warehousing
Wal-Mart. Retrieved from http://www.tgc.com/dsstar/99/
0824/100966.html
KEY TERMS
Data Modeling: The process of producing a model
of a collection of data which encapsulates its semantics
and hopefully its structure.
Dimension Table: Dimension tables contain the tex-
tual descriptors of the business process being modeled
and their depth and breadth define the analytical useful-
ness of the data warehouse.
Dimensional Model: The dimensional model is the
data model used in data warehouses and data marts, the
most common being the star schema, comprising a fact
table surrounded by dimension tables.
Fact Table: the primary table in a dimensional model
where the numerical performance measurements of the
business are stored (Kimball, 2002).
Factless Tact Table: A fact table which contains no
measured facts (Kimball 2002).
Normalization: The process developed by E.C. Codd
whereby each attribute in each table of a relational
database depend entirely on the key(s) of that table. As
a result, relational databases comprise many tables,
each containing data relating to one entity.
Snowflake Schema: A dimensional model having
some or all of the dimension tables normalized.
ENDNOTES
1
learning or literature concerned with human cul-
ture (Compact OED)
2
Now Delve
TEAM LinG

Hyperbolic Space for Interactive Visualization
Jrg Andreas Walter
University of Bielefeld, Germany
INTRODUCTION
For many tasks of exploratory data analysis, visualiza-
tion plays an important role. It is a key for efficient
integration of human expertise not only to include his
background knowledge, intuition and creativity, but also
his powerful pattern recognition and processing capa-
bilities. The design goals for an optimal user interaction
strongly depend on the given visualization task, but they
certainly include an easy and intuitive navigation with
strong support for the users orientation.
Since most of available data display devices are two-
dimensional paper and screens the following
problem must be solved: finding a meaningful spatial
mapping of data onto the display area. One limiting
factor is the restricted neighborhood around a point in
a Euclidean 2-D surface. Hyperbolic spaces open an
interesting loophole. The extraordinary property of ex-
ponential growth of neighborhood with increasing ra-
dius around all points enables us to build novel
displays and browsers. We describe a versatile hyper-
bolic data viewer for building data landscapes in a non-
Euclidean space, which is intuitively browseable with a
very pleasing focus and context technique.
BACKGROUND
The Hyperbolic Space H2
2,300 years ago, the Greek mathematician Euclid
founded his geometry on five axioms. The fifth, the
parallel axiom, appeared unnecessary to many of his
colleagues. And they tried hard to prove it derivable
without success. After almost 2,000 years Lobachevsky
(1793-1856), Bolyai (1802-1860), and Gauss (1777-
1855) negated the axiom and independently discovered
the non-Euclidean geometry. There exist only two ge-
ometries with constant non-zero curvature. Through our
sight of common spherical surfaces (e.g., earth, orange)
we are familiar with the spherical geometry and its
constant positive curvature. Its counterpart with con-
stant negative curvature is known as the hyperbolic
plane H2 (with analogous generalizations to higher di-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
575
0
mensions). Unfortunately, there is no good embedding
of the H2 in R3, which makes it harder to grasp the unusual
properties of the H2. Local patches resemble the situation
at a saddle point, where the neighborhood grows faster
than in flat space. Standard textbooks on Riemannian
geometry (see, e.g., Morgan, 1993) examine the relation-
ship and expose that the area a of a circle of radius r are
given by
a(r) = 4 sinh2(r/2) (1)
This bears two remarkable asymptotic properties: (i)
for small radius r the space looks flat since a(r) = r2.
(ii) For larger r both grow exponentially with the
radius. As observed in Lamping & Rao (1994, 1999) and
Lamping et al. (1995), this property makes the hyper-
bolic space ideal for embedding hierarchical structures
(since the number of leaves grows exponentially with
the tree depth). This led to the development of the
hyperbolic tree browser at Xerox Parc (today
starviewer: see www.inxight.com).
The question how effective is visualization and navi-
gation in the hyperbolic space was studied by Pirolli et
al. (2001). By conducting eye-tracker experiments they
found that the focus+context navigation can signifi-
cantly accelerate the information foraging. Risden et
al. (2000) compared traditional and hyperbolic brows-
ers and found significant improvement in task execution
time for this novel display type.
MAIN THRUST
In order to use the visualization potential of the H2 we
must solve two problems:
(P1) How can we accommodate the data in the
hyperbolic space, and
(P2) how to project the H2 onto a suitable display?
In the following the answers are described in reverse
sequence, first for the second problem (P2) and then
three principal techniques for P1, which are today avail-
able for different data types.
TEAM LinG

Figure 1. Regular H2 tessellation with equilateral triangles (here 8 triangles meet at each vertex). Three
snapshots of a simple focus transfer are visible. Note the circular appearance of lines in the Poincar disk PD
and the fish-eye-lens effect: the triangles in the center appear larger and take less space in regions further
away.
Poincare Poincare
Solution for P2: Poincar Disk PD
For practical and technological reasons most available
displays are flat. The perfect projection into the flat
display area should preserve length, area, and angles
(=form). But it lays in the nature of a curved space to resist
the attempt to simultaneously achieve these goals. Con-
sequently several projections or maps of the hyperbolic
space were developed, four are especially well examined:
(i) the Minkowski, (ii) the upper-half plane, (iii) the Klein-
Beltrami, and (iv) the Poincar or disk mapping. For our
purpose the latter is particularly suitable. Its main charac-
teristics are:
Display Compatibility: The infinite large area
of the H2 is mapped entirely into a circle, the
Poincar disk PD.
Circle Rim is Infinity : All remote points are
close to the rim, without touching it.
Focus+Context: The focus can be moved to each
location in H2, like a fovea. The zooming factor
is 0.5 in the center and falls (exponentially) off
with distance to the fovea. Therefore, the context
appears very natural. As more remote things are,
the less spatial representation is assigned in the
current display (compare Figure 1).
Lines Become Circles: All H2-lines appear as
circle arc segments of centered straight lines in
PD (both belong to the set of so-called general-
ized circles). There extensions cross the PD-rim
always perpendicular on both ends.
Conformal Mapping: Angles (and therefore
form) relations are preserved in PD, area and
length relations obviously not.
576
Hyperbolic Space for Interactive Visualization
Poincare
Regular Tessellations with triangles offer richer
possibilities than the R2. It turns out that there is an
infinite set of choices to tessellate H2: for any
integer n 7, one can construct a regular tessella-
tions in which n triangles meet at each vertex (in
contrast to the plane with allows only n=3,4,6 and
the sphere only n=3,4,5). Figure 1 depicts an ex-
amples for n=8.
Moving Around and Changing the Focus: For
changing the focus point in PD we need a transla-
tion operation, which can be bound to mouse click
and drag events. In the Poincar disk model the
Mbius transformation T(z) is the appropriate so-
lution. By describing the Poincar disk PD as the
unit circle in the complex plane, the isometric
transformations for a point zPD can be written as
z = T(z; c, ) = ( z+c)/(c* z+1), with | |=1,
|c|<1. (2)
Here the complex number describes a pure rota-
tion of PD around the origin 0 (the star * denotes
complex conjugation). The following translation by c
maps the origin to c and -c becomes the new center 0 (if
=1). The Mbius transformations are also called the
circle automorphies of the complex plane, since they
describe the transformations from circles to (general-
ized) circles. Here they serve to translate H2 straight
lines to lines both appearing as generalized circles in
the PD projection. For further details, see for example,
Lamping & Rao (1999) or Walter (2004).
Three Layout Techniques in H2
Now we turn to the question raised earlier: how to accom-
modate data in the hyperbolic space. In the following
TEAM LinG
Hyperbolic Space for Interactive Visualization
section three known layout techniques for the H2 are
shortly reviewed.
Hyperbolic Tree Layout (HTL) for Tree-
Like Graph Data
A first solution to this question for the case of acyclic, tree-
like graph data in H2 was provided by Lamping & Rao
(1994, 1999). Each tree node receives a certain open space
pie segment, where the node chooses the locations of its
siblings. For all its siblings i it calls recursively the layout-
routine after applying the Mbius transformation in order
to center i.
Tamara Munzner developed another graph layout al-
gorithm for the three-dimensional hyperbolic space
(1997). While she gains much more space for the layout,
the problem of more complex navigation (and viewport
control) in 3-D and, more serious, the problem of occlu-
sion appears.
The next two layout techniques are freed from the
requirement of hierarchical data.
Hyperbolic Self-Organizing Map (HSOM)
The standard Self-Organizing Map (SOM) algorithm is
used in many applications for learning and visualization
(Kohonen, 2001). Figure 2 illustrates the basic opera-
tion. The feature map is built by a lattice of nodes (or
formal neurons) a A, each with a reference vector or
prototype vector wa attached, projecting into the input
space X. The response of a SOM to an input vector x is
determined by the reference vector of wa of the discrete
best-matching unit (BMU) a BMU, that is, the node which
Figure 2. The Hyperbolic Self-Organizing Map is formed
by a grid of processing units, called neurons. Here the
usual rectangular grid is replaced by a part of a
tessellation grid seen in Figure 1.
a*
x The goal of the MDS algorithm is to find a spatial
w a*
Grid of
Neurons a
Input Space X
has its prototype vector wa closest to the given input a
BMU
= argmin a| wa - x |.
H
The distribution of the reference vectors wa, is itera-
tively adapted by a sequence of training vectors x. After
finding the aBMU all reference vectors are updated to-
wards the stimulus x: wa new:= wa old + h(d a, aBMU)(x - wa).
Here h(.) is a bell-shaped Gaussian centered at the BMU
and decaying with increasing distance d a, aBMU =|ga - g aBMU|
in the node ga. Thus, each node or neuron in the
neighborhood of the aBMU participates in the current learn-
ing step (as indicated by the gray shading in Figure 2).
This neighborhood cooperation in the adaptation
algorithm has important advantages: (i) it is able to
generate topological order between the wa, which means
that similar inputs are mapped to neighboring nodes;
(ii) As a result, the convergence of the algorithm can be
sped up by involving a whole group of neighboring
neurons in each learning step.
The structure of this neighborhood is essentially
governed by the structure of h(a, aBMU) = h(d a, aBMU)
therefore also called the neighborhood function.
While most learning and visualization applications choose
d a, aBMU as distances in an rectangular (2-D, 3-D) lattice
this can be generalized to the non-Euclidean case as
suggested by Ritter (1999). The core idea of the Hyper-
bolic Self-Organizing Map (HSOM) is to employ an H2-
grid of nodes. A particular convenient choice is to take the
ga in PD of a finite patch of the triangular tessellation grid
as displayed in Figure 2. The internode distance is com-
puted in the appropriate Poincar metric (see equation 4).
Hyperbolic Multidimensional Scaling
(HMDS)
Multidimensional scaling refers to a class of algo-
rithms for finding a suitable embedding of N objects in
a low dimensional usually Euclidean space given
only pairwise proximity values (in the following we
represent proximity as dissimilarity or distance values
between pairs of objects, mathematically written as
dissimilarity ij 0 between the i and j item). Often the
raw dissimilarity distribution is not suitable for the
low-dimensional embedding and an additional prepro-
cessing step is applied. We model it here as a mono-
tonic transformation D(.) of dissimilarities ij into
disparities Dij = D(ij).
representation xi of each object i in the M-dimensional
space, where the pair distances d ij = d(xi , xj) (in the
target space) match the disparities Dij as faithfully as
possible i j: Dij ij.
One of the most widely known MDS algorithms was
introduced by Sammon (1969). He formulates a mini-
577
TEAM LinG

mization problem of a cost function which sums over the
squares of disparities-distance misfits
E({xi}) = i=1N j>i wi j (ij - Dij)2. (3)
The factors wi j are introduced to weight the disparities
individually and also to normalize the cost function E() to
be independent to the absolute scale of the disparities Dij.
The set of xi is found by a gradient descent procedure,
minimizing iteratively the cost or stress function [see
Sammon (1969) or Cox (1994) for further details on this and
other MDS algorithms].
The recently introduced Hyperbolic Multi-Dimen-
sional Scaling (HMDS) combines the concept of MDS
and hyperbolic geometry (Walter & Ritter, 2002). In-
stead of finding a MDS solution in the low-dimensional
Euclidean RM and transferring it to the H2 (which can
not work well), the MDS formalism operates in the
hyperbolic space from the beginning. The key is to
replace the Euclidean distance in the target space by the
appropriate distance metric for the Poincar model.
dij = 2 arctanh [ | xi - xj | / |1- x i xj* | ] with xi , xj PD;
(4)
While the gradients ij,q/ xi, q required for the gradi-
ent descent are rather simple to compute for the Euclidean
geometry, the case becomes complex for HMDS, see
Walter (2002, 2004) for details.
Disparity preprocessing: Due to the non-lin-
earity of the distance function above, the prepro-
cessing function D(.) has more influence in H2.
Consider, for example, linear rescaling of the
dissimilarities Dij = dij. In the Euclidean case the
visual structure is not affected only magnified
by . In contrast in H2, a scales the distribution and
with it the amount of curvature felt by the data. The
optimal depends on the given task, the dataset,
and its dissimilarity structure. One way is to set
manually and choose a compromise between vis-
Table 1.
HTL
Data type
Acyclic graph data
Scaling #objects N O( N )
Layout
Recursive space
partitioning
New object
Partial re-layout
Result
Good for tree data
578
Hyperbolic Space for Interactive Visualization
ibility of the entire structure and space for naviga-
tion in the detail-rich areas.
Comparison: The following table compares the main
properties of the three available layout techniques.
All three techniques share the concept of spatial
proximity representing similarities in the data. In HTL
close relatives are directly connected. Skupin (2002)
pointed out that we humans learn early the usage and the
versatile concepts of maps and suggested to build dis-
plays implementing this underlying map metaphor. The
HSOM can handle many objects and can generate topic
maps, while the HMDS is ideal to smaller sets, pre-
sented on the object level (see below Figures 4 & 5).
Application Examples
Even though the look and feel of an interactive visualiza-
tion and navigation is hardly compressible to paper
format, we present some application screenshots of
visualization experiments. Figure 3 displays an applica-
tion of browsing image collections with an HMDS. Here
the direct image features based on color are employed
to define the concept of similarity.
Figures 4 and 5 depict the hybrid combination of
HSOM and HMDS for an application from the field of
text mining of newsfeed articles. The similarity concept
is based on semantic information gained via a standard
bag-of-words model of the unstructured text, here
Reuters news articles [see Walter et al. (2003) for
further details].
FUTURE TRENDS
A closer look at the comparative table above suggests a
hybrid architecture of techniques. This includes the com-
bination of HSOM and HMDS for a two-stage navigation
and retrieval process. This embraces a coarse grain theme
map (e.g., with the HSOM) and a detailed map using
HSOM HMDS
Dissimilarity, distance
Vectorial data
data
O( N ) O( N 2 )
Assignment to grid of Spatial arrangement
neurons with topographic resembles similarity
ordering structure of the data
Map to BMU (possibly Iterate cost minimization
adapt) procedure
Handles large data Good for up to a few
collections; coarse topic / hundred object; detail
theme maps level map
TEAM LinG
Hyperbolic Space for Interactive Visualization
Figure 3. Snapshot of an Hyperbolic Image Viewer using
a color distance metric for pairwise dissimilarity
definition for 100 pictures. Note the dynamic zooming of
images is dependent on the distance to the current focus
point.
Figure 5. Screenshot of the HMDS visualizing all
documents in the node previously selected in the HSOM
(direction 5o clock). The news headlines where enabled
for the articles cluster now in focus They reveal that the
object group are semantically close and all related to a
strike in the oilseed company Cargill U.K.
H2-MDS Earn
Acquisition
Money-FX
Crude
Grain
Trade
Interest
Wheat
Ship
Corn
Other
CARGILL_U.K._STRIKE_TALKS_POSTPONED-486
CARGILL_U.K._STRIKE_TALKS_POSTPONED_TILL_MONDAY-1966
CARGILL_STRIKE_TALKS_CONTINUING_TODAY-5833
CARGILL_U.K._STRIKE_TALKS_TO_RESUME_THIS_AFTERNOO
CARGILL_U.K._STRIKE_TALKS_BREAK_OFF_WITHOUT_RESULT-12425
CARGILL_U.K._STRIKE_TALKS_TO_RESUME_TUESDAY-
CARGILL_U.K._STRIKE_TALKS_TO_CONTINUE_MONDAY-3710
Figure 4. A HSOM projection of a large collection of
newswire articles (Reuter-21578) forms semantically 0
related category clusters as seen by the glyphs indicating
the dominant out-of-band category label of the news
objects gathered by the HSOM in each node. The
similarity of objects is here derived from the angle of the
feature vectors in the bag-of-word or vector space
model of unstructured text [standard model in information
retrieval, after Salton (1988)]
HMDS, as suggested in the Hyperbolic Hybrid Data
Viewer prototype in Walter et al. (2003).
The HMDS allows to support different notions of
similarity and furthermore to dynamically modulate the
actual distance metric while observing the change in the
spatial arrangement of the object.
This feature is valuable in various multi-media appli-
cations with various natural and task-depended notions
of similarity. For example, when browsing a collection
of images, the similarity of objects can be based on textual
description, metadata, or image features, for example,
using color, shape, and texture. Due to the general data
type dissimilarity the dynamic modulation of the distance
metric is possible and can be controlled via the GUI or
other techniques like relevance feedback technique
proved successful in the field of information retrieval.
CONCLUSION
Information visualization can benefit from the use of the
hyperbolic space as a projection manifold. On the one
hand it gains the exponentially growing space around
each point, which provides extra space for compressing
object relationships. On the other hand the Poincar
model offers superb visualization and navigation prop-
erties, which were found to yield significant improve-
ment in task time compared to traditional browsing methods
579
TEAM LinG

(Risden et al., 2000; Pirolli et al., 2001). By simple mouse
interaction the focus can be transferred to any location of
interest. The core area close to the center of the Poincar
disk magnifies the data with a maximal zoom factor and
decreases gradually to the outer area. The object place-
holder (text box, image thumbnails, etc.) is scaled in propor-
tion. The fovea is an area with high resolution, while remote
areas are gradually compressed but are still visible as
context. Interestingly, this situation resembles the log-
polar density distribution of neurons in the retina, which
governs the natural resolution allocation in our visual
perception system.
REFERENCES
Cox, T., & Cox, M. (1994). Multidimensional scaling. In
Monographs on statistics and appied probability.
Chapman & Hall.
Kohonen, T. (2001). Self-organizing maps. Berlin:
Springer.
Lamping, J., & Rao, R. (1994). Laying out and visualizing
large trees using a hyperbolic space. In ACM Symp User
Interface Software and Technology (pp. 13-14).
Lamping, J., & Rao, R. (1999). The hyperbolic browser: A
Focus+Context technique for visualizing large hierar-
chies. In Readings in Information Visualization (pp.
382-408). Morgan Kaufmann.
Lamping, J., Rao, R., & Pirolli, P. (1995). A focus+context
technique based on hyperbolic geometry for viewing
large hierarchies. In ACM SIGCHI Conf Human Factors in
Computing Systems (pp. 401-408).
Morgan, F. (1993). Riemannian geometry: A beginners
guide. Jones and Bartlett Publishers.
Munzner, T. (1997). H3: Laying out large directed graphs
in 3d hyperbolic space. In Proceedings of IEEE Sympo-
sium on Information Visualization (pp. 2-10).
Pirolli, P., Card, S.K., & Van Der Wege. (2001). Visual
information foraging in a focus + context visualization.
In CHI (pp. 506-513).
Risden, K., Czerwinski, M., Munzner, T., & Cook, D.
(2000). An initial examination of ease of use for 2D and
3D information visualizations of Web content. Interna-
tional Journal of Human Computer Studies, 53(5), 695-
714.
Ritter, H. (1999). Self-organizing maps on non-Euclid-
ean spaces. In Kohonen Maps (pp. 97-110). Elsevier.
580
Hyperbolic Space for Interactive Visualization
Salton, G., & Buckley, C. (1988). Term-weighting ap-
proaches in automatic text retrieval. Information Process-
ing and Management, 5(24), 513-523.
Sammon, J. (1969). A non-linear mapping for data structure
analysis. IEEE Transactions Computers, 18, 401-409.
Skupin, A. (2002). A cartographic approach to visualizing
conference abstracts. In IEEE Computer Graphics and
Applications (pp. 50-58).
Walter, J. (2004). H-MDS: A new approach for interactive
visualization with multidimensional scaling in the hyperbolic
space. Information Systems, 29(4), 273-292.
Walter, J., Ontrup, J., Wessling, D., & Ritter, H. (2003).
Interactive visualization and navigation in large data
collections using the hyperbolic space. In IEEE Interna-
tional Conference on Data Mining (ICDM03) (pp. 355-
362).
Walter, J., & Ritter, H. (2002). On interactive visualization
of high-dimensional data using the hyperbolic plane. In
ACM SIGKDD International Conference of Knowledge
Discovery and Data Mining (pp. 123-131).
KEY TERMS
Focus+Context Technique: Allows to interactively
transfer the focus as desired while the context of the
region in focus remains in view with gradually degrading
resolution to the rim. In contrast allows the standard
zoom+panning technique to select the magnification
factor, trading detail and overview and involving harsh
view cut-offs.
H2: Two-dimensional hyperbolic space (plane).
H2DV: The Hybrid Hyperbolic Data Viewer incorpo-
rates a two stage browsing and navigation using the
HSOM for a coarse thematic mapping of large object
collections and the HMDS for detailed inspection of
smaller subsets in the object level.
HMDS: Hyperbolic Multi-Dimensional Scaling for
laying out objects in the H2 such that the spacial ar-
rangement resembles the dissimilarity structure of the
data as close as possible.
HSOM: Hyperbolic Self-Organizing Mapping. Exten-
sion of Kohonens topographic map, which offers the
exponential growth of neighborhood for the inner nodes.
Hyperbolic Geometry: Geometry with constant nega-
tive curvature in contrast to the flat Euclidean geometry.
TEAM LinG
Hyperbolic Space for Interactive Visualization

The space around each point grows exponentially and

allows better layout results than the corresponding flat 0
space.
Hyperbolic Tree Viewer: Visualizes tree-like graphs
in the Poincar mapping of the H2 or H3.
Poincar Mapping Of The H2: A conformal mapping
from the entire H2 into the unit circle in the flat R2 (disk
model)

581

TEAM LinG
582
Identifying Single Clusters in Large Data Sets
Frank Klawonn
University of Applied Sciences Braunschweig/Wolfenbuettel, Germany
Olga Georgieva
Institute of Control and System Research, Bulgaria
INTRODUCTION
Most clustering methods have to face the problem of
characterizing good clusters among noise data. The
arbitrary noise points that just do not belong to any class
being searched for are of a real concern. The outliers or
noise data points are data that severely deviate from the
pattern set by the majority of the data, and rounding and
grouping errors result from the inherent inaccuracy in
the collection and recording of data. In fact, a single
outlier can completely spoil the least squares (LS)
estimate and thus the results of most LS based cluster-
ing techniques such as the hard C-means (HCM) and the
fuzzy C-means algorithm (FCM) (Bezdek, 1999).
For these reasons, a family of robust clustering
techniques has emerged. There are two major families
of robust clustering methods. The first includes tech-
niques that are directly based on robust statistics. The
second family, assuming a known number of clusters, is
based on modifying the objective function of FCM in
order to make the parameter estimates more resistant to
the data noise. Among them one promising approach is
the noise clustering (NC) technique (Dave, 1991;
Klawonn, 2004). It maintains the principle of probabi-
listic clustering, but an additional noise cluster is intro-
duced. NC was developed and investigated in the context
of a variety of objective function-based clustering algo-
rithms and it has demonstrated its reliable ability to
detect clusters amongst noise data.
BACKGROUND
Objective function-based clustering aims at minimizing
an objective function that indicates a kind of fitting
error of the determined clusters to the given data. In this
objective function, the number of clusters has to be
fixed in advance. However, as the number of clusters is
usually unknown, an additional scheme has to be applied
to determine the number of clusters (Guo, 2002; Tao,
2002). The parameters to be optimized are the member-
ship degrees that are values of belonging of each data
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
point to every cluster, and the parameters, characterizing
the cluster, which finally determine the distance values. In
the simplest case, a single vector named cluster centre
(prototype) represents each cluster. The distance of a
data point to a cluster is simply the Euclidean distance
between the cluster centre and the corresponding data
point. More generally, one can use the squared inner-
product distance norm, in which by a norm inducing
symmetric and positive matrix different variances in the
directions of the coordinate axes of the data space are
accounted for. If the norm inducing matrix is the identity
matrix we obtain the standard Euclidean distance that
form spherical clusters. Clustering approaches that use
more complex cluster prototypes than only the cluster
centres, leading to adaptive distance measures, are for
instance the Gustafson-Kessel (GK) algorithm (Gustafson,
1979), the volume adaptation strategy (Hppner, 1999;
Keller, 2003) and the Gath-Geva (GG) algorithm (Gath,
1989). The latter one is not a proper objective function
algorithm, but corresponds to a fuzzified expectation
maximization strategy. No matter which kind cluster pro-
totype is used, the assignment of the data to the clusters
is based on the corresponding distance measure. In hard
clustering, a data object is assigned to the closest cluster,
whereas in fuzzy clustering a membership degree, that is,
a value that belongs to the interval [0,1] is computed. The
highest membership degree of a data corresponds to the
closest cluster.
Noise clustering has a benefit of the collection of
the noise points in one single cluster. A virtual noise
prototype with no parameters to be adjusted is intro-
duced that has always the same distance to all points in
the data set. The remaining clusters are assumed to be
the good clusters in the data set. The objective function
that considers the noise cluster is defined in the same
manner as the general scheme for the clustering mini-
mization functional. The main problem of NC is the
proper choice of the noise distance. If it is set too small,
then most of the points will get classified as noise
points, while for a large noise distance most of the
points will be classified into clusters other than the
noise cluster. A right selection of the distance will
result in a classification where the points that are actu-
TEAM LinG
Identifying Single Clusters in Large Data Sets
ally close enough to the good clusters will get correctly
classified into a good cluster, while the points that are away
from the good clusters will get classified into the noise
cluster. The detection of noise and outliers is a serious
problem and has been addressed in various approaches
(Leski, 2003; Yang, 2004; Zhang, 2003). However, all
these methods need complex additional computations.
MAIN THRUST
The purpose of most clustering algorithms is to parti-
tion a given data set into clusters. However, in data
mining tasks partitioning is not always the main goal.
Finding interesting patterns or substructures in a large
data set in the form of one or a few clusters that do not
cover or partition the data set completely is an impor-
tant issue in data mining. For this purpose a new cluster-
ing algorithm named noise clustering with one good
cluster, based on the noise clustering technique and able
to detect single clusters step-by-step in a given data set,
has been recently developed (Georgieva, 2004). In addi-
tion to identifying clusters step by step, as a side-effect
noise data are detected automatically.
The algorithm assesses the dynamics of the number
of points that are assigned to only one good cluster of
the data set by slightly decreasing the noise distance.
Starting with some large enough noise distance it is
decreased with a prescribed decrement till the reason-
able smallest distance is reached. The number of data
belonging to the good cluster is calculated for every
noise distance using the formula for the hard member-
ship values or fuzzy membership values, respectively.
Note that in this scheme only one cluster centre has to
be computed, which in the case of hard noise clustering
is the mean value of the good cluster data points and in
the case of fuzzy noise clustering is the weighted aver-
age of all data points. It is obvious that by decreasing the
noise distance a process of loosing data, that is,
separating them to the noise cluster, will begin. Con-
tinuing to decrease the noise distance, we will start to
separate points from the good cluster and add them to
the noise cluster. A further reduction of the noise
distance will lead to a decreasing amount of data in the
good cluster until the cluster will be entirely empty, as
all data will be assigned to the noise cluster. The de-
scribed dynamics can be illustrated in a curve viewing
the number of data points assigned to the good cluster
over the noise distance. In this curve a plateau will
indicate that we are in a phase of assigning proper noise
data to the noise cluster, whereas a strong slope means
that we actually loose data belonging to the good cluster
to the noise cluster.
Generally, a number of clusters with different shapes
and densities exist in large data sets and thus a compli- I
cated dynamics of the data assigned to the single cluster
will be observed. However, the smooth part of the
considered curve corresponds to the situation that a
relatively small amount of data is removed, which is
usually caused by loosing noise data. When we loose
data from a good cluster (with higher density than the
noise data), a small decrease of the noise distance will
lead to a large amount of data we loose to the noise
cluster, so that we will see a strong slope in our curve
instead of a plateau. Thus, a strong slope indicates that
at least one cluster is just removed and separated from
the (single) good cluster we try to find. By this way the
algorithm determines the number of clusters and de-
tects noise data. It does not depend on the initialisation,
so that the danger of converging into local optima is
further reduced compared to standard fuzzy clustering
(Hppner, 2003).
The described procedure is implemented in a cluster
identification algorithm that assesses the dynamics of
the quantity of the points assigned to the good cluster or
equivalently assigned to the noise cluster through the
slight decrease of the noise distance. By detecting the
strong slopes the algorithm separates one cluster at
every algorithm pass. A significant reduction of the
noise is achieved even in the first algorithm pass. The
clustering procedure is repeated by proceeding with a
smaller data set as the original one is reduced by the
identified noise data and data belonging to the already
identified cluster(s).
The curve that is determined by the dynamics of the
data assigned to the noise cluster is smoother in case of
fuzzy noise clustering compared to the hard clustering
case due to the fuzzily defined membership values. Also
local minima of this curve could be observed due to the
given freedom of the points to belong to both good and
noise cluster simultaneously. Fuzzy clustering can deal
better with complex data sets than hard clustering due to
the given relative degree of membership of a point to the
good cluster. However, for the same reason the amount
of the identified noise points is less than in the hard
clustering case.
FUTURE TRENDS
Whereas the standard clustering partitions the whole
data set, the main goal of the noise clustering with one
good cluster is to identify single clusters even in the
case when a large part of the data does not have any kind
of group structure at all. This will have a large benefit in
some application areas of cluster analysis like, for
583
TEAM LinG

instance, gene expression data and astrophysics data
analysis, where the ultimate goal is not to partition the
data, but to find some well-defined clusters that only cover
a small fraction of the data. By removing a large amount of
the noise data the obtained clusters are used to find some
interesting substructures in the data set.
One future improvement will lead to an extended pro-
cedure that first finds the location of the centres of the
interesting clusters using the standard Euclidean distance.
Then, the algorithm could be started again with this cluster
centres but using more sophisticated distance measures as
GK or volume adaptation strategy that can better adapt to
the shape of the identified cluster. For extremely large
data sets the algorithm can be combined with speed-up
techniques as the one proposed by Hppner (2002).
Another further application consists in incorporation
of the proposed algorithm as an initialisation step for
other more sophisticated clustering algorithm providing
the number of clusters and their approximate location.
CONCLUSION
A clustering algorithm, based on the principles of noise
clustering and able to find good clusters with variable
shape and density step-by-step, is proposed. It can be
applied to find just a few substructures, but also as an
iterative method to partition a data set including the
identification of the number of clusters and noise data.
The algorithm is applicable in terms of both hard and
fuzzy clustering. It automatically determines the number
of clusters, while in the standard objective function-
based clustering additional strategies have to be applied
in order to define the number of clusters.
The identification algorithm is independent of the
dimensionality of the considered problem. It does not
require comprehensive or costly computations and the
computation time increases linearly only with the num-
ber of data points.
REFERENCES
Bezdek, J.C., Keller, J., Krisnapuram, R., & Pal, N.R. (1999).
Fuzzy models and algorithms for pattern recognition and
image processing. Kluwer Academic Publishers.
Dave, R.N. (1991). Characterization and detection of
noise in clustering. Pattern Recognition Letters, 12,
657-664.
584
Identifying Single Clusters in Large Data Sets
Gath, I., & Geva, A.B. (1989). Unsupervised optimal fuzzy
clustering. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 7, 773-781.
Georgieva, O., & Klawonn, F. (2004). A clustering algo-
rithm for indentification of single clusters in large datat
sets. In Proceedings of East West Fuzzy Colloquium,
81(pp. 118-125), Sept. 8-10, Zittau, Germany.
Guo, P., Chen, C.L.P., & Lyu, M.R. (2002). Cluster number
selection for a small set of samples using the Bayesian
Ying-Yang model. IEEE Transactions on Neural Net-
works, 13, 757-763.
Gustafson, D., & Kessel, W. (1979). Fuzzy clustering with
a fuzzy covariance matrix. In Advances in fuzzy set theory
and applications (pp. 605-620). North-Holland.
Hppner, F. (2002). Speeding up fuzzy c-means: Using a
hierarchical data organisation to control the precision of
membership calculation. Fuzzy Sets and Systems, 12 (3),
365-378.
Hppner, F., & Klawonn, F. (2003). A contribution to
convergence theory of fuzzy c-means and derivatives.
IEEE Transactions on Fuzzy Systems, 11, 682-694.
Hppner, F., Klawonn, F., Kruse, R., & Runkler, T. (1999).
Fuzzy cluster analysis. Chichester: John Wiley & Sons.
Keller, A., & Klawonn, F. (2003). Adaptation of cluster
sizes in objective function based fuzzy clustering. In C.T.
Leondes (ed.), Intelligent Systems: Technology and
Applications IV. Database and Learning Systems (pp.
181-199). Boca Raton, FL: CRC Press.
Klawonn, F. (2004). Noise clustering with a fixed fraction
of noise. In A. Lotfi & J.M. Garibaldi (Eds.), Applications
and Science in Soft Computing (pp. 133-138). Berlin:
Springer.
Leski, J.M. (2003). Generalized weighted conditional fuzzy
clustering. IEEE Transactions on Fuzzy Systems, 11,
709-715.
Tao, C.W. (2002). Unsupervised fuzzy clustering with
multi-center clusters. Fuzzy Sets and Systems, 128,
305-322.
Yang, T.-N., & Wang, S.-D. (2004). Competitive algo-
rithms for the clustering of noisy data. Fuzzy Sets and
Systems, 141, 281-299.
Zhang, J.-S., & Leung, Y.-W. (2003). Robust clustering
by pruning outliers. IEEE Transactions on Systems,
Man and Cybernetics Part B, 33, 983-999.
TEAM LinG
Identifying Single Clusters in Large Data Sets
KEY TERMS
Cluster Analysis: Partition a given data set into
clusters where data assigned to the same cluster should
be similar, whereas data from different clusters should be
dissimilar.
Cluster Centres (Prototypes): Clusters in objective
function-based clustering are represented by prototypes
that define how the distance of a data object to the
corresponding cluster is computed. In the simplest case
a single vector represents the cluster, and the distance
to the cluster is the Euclidean distance between cluster
centre and data object.
Fuzzy Clustering: Cluster analysis where a data
object can have membership degrees to different clus-
ters. Usually it is assumed that the membership degrees
of a data object to all clusters sum up to one, so that a
membership degree can also be interpreted as the prob-
ability the data object belongs to the corresponding
cluster.
Hard Clustering: Cluster analysis where each data
object is assigned to a unique cluster. I
Noise Clustering: An additional noise cluster is in-
duced in objective function-based clustering to collect
the noise data or outliers. All data objects are assumed to
have a fixed (large) distance to the noise cluster, so that
only data far away from all other clusters will be assigned
to the cluster.
Objective Function-Based Clustering: Cluster analy-
sis is carried on minimizing an objective function that
indicates a fitting error of the clusters to the data.
Robust Clustering: Refers to clustering techniques
that behave robust with respect to noise, i.e.that is,
adding some noise to the data in the form changing the
values of the data objects slightly as well as adding some
outliers will not drastically influence the clustering
result.
585
TEAM LinG
586
Immersive Image Mining in Cardiology
Xiaoqiang Liu
Delft University of Technology, The Netherlands and Donghua University, China
Henk Koppelaar
Delft University of Technology, The Netherlands
Ronald Hamers
Erasmus Medical Thorax Center, The Netherlands
Nico Bruining
Erasmus Medical Thorax Center, The Netherlands
INTRODUCTION
Buried within the human body, the heart prohibits direct
inspection, so most knowledge about heart failure is
obtained by autopsy (in hindsight). Live immersive in-
spection within the human heart requires advanced data
acquisition, image mining and virtual reality techniques.
Computational sciences are being exploited as means to
investigate biomedical processes in cardiology.
IntraVascular UltraSound (IVUS) has become a clini-
cal tool in recent several years. In this immersive data
acquisition procedure, voluminous separated slice im-
ages are taken by a camera, which is pulled back in the
coronary artery. Image mining deals with the extraction of
implicit knowledge, image data relationships, or other
patterns not explicitly stored in the image databases (Hsu,
Lee, & Zhang, 2002). Human medical data are among the
most rewarding and difficult of all biological data to mine
and analyze, which has the uniqueness of heterogeneity
and are privacy-sensitive (Cios & Moore, 2002). The goals
of immersive IVUS image mining are providing medical
quantitative measurements, qualitative assessment, and
cardiac knowledge discovery to serve clinical needs on
diagnostics, therapies, and safety level, cost and risk
effectiveness etc.
BACKGROUND
Heart disease is the leading cause of death in industrial-
ized nations and is characterized by diverse cellular ab-
normalities associated with decreased ventricular func-
tion. At the onset of many forms of heart disease, cardiac
hypertrophy and ventricular changes in wall thickness or
chamber volume occur as a compensatory response to
maintain cardiac output. These changes eventually lead
to greater vascular resistance, chamber dilation, wall
fibrosis, which ultimately impair the ability of the ven-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
tricles to pump blood and lead to overt failure. To diag-
nose the many possible anomalies and heart diseases is
difficult because physicians cant literally see in the
human heart. Various data acquisition techniques have
been invented to partly remedy the lack of sight: non-
invasive inspection including CT (Computered Tomogra-
phy), Angiography, MRI (Magnetic Resonance Imag-
ing), ECG signals etc. These techniques do not take into
account crucial features of lesion physiology and vascu-
lar remodeling to really mine blood-plaque. IVUS, a mini-
mal-invasive technique, in which a camera is pulled back
inside the artery, and the resulting immersive tomographic
images are used to remodel the vessel. This remodeling
vessel and its virtual reality (VR) aspect offer interesting
future alternatives for mining these data to unearth anoma-
lies and diseases in the moving heart and coronary ves-
sels at earlier stage. It also serves in clinical trials to
evaluate results of novel interventional techniques, e.g.
local kill by heating cancerous cells via an electrical
current through a piezoelectric transducer as well as local
nano-technology pharmaceutical treatments. Figure 1
explains some aspects of IVUS technology.
However, IVUS images are more complicated than
medical data in general since they suffer from some arti-
facts during immersed data acquisition (Mintz et al., 2001):
Non-uniform rotational distortion and motion arti-
facts.
Ring-down, blood speckle, and near field artifacts.
Obliquity, eccentricity, and problems of vessel cur-
vature.
Problems of spatial orientation.
The second type of artifacts is treated by image pro-
cessing and therefore falls outside the scope of this paper.
The pumping heart, respiring lungs and moving immersed
catheter camera cause the other three types of artifacts.
These cause distortion on longitudinal position, x-y po-
TEAM LinG
Immersive Image Mining in Cardiology

Figure 1. IVUS immersive data acquisition, measurements and remodeling

I

A: IVUS catheter/endosonics and the wire. B: angiograophy of a contrast-filled section of coronary artery. C: two
IVUS cross-sectional images and some quantitative measurements on them. D: Virtual L-View of the vessel
reconstruction. E: Virtual 3D-Impression of the vessel reconstruction.

sition, and spatial orientation of the slices on the coronary
vessel. For example, it has been reported that more than
5 mm of longitudinal catheter motion relative to the vessel
may occur during one cardiac cycle (Winter et al., 2004),
when the catheter was pulled back at 0.5 mm/sec and the
non-gated samples were stored on S-VHS videotape at a
rate of 25 images/sec. Figure 2 explains the longitudinal
displacement caused by cardiac cycles during a camera
pullback in a segment of coronary artery. The catheter
position equals to the sum of the pullback distance and
the longitudinal catheter displacement. In F, the absolute
catheter positions of solid dots are in disorder, which will
cause a disordered sequence of camera images. The
consecutive image samples selected in relation to the
positions of the catheter relative to the coronary vessel
wall are highlighted in G. In conclusion, these samples
used for analysis are anatomically dispersed in space (III,
I, V, II, IV, and VI).

Figure 2. Trajectory position anomalies of the invasive camera

587

TEAM LinG
 Immersive Image Mining in Cardiology

The data of IVUS sources are voluminous separated
slices with artifacts, and the quantitative measurements,
the physicians interpretations are also essential compo-
nents to remodel the vessel, but the accompanying math-
ematical models are poorly characterized. Mining in these
datasets is more difficult and it inspires interdisciplinary
work in medicine, physics and computer science.
MAIN THRUST
Immersive Image Mining Processes in
Cardiology IVUS
IVUS image mining includes a series of complicated mining
procedures due to the complex immersive data: data recon-
ciliation, image mining, remodeling and VR display, and
knowledge discovery. Figure 3 shows the function-driven
processes.
The data reconciliation compensates or decreases ar-
tifacts to improve data, usually fused with non-invasive
inspection cardiology data such as angiography and ECG
etc. This data mining extracts features in the individual
dataset to use for data reconciliation.
Cardiac computation is taken on the reconciled indi-
vidual dataset. Quantitative measurement calculates fea-
tures such as lumen, atheroma, calcium and stent on every
slice. Vessel remodeling based on these measurements
forms a straight 3D volume, and the fusion with the pull-
back path determined from angiography yields an accurate
spatio-temporal (4D) model of the coronary vasculature.
This 4D model is used for further cardiac computation to
gain VR display and volume measurements on the vessel.
IVUS images are fundamentally different from histology
and cannot be used to detect and quantify specific
histologic contents directly. But based on quantitative
measurements and VR display, combined cardiac knowl-
edge, the qualitative assessment such as atheroma mor-
phology, unstable lesions and complications after inter-
vention may be gained semi-automatically or artificially
by the physicians.
These quantitative measurements and qualitative as-
sessments may be organized and stored in a database or
data warehouse. Statistics-based mining on colony data
and mining on individual history data lead to knowledge
discovery of heart diseases.
Individual Data Mining to Reconcile the
Data
Data reconciliation is the basis and most important step
to cardiac computation, and it is expecting more effective
methods to attack the data artifacts. Three types of
methods are applied: parsimonious data acquiring, data
fusion of invasive and non-invasive data, and hybrid
motion compensation.
Parsimonious Phase-Dependent Data Acquisition:
Cardiac knowledge dictates systolic/diastolic tim-
ing features. It has been suggested to select IVUS
images recorded in the end-diastolic phase of the
cardiac cycle, in which the heart is motionless and

Figure 3. Immersive IVUS image mining

L-Mode
Cardiac
Immersive 3D reconstruction
knowledge
image Picture in picture
acquisition Expert
knowledge
Knowledge Vessel remodeling
discovery & Visual display

Atheroma morphology
Qualitative
Unstable lesions
Data Cardiac assessments
Ruptured plaque
reconciliation computation
Complications after
Quantitative
intervention
measurements

Non-invasive Border Identification
data acquisition Lumen measurements
Calcium measurements

588

TEAM LinG
Immersive Image Mining in Cardiology
blood flow has ceased so that their influences on the
catheter can be neglected. Online ECG-gated pull-
back has been used to acquire phase-dependent
data but the technology is expensive and prolongs
the acquisition procedure. Instead, a retrospective
image-based gating mining method has been stud-
ied (Winter et al, 2004; Zhu, Oakeson, & Friedman,
2003). In this method, the different features are
mined from sampled IVUS images over time by
transforming the data with spectral analysis, to
discover the most prominent repetition frequencies
of appearance of these image features. From this
mining, the near images at the end-diastolic phases
can be inferred. The selection of images is parsimo-
nious: only about 5% of the dataset are selected,
wherein about 10% of the selections are
mispositioned.
Data Fusion: The motion vessel courses provide
helpful information to identify space and time of the
IVUS camera in the coronary of the moving heart.
Fusion of complementary information from two or
more differing modalities enhances insights into the
underlying anatomy and physiology. Combining
non-invasive data mining techniques to battle mea-
surement errors is a preferred method. The position-
ing for the camera could be remedied if from
angiograms the outer form of a vessel is available,
as a path-road for the camera. Fusing the route and
IVUS data, a simulator generates a VR reconstruc-
tion of the vessel (Wahle, Mitchell, Olszewski, Long,
& Sonka, 2000; Sarry & Boire, 2001; Ding & Fried-
man, 2000; Rotger, Radeva, Mauri, & Fernandez-
Nofrerias, 2002; Weichert, Wawro, & Wilke, 2004).
This should help to detect the absolute catheter
spatial positions and orientations, but usually the
routes are static and data are parsimonious, phase-
dependent, or without exhibiting the distortion.
There are few papers on the consecutive motion
tracking of the coronary tree (Chen & Carrol, 2003;
Shechter, Devernay, Coste-Maniere, Quyyumi, &
McVeigh, 2003), but they omitted the stretch of the
arteries that is an important property for accurate
positioning analyses.
Hybrid Motion Compensation: Physical prior knowl-
edge would predict space and time of the global
position of IVUS camera, if it would be fully mod-
eled. The many mechanical and electrical mecha-
nisms in a heart make a full model intractable, but if
most of its mechanisms could be dispensed with, the
prediction model would become considerably sim-
plified. A hybrid approach could solve the problem
if a full Courant-type model would be available,
coined by the term Computational Cardiology
(Cipra, 1999). An effective and pragmatic model is
futuristic yet. Timinger reconstructed the catheter I
position on 3D roadmap by a motion compensation
algorithm based on an affine model for compensat-
ing the respiratory motion and ECG gating method
for the catheter positions acquired using a magnetic
tracking system (Timinger, Krueger, Borgert, &
Grewer, 2004). Fusing the empirical features of the
coronary longitudinal movement with a motion com-
pensation model is a novel way to resolve the
longitudinal distortion of the IVUS dataset (Liu,
Koppelaar, Koffijberg, Bruining, & Hamers, 2004).
Cardiac Computation to Quantitative
and Qualitative Analysis
Cardiac computation aims at quantitative measurements,
remodeling and qualitative assessments on the vessel
and the lesion zones. The technologies of border detec-
tion of image processing and pattern recognition of the
vessel layers are important in the process of cardiac
calculation (Koning, Dijkstra, von Birgelen, Tuinenburg,
et al., 2002). For qualitative assessments, expert knowl-
edge of physicians must be fused within reasoning to get
the assessments.
Quantitative Measurements
Cross-Sectional Slices Calculation: The normal
coronary artery consists of the lumen surrounded
by intima, media, and adventitia of the vessel wall
(Halligan & Higano, 2003). The innermost layer
consists of a complex of three elements: intima,
atheroma (diseased arteries), and internal elastic
membrance. After gaining vessel layers and their
attributes through edge identification and pattern
recognition of image processing, every slice is cal-
culated separately and the measurements such as
lumen area, EEM (external elastic membrance) area,
and maximum atheroma thickness can be reported in
detail.
Vessel Remodeling and Virtual Display: Based on
the above calculation results, vessel layers includ-
ing plaques can be remodeled and visualized in
longitude. VR display is an important way to assist
inspecting the artery and the distribution of plaque
and lesions, to help navigate in guided surgery
facilities for minimally invasive surgery. For ex-
amples, L-Mode display sets of slices taken from a
single cut plane in longitude (Figure 1, panel D); 3D
reconstruction display a shaded or wire-frame im-
age of the vessel to give an entire view (Figure 1,
Panel E).
589
TEAM LinG

Derived Measurements: Calculating on the virtual
remodeled vessel, the derived measurements can be
obtained such as hemodynamics, length, volumes
of the vessel and special lesion zones.
Qualitative Assessments: IVUS images are funda-
mentally different from histology and cannot be
used to detect and quantify specific histologic con-
tents directly. However, based on quantitative mea-
surements and a virtual model, as well as combined
cardiac knowledge, the qualitative assessment such
as atheroma morphology, unstable lesions and com-
plications after intervention may be gained semi-
automatically or artificially to serve physicians.
Knowledge Discovery in Cardiology
Once a large amount of quantitative and qualitative fea-
tures are in hand, mining on these data can reveal
knowledge about the forming and regression of some
heart diseases, and can simulate the different stages of the
disease as well as assess surgical treatment procedure
and risk level.
Some papers are reported on medical knowledge dis-
covery in cardiology using data mining. Pressure calcu-
lation by using fluid dynamic equations on 3D IVUS
volumetric measurements predicts the physiologic sever-
ity of coronary lesions (Takayama & Hodgson, 2001).
Artificial intelligence methods: a structural description,
syntactic reasoning and pattern recognition method are
applied on angiography to recognize stenosis of coronary
artery lumen (Ogiela & Tadeusiewicz, 2002). Logical Analy-
sis of Data is a method based on combinatory, optimiza-
tion and theory of Boolean functions is used on 21
variables dataset to predict coronary risk, but these dataset
do not include any medical images information (Alexe,
2003). Mining in single proton emission computed tomog-
raphy images accompanied by clinical information and
physician interpretation using inductive machine learn-
ing and heuristic approaches to mimic cardiologists
diagnosis (Kurgan, Cios, Tadeusiewicz, Ogiela, &
Goodenday, 2001).
Immersive data mining or fusing the mined data with
other cardiac data is a challenge for improving medical
knowledge in cardiology. Mining will contribute to some
difficult cardiology applications, for example:
interventional procedures and healthcare; mining coro-
nary vessel movement anomalies; highlight local abnor-
mal cellular growth; accurate heart and virtual vessel
reconstruction; adjust ferro-electric materials to monitor
heart movement anomalies; prophylactic patient monitor-
ing etc. Some technicalities should be considered in this
mining field.
590
Immersive Image Mining in Cardiology
Volume and complexity in data: First, IVUS data
have a large volume in one acquisition procedure for
one person. Second, usually tracing a patient needs
a long time: more than several years. Third, the data
from patients maybe are incomplete. Finally,
immersive data acquisition is a complicated proce-
dure depending on body condition, immersive me-
chanical system, and even operating procedure.
The immersive complexity may even bias the histori-
cal data from the same heart.
Data standards and semantics: IVUS images, espe-
cially the qualitative assessments, need a consis-
tent standard and semantics to support mining.
IVUS documents (Mintz, et al., 2001) and DICOM
IVUS standards may be the base to follow. It is
necessary to consider the importance of relative
parameters, spatial positions, multi-interpretations
of image quantitative measurements, which also
need to be addressed in the IVUS standards.
Data fusion: Since heart diseases are complicated,
IVUS is one of the preferred diagnose tools, so
mining IVUS needs considering other clinical infor-
mation including physicians interpretation at the
same time.
FUTURE TRENDS
Immersive image mining in cardiology is a new challenge
for medical informatics. In mining our body physicians
thus inspire interdisciplinary work in medicine, physics
and computer science which will improve monitoring
heart data to many deep applications serving clinical
needs in diagnostics, therapies, safety level, cost and risk
effectiveness. For instance, in due course of time
nanotechnology will mature to a degree of immersive
medical mining equipment, physicians will directly con-
trol medicine by instruction via mobile communication
because of a transducer inside the human body.
CONCLUSION
Over the last several years, IVUS has developed into an
important clinical tool in the assessment of atherosclero-
sis. The limitations on data artifacts and difficult discern-
ing between entities with similar echodensities (Halligan
& Higano, 2003) are waiting for better solution. Hospitals
have stored a large amount of immersive data, which may
be mined for effective application in cardiac knowledge
discovery.
Data mining technologies play crucial roles in the
whole procedures in IVUS application: from data acquisi-
TEAM LinG
Immersive Image Mining in Cardiology
tion, data reconciliation, image processing, vessel remod-
eling and virtual display, knowledge discovery, disease
diagnosing, clinical treatment, etc. Reconciling coronary
longitudinal movement with a motion compensation model
is a novel way to resolve the longitudinal distortion of the
IVUS dataset. This fusion with other cardiac datasets and
online processing are very important for future applica-
tion, which means more effective and efficient mining
methods on the complicated datasets need to be studied.
REFERENCES
Alexe, S. (2003). Coronary risk prediction by logical analy-
sis of data. Annals of Operations Research, 119(1-4), 15-
42.
Chen, S-Y. J. & Carroll, J. D. (2003). Kinematic and defor-
mation analysis of 4-D coronary arterial trees recon-
structed from cine angiograms. IEEE Transactions on
Medical Imaging, 22(6), 710-720.
Cios, K. J. & Moore, W. (2002). Uniqueness of medical
data mining. Artificial Intelligence in Medicine Journal,
26(1-2), 1-24.
Cipra, B. A. (1999). Failure in sight for a mathematical
model of the heart. SIAM News, 32(8).
Ding, Z. & Friedman, M.H. (2000). Quantification of 3D
coronary arterial motion using clinical biplane
cineangiograms. The International Journal of Cardiac
Imaging, 16(5), 331-346.
Halligan, S. & Higano, S. T. (2003). Coronary assessment
beyond angiography. Applications in Imaging Cardiac
Interventions, 12, 29-35.
Hsu, W., Lee, M. L., & Zhang, J. (2002). Image mining:
Trends and developments. Journal of Intelligent Infor-
mation System: Special Issue on Multimedia Data Min-
ing, 19(1), 7-23.
Koning, G., Dijkstra, J., von Birgelen, C., Tuinenburg, J. C.
et al. (2002). Advanced contour detection for three-di-
mensional intracoronary ultrasound: A validationin
vitro and in vivo. The International Journal of Cardio-
vascular Imaging, 18, 235-248.
Kurgan, L. A., Cios, K. J., Tadeusiewicz, R., Ogiela, M., &
Goodenday, L. S. (2001). Knowledge discovery approach
to automated cardiac SPECT diagnosis. Artificial Intelli-
gence in Medicine, 23(2), 149-169.
Liu, X., Koppelaar, H., Koffijberg, H., Bruining, N., &
Hamers, R. (2004, October). Data reconciliation of immersive
heart inspection. Paper presented at the IEEE Interna-
tional Conference on Systems, Man and Cybernetics,
Hague, Netherlands. I
Mintz, G. S., Nissen, S. E., Anderson, W. D., Bailey, S. R.,
Erbel, R., Fitzgerald, P. J., Pinto, F. J., Rosenfield, K.,
Siegel, R. J., Tuzcu, E. M., & Yock, P. G. (2001). ACC clinical
expert consensus document on standards for the acqui-
sition, measurement and reporting of intravascular ultra-
sound studies: A report of the American college of cardi-
ology task force on clinical expert consensus documents.
Journal of the American College of Cardiology, 37,
14781492.
Ogiela, M. R. & Tadeusiewicz, R. (2002). Syntactic reason-
ing and pattern recognition for analysis of coronary artery
images. Artificial Intelligence in Medicine, 26, 145-159.
Rotger, B., Radeva, P., Mauri, J., & Fernandez-Nofrerias,
E. (2002). Internal and external coronary vessel images
registration. In M. T. Escrig, F. Toledo, & E. Golobardes
(Eds.), Topics in Artificial Intelligence, 5th Catalonian
Conference on AI, Castelln (pp. 408-418). Spain.
Sarry, L. & Boire, J. Y. (2001). Three-Dimensional tracking
of coronary arteries from biplane angiographic sequences
using parametrically deformationable models. IEEE Trans-
actions on Medical Imaging, 20(12), 1341-1351.
Shechter, G., Devernay, F., Coste-Maniere, E., Quyyumi,
A., & McVeigh, E. R. (2003). Three-Dimentional motion
tracking of coronary arteries in biplane cineangiograms.
IEEE Transaction on Medical Imaging, 2, 1-16.
Takayama, T. & Hodgson, J. M. (2001). Prediction of the
physiologic severity of coronary lesions using 3D IVUS:
Validation by direct coronary pressure measurements.
Catheterization and Cardiovascular Interventions,
53(1), 48-55.
Timinger, H., Krueger, S., Borgert, J., & Grewer, R. (2004).
Motion compensation for interventional navigation on
3D static roadmaps based on an affine model and gating.
Physics in Medicine and Biology, 49, 719-732.
Wahle A., Mitchell, S. C., Olszewski, M. E., Long R. M., &
Sonka, M. (2000). Accurate visualization and quantifica-
tion of coronary vasculature by 3-D/4-D fusion from
biplane angiography and intravascular ultrasound. EBiOS
2000, Biomonitoring and Endoscopy Technologies (pp.
144-155). Amsterdam NL, SPIE Europto, 4158.
Weichert, F., Wawro, M., & Wilke, C. (2004). A 3D com-
puter graphics approach to brachytherapy planning. The
International Journal of Cardiovascular Imaging, 20,
173-182.
Winter, S. A. de, Hamers, R. Degertekin, M., Tanabe, K.,
Lemos, P.A., Serruys, P.W., Roelandt, J. R. T. C., &
591
TEAM LinG

Bruining, N. (2004). Retrospective image-based gating of
intracoronary ultrasound images for improved quantita-
tive analysis: The intelligate method. Catheterization
and Cardiovascular Interventions, 61(1), 84-94.
Zhu, H., Oakeson, K. D., & Friedman, M. H. (2003). Re-
trieval of cardiac phase from IVUS sequences. In F. Wil-
liam, M.F. Walker, & Insana (Eds.), Medical Imaging
2003: Ultrasonic Imaging and Signal Processing, Pro-
ceedings of SPIE 5035 (pp. 135-146).
KEY TERMS
Cardiac Data Fusion: Fusion of complementary infor-
mation from two or more differing cardiac modalities
(IVUS, ECG, CT, physicians interpretation etc.) enhances
insights into the underlying anatomy and physiology.
Computational Cardiology: Using mathematic and
computer model to simulate the heart motion and its
properties as a whole.
Immersive IVUS Images: The real-time cross-sec-
tional images obtained from a pullback IntraVascular
592
Immersive Image Mining in Cardiology
UltraSound transducer in human arteries. The dataset is
usually a volume with artifacts caused by the complicated
immersed environments.
IVUS Data Reconciliation: Mining in the IVUS indi-
vidual dataset to compensate or decrease artifacts to get
improved data using for further cardiac calculation and
medical knowledge discovery.
IVUS Standards and Semantics: It refers to the stan-
dard and semantics of IVUS data and their medical quan-
titative measurements, qualitative assessments. The con-
sistent definition and description improve medical data
management and mining.
Medical Image Mining: It involves extracting the
most relevant image features into a form suitable for data
mining for medical knowledge discovery; or generating
image patterns to improve the accuracy of images re-
trieved from image databases.
Virtual Reality Vasculature Reconstruction: For
effective applications of intravascular analyses and
brachytherapy, reconstruct and visualize the vessel-walls
interior structure in a single 3D/4D model by fusing
invasive IVUS data and non-invasive angiography on?
TEAM LinG

Imprecise Data and the Data Mining Process
Marvin L. Brown
Grambling State University, USA
John F. Kros
East Carolina University, USA
INTRODUCTION
Missing or inconsistent data has been a pervasive prob-
lem in data analysis since the origin of data collection.
The management of missing data in organizations has
recently been addressed as more firms implement large-
scale enterprise resource planning systems (see Vosburg
& Kumar, 2001; Xu et al., 2002). The issue of missing
data becomes an even more pervasive dilemma in the
knowledge discovery process, in that as more data is
collected, the higher the likelihood of missing data
becomes.
The objective of this research is to discuss impre-
cise data and the data mining process. The article begins
with a background analysis, including a brief review of
both seminal and current literature. The main thrust of
the chapter focuses on reasons for data inconsistency
along with definitions of various types of missing data.
Future trends followed by concluding remarks com-
plete the chapter.
BACKGROUND
The analysis of missing data is a comparatively recent
discipline. However, the literature holds a number of
works that provide perspective on missing data and data
mining. Afifi and Elashoff (1966) provide an early
seminal paper reviewing the missing data and data min-
ing literature. Little and Rubins (1987) milestone work
defined three unique types of missing data mechanisms
and provided parametric methods for handling these
types of missing data. These papers sparked numerous
works in the area of missing data. Lee and Siau (2001)
present an excellent review of data mining techniques
within the knowledge discovery process. The refer-
ences in this section are given as suggested reading for
any analyst beginning their research in the area of data
mining and missing data.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
593
I
MAIN THRUST
The article focuses on the reasons for data inconsistency
and the types of missing data. In addition, trends regarding
missing data and data mining are discussed along with
future research opportunities and concluding remarks.
REASONS FOR DATA
INCONSISTENCY
Data inconsistency may arise for a number of reasons,
including:
Procedural Factors
Refusal of Response
Inapplicable Responses
These three reasons tend to cover the largest areas of
missing data in the data mining process.
Procedural Factors
Data entry errors are common and their impact on the
knowledge discovery process and data mining can gen-
erate serious problems. Inaccurate classifications, er-
roneous estimates, predictions, and invalid pattern rec-
ognition may also take place. In situations where data-
bases are being refreshed with new data, blank responses
from questionnaires further complicate the data mining
process. If a large number of similar respondents fail to
complete similar questions, the deletion or
misclassification of these observations can take the
researcher down the wrong path of investigation or lead
to inaccurate decision-making by end users.
Refusal of Response
Some respondents may find certain survey questions
offensive or they may be personally sensitive to certain
TEAM LinG

questions. For example, some respondents may have no
opinion regarding certain questions such as political or
religious affiliation. In addition, questions that refer to
ones education level, income, age or weight may be
deemed too private for some respondents to answer.
Furthermore, respondents may simply have insuffi-
cient knowledge to accurately answer particular ques-
tions. Students or inexperienced individuals may have
insufficient knowledge to answer certain questions (such
as salaries in various regions of the country, retirement
options, insurance choices, etc).
Inapplicable Responses
Sometimes questions are left blank simply because the
questions apply to a more general population rather than
to an individual respondent. If a subset of questions on
a questionnaire does not apply to the individual respon-
dent, data may be missing for a particular expected
group within a data set. For example, adults who have
never been married or who are widowed or divorced are
likely to not answer a question regarding years of marriage.
TYPES OF MISSING DATA
The following is a list of the standard types of missing
data:
Data Missing at Random
Data Missing Completely at Random
Non-Ignorable Missing Data
Outliers Treated as Missing Data
It is important for an analyst to understand the differ-
ent types of missing data before they can address the
issue. Each type of missing data is defined next.
[Data] Missing At Random (MAR)
Rubin (1978), in a seminal missing data research paper,
defined missing data as MAR when given the variables
X and Y, the probability of response depends on X but
not on Y. Cases containing incomplete data must be
treated differently than cases with complete data. For
example, if the likelihood that a respondent will provide
his or her weight depends on the probability that the
respondent will not provide his or her age, then the
missing data is considered to be Missing At Random
(MAR) (Kim, 2001).
594
Imprecise Data and the Data Mining Process
[Data] Missing Completely At Random
(MCAR)
Kim (2001), based on an earlier work, classified data as
MCAR when the probability of response [shows that]
independence exists between X and Y. MCAR data
exhibits a higher level of randomness than does MAR. In
other words, the observed values of Y are truly a random
sample for all values of Y, and no other factors included
in the study may bias the observed values of Y.
Consider the case of a laboratory providing the
results of a chemical compound decomposition test in
which a significant level of iron is being sought. If
certain levels of iron are met or missing entirely and no
other elements in the compound are identified to corre-
late then it can be determined that the identified or
missing data for iron is MCAR.
Non-Ignorable Missing Data
In contrast to the MAR situation where data missingness
is explained by other measured variables in a study; non-
ignorable missing data arise due to the data missingness
pattern being explainable and only explainable by
the very variable(s) on which the data are missing.
For example, given two variables, X and Y, data is
deemed Non-Ignorable when the probability of response
depends on variable X and possibly on variable Y. For
example, if the likelihood of an individual providing his
or her weight varied within various age categories, the
missing data is non-ignorable (Kim, 2001). Thus, the
pattern of missing data is non-random and possibly
predictable from other variables in the database.
In practice, the MCAR assumption is seldom met. Most
missing data methods are applied upon the assumption of
MAR. And in correspondence to Kim (2001), Non-Ignor-
able missing data is the hardest condition to deal with, but
unfortunately, the most likely to occur as well.
Outliers Treated As Missing Data
Many times it is necessary to classify these outliers as
missing data. Pre-testing and calculating threshold
boundaries are necessary in the pre-processing of data
in order to identify those values which are to be classi-
fied as missing. Data whose values fall outside of pre-
defined ranges may skew test results. Consider the case
of a laboratory providing the results of a chemical
compound decomposition test. If it has been predeter-
mined that the maximum amount of iron that can be
TEAM LinG
Imprecise Data and the Data Mining Process
contained in a particular compound is 500 parts/million,
then the value for the variable iron should never exceed
that amount. If, for some reason, the value does exceed
500 parts/million, then some visualization technique
should be implemented to identify that value. Those
offending cases are then presented to the end users.
COMMONLY USED METHODS OF
ADDRESSING MISSING DATA
Several methods have been developed for the treatment
of missing data. The simplest of these methods can be
broken down into the following categories:
Use Of Complete Data Only
Deleting Selected Cases Or Variables
Data Imputation
These categories are based on the randomness of the
missing data, and how the missing data is estimated and
used for replacement. Each category is discussed next.
Use of Complete Data Only
Use of complete data only is generally referred to as the
complete case approach and is readily available in all
statistical analysis packages. When the relationships
within a data set are strong enough to not be significantly
affected by missing data, large sample sizes may allow
for the deletion of a predetermined percentage of cases.
While the use of complete data only is a common ap-
proach, the cost of lost data and information when cases
containing missing value are simply deleted can be dra-
matic. Overall, this method is best suited to situations
where the amount of missing data is small and when
missing data is classified as MCAR.
Delete Selected Cases or Variables
The simple deletion of data that contains missing values
may be utilized when a non-random pattern of missing
data is present. However, it may be ill-advised to elimi-
nate ALL of the samples taken from a test. This method
tends to be the method of last choice.
Data Imputation Methods
A number of researchers have discussed specific impu-
tation methods. Seminal works include Little & Rubin
(1987), and Rubin (1978) has published articles regard-
ing imputation methodologies. Imputation methods are
procedures resulting in the replacement of missing
values by attributing them to other available data. This I
research investigates the most common imputation
methods including:
Case Deletion
Mean Substitution
Cold Deck Imputation
Hot Deck Imputation
Regression Imputation
These methods were chosen mainly as they are very
common in the literature and their ease and expediency
of application (Little & Rubin, 1987). However, it has
been concluded that although imputation is a flexible
method for handling missing-data problems it is not
without its drawbacks. Caution should be used when
employing imputation methods as they can generate
substantial biases between real and imputed data. None-
theless, imputation methods tend to be a popular method
for addressing the issue of missing data.
Case Deletion
The simple deletion of data that contains missing val-
ues may be utilized when a nonrandom pattern of miss-
ing data is present. Large sample sizes permit the
deletion of a predetermined percentage of cases, and/
or when the relationships within the data set are strong
enough to not be significantly affected by missing data.
Case deletion is not recommended for small sample
sizes or when the user knows strong relationships within
the data exist.
Mean Substitution
This type of imputation is accomplished by estimating
missing values by using the mean of the recorded or
available values. It is important to calculate the mean
only from valid responses that are chosen from a veri-
fied population having a normal distribution. If the data
distribution is skewed, the median of the available data
can be used as a substitute.
The main advantage of mean substitution is its ease
of implementation and ability to provide all cases with
complete information. Although a common imputation
method, three main disadvantages to mean substitution
do exist:
Understatement of variance
Distortion of actual distribution of values mean
substitution allows more observations to fall into
the category containing the calculated mean than
may actually exist
595
TEAM LinG
 Imprecise Data and the Data Mining Process

Depression of observed correlations due to the Once the most similar case has been identified, hot
repetition of a constant value deck imputation substitutes the most similar complete
cases value for the missing value. Since case two con-
Obviously, a researcher must weigh the advantages tains the value of 23 for item four, a value of 23 replaces
against the disadvantages before implementation. the missing data point for case three. The advantages of
hot deck imputation include conceptual simplicity, main-
Cold Deck Imputation tenance and proper measurement level of variables, and
the availability of a complete set of data at the end of the
Cold deck imputation methods select values or use imputation process that can be analyzed like any complete
relationships obtained from sources other than the cur- set of data. One of hot decks disadvantages is the difficulty
rent data. With this method, the end user substitutes a in defining what is similar, Hence, many different
constant value derived from external sources or from schemes for deciding on what is similar may evolve.
previous research for the missing values. It must be
ascertained by the end user that the replacement value Regression Imputation
used is more valid than any internally derived value.
Unfortunately, feasible values are not always provided Regression Analysis is used to predict missing values
using cold deck imputation methods. Many of the same based on the variables relationship to other variables in
disadvantages that apply to the mean substitution method the data set. Single and/or multiple regression can be
apply to cold deck imputation. Cold deck imputation used to impute missing values. The first step consists of
methods are rarely used as the sole method of imputa- identifying the independent variables and the dependent
tion and instead are generally used to provide starting variable. In turn, the dependent variable is regressed on
values for hot deck imputation methods. the independent variables. The resulting regression equa-
tion is then used to predict the missing values. Table 2
Hot Deck Imputation displays an example of regression imputation.
From the table, twenty cases with three variables
Generally speaking, hot deck imputation replaces miss- (income, age, and years of college education) are listed.
ing values with values drawn from the next most similar Income contains missing data and is identified as the
case. The implementation of this imputation method dependent variable while age and years of college edu-
results in the replacement of a missing value with a value cation are identified as the independent variables.
selected from an estimated distribution of similar re- The following regression equation is produced for
sponding units for each missing value. In most instances, the example
the empirical distribution consists of values from re-
sponding units. For example, Table 1 displays a data set
containing missing data. Table 2. Illustration of regression imputation
From Table 1, it is noted that case three is missing
Years of College Regression
data for item four. In this example, case one, two, and Case Income Age Education Prediction
four are examined. Using hot deck imputation, each of 1 $95,131.25 26 4 $96,147.60
the other cases with complete data is examined and the 2 $108,664.75 45 6 $104,724.04
value for the most similar case is substituted for the 3 $98,356.67 28 5 $98,285.28
missing data value. Case four is easily eliminated, as it 4 $94,420.33 28 4 $96,721.07
5 $104,432.04 46 3 $100,318.15
has nothing in common with case three. Case one and
6 $97,151.45 38 4 $99,588.46
two both have similarities with case three. Case one has 7 $98,425.85 35 4 $98,728.24
one item in common whereas case two has two items in 8 $109,262.12 50 6 $106,157.73
common. Therefore, case two is the most similar to 9 $95,704.49 45 3 $100,031.42
case three. 10 $99,574.75 52 5 $105,167.00
11 $96,751.11 30 0 $91,037.71
12 $111,238.13 50 6 $106,157.73
Table 1. Illustration of hot deck imputation: incomplete 13 $102,386.59 46 6 $105,010.78
data set 14 $109,378.14 48 6 $105,584.26
15 $98,573.56 50 4 $103,029.32
16 $94,446.04 31 3 $96,017.08
Case Item 1 Item 2 Item 3 Item 4
17 $101,837.93 50 4 $103,029.32
1 10 22 30 25
2 23 20 30 23 18 ??? 55 6 $107,591.43
3 25 20 30 ??? 19 ??? 35 4 $98,728.24
4 11 25 10 12 20 ??? 39 5 $101,439.40

596

TEAM LinG
Imprecise Data and the Data Mining Process
y = $79,900.95 + $268.50(age) + (integration of domain-specific logic into data mining
$2,180,97(years of college education)
Predictions of income can be made using the regres-
sion equation and the right-most column of the table
displays these predictions. For cases eighteen, nine-
teen, and twenty, income is predicted to be $107,591.43,
$98,728.24, and $101,439.40, respectfully. An advan-
tage to regression imputation is that it preserves the
variance and covariance structures of variables with
missing data.
Although regression imputation is useful for simple
estimates, it has several inherent disadvantages:
Regression imputation reinforces relationships
that already exist within the data as the method is
utilized more often, the resulting data becomes
more reflective of the sample and becomes less
generalizable
Understates the variance of the distribution
An implied assumption that the variable being
estimated has a substantial correlation to other
attributes within the data set
Regression imputation estimated value is not con-
strained and therefore may fall outside predeter-
mined boundaries for the given variable
In addition to these points, there is also the problem
of over-prediction. Regression imputation may lead to
over-prediction of the models explanatory power. For
example, if the regression R 2 is too strong,
multicollinearity most likely exists. Otherwise, if the
R2 value is modest, errors in the regression prediction
equation will be substantial. Overall, regression imputa-
tion not only estimates the missing values but also
derives inferences for the population.
FUTURE TRENDS
Poor data quality has plagued the knowledge discovery
process and all associated data mining techniques. Fu-
ture data mining systems should be sensitive to noise
and have the ability to deal with all types of data pollu-
tion, both internally and in conjunction with end users.
Systems should still produce the most significant find-
ings from the data set possible even if noise is present.
As data mining continues to evolve and mature as a
viable business tool, it will be monitored to address its
role in the technological life cycle. Tools for dealing
with missing data will grow from being used as a hori-
zontal solution (not designed to provide business-spe-
cific end solutions) into a type of vertical solution
solutions). As gigabyte-, terabyte-, and petabyte-size I
data sets become more prevalent in data warehousing
applications, the issue of dealing with missing data will
itself become an integral solution for the use of such
data rather than simply existing as a component of the
knowledge discovery and data mining processes (Han &
Kamber, 2001).
Although the issue of statistical analysis with miss-
ing data has been addressed since the early 1970s, the
advent of data warehousing, knowledge discovery, data
mining and data cleansing has pushed the concept of
dealing with missing data into the limelight. Brown &
Kros (2003) provide an overview of the trends, tech-
niques, and impacts of imprecise data on the data mining
process related to the k-Nearest Neighbor Algorithm,
Decision Trees, Association Rules, and Neural Networks.
CONCLUSION
It can be seen that future generations of data miners will
be faced with many challenges concerning the issues of
missing data. This article gives a background analysis
and brief literature review of missing data concepts. The
authors addressed reasons for data inconsistency and
methods for addressing missing data. Finally, the au-
thors offered their opinions on future developments and
trends on issues expected to face developers of knowl-
edge discovery software and the needs of end users
when confronted with the issues of data inconsistency
and missing data.
REFERENCES
Afifi, A., & Elashoff, R. (1966). Missing observations
in multivariate statistics I: Review of the literature.
Journal of the American Statistical Association, 61,
595-604.
Brown, M.L., & Kros, J.F. (2003). The impact of missing
data on data mining. In J. Wang (Ed.), Data mining
opportunities and challenges (pp. 174-198). Hershey,
PA: Idea Group Publishing.
Han, J., & Kamber, M. (2001). Data mining: Concepts
and techniques. San Francisco: Academic Press.
Kim, Y. (2001). The curse of the missing data. Re-
trieved from http://209.68.240.11:8080/2ndMoment/
978476655/addPostingForm/
Lee, S., & Siau, K. (2001). A review of data mining
techniques. Industrial Management & Data Systems,
101(1), 41-46.
597
TEAM LinG

Little, R., & Rubin, D. (1987). Statistical analysis with
missing data. New York: Wiley.
Rubin, D. (1978). Multiple imputations in sample surveys:
A phenomenological Bayesian approach to nonresponse.
In Imputation and Editing of Faulty or Missing Survey
Data (pp. 1-23). Washington, DC: U.S. Department of
Commerce.
Vosburg, J., & Kumar, A. (2001). Managing dirty data in
organizations using ERP: Lessons from a case study.
Industrial Management & Data Systems, 101(1), 21-31.
Xu, H., Horn Nord, J., Brown, N., & Nord, G.D. (2002).
Data quality issues in implementing an ERP. Industrial
Management & Data Systems, 102(1), 47-58.
KEY TERMS
[Data] Missing Completely at Random (MCAR): When
the observed values of a variable are truly a random
sample of all values of that variable (i.e., the response
exhibits independence from any variables).
598
Imprecise Data and the Data Mining Process
Data Imputation: The process of estimating missing
data of an observation based on the valid values of other
variables.
Data Missing at Random (MAR): When given the
variables X and Y, the probability of response depends
on X but not on Y.
Inapplicable Responses: Respondents omit answer
due to doubts of applicability.
Knowledge Discovery Process: The overall pro-
cess of information discovery in large volumes of ware-
housed data.
Non-Ignorable Missing Data: Arise due to the data
missingness pattern being explainable, non-random, and
possibly predictable from other variables.
Procedural Factors: Inaccurate classifications of
new data, resulting in classification error or omission.
Refusal of Response: Respondents outward omission
of a response due to personal choice, conflict, or inexpe-
rience.
TEAM LinG
 599

Incorporating the People Perspective into I

Data Mining
Nilmini Wickramasinghe
Cleveland State University, USA

INTRODUCTION namely by incorporating a people-based perspective into

Todays economy is increasingly based on knowledge
and information (Davenport & Grover, 2001). Knowl-
edge is now recognized as the driver of productivity and
economic growth, leading to a new focus on the roles of
information technology and learning in economic per-
formance. Organizations trying to survive and prosper in
such an economy are turning their focus to strategies,
processes, tools, and technologies that can facilitate the
creation of knowledge. A vital and well-respected tech-
nique in knowledge creation is data mining, which en-
ables critical knowledge to be gained from the analysis
of large amounts of data and information. Traditional
data mining and the KDD process (knowledge discovery
in data bases) tends to view the knowledge product as a
homogeneous product. Knowledge, however, is a multi-
faceted construct, drawing upon various philosophical
perspectives including Lockean/Leibnitzian and
Hegelian/Kantian, exhibiting subjective and objective
aspects, as well as having tacit and explicit forms
(Nonaka, 1994; Alavi & Leidner, 2001; Schultze &
Leidner, 2002; Wickramasinghe et al., 2003). The the-
sis of this article is that taking a broader perspective of
the resultant knowledge product from the KDD process,
the traditional KDD process, not only provides a more
complete and macro perspective on knowledge creation
but also a more balanced approach, which in turn serves
to enhance the knowledge base of an organization and
facilitates the realization of effective knowledge. The
implications for data mining are clearly far-reaching and
are certain to help organizations more effectively realize
the full potential of their knowledge assets, improve the
likelihood of using/reusing the created knowledge, and
thereby enables them to be well positioned in todays
knowledgeable economy.
BACKGROUND
Knowledge Creation through Data
Mining and the KDD Process
KDD, and more specifically, data mining, approaches
knowledge creation from a primarily technology driven
perspective. In particular, the KDD process focuses on
how data are transformed into knowledge by identifying
valid, novel, potentially useful, and ultimately under-

Figure 1. Integrated view of the knowledge discovery process (Adapted from Wickramasinghe et al., 2003)

Knowledge
evolution

Data Information Knowledge

Steps in knowledge
discovery

Selection Preprocessing Transformation Data Mining Interpretation / Evaluation

Data Target Preprocessed Transformed Patterns Knowledge

Data Data Data

Types of
data mining
Exploratory Predictive
Data Mining Data Mining

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG

standable patterns in data (Spiegler, 2003; Fayyad,
Piatetsky-Shapiro, & Smyth, 1996). KDD is primarily
used on data sets for creating knowledge through model
building or by finding patterns and relationships in data.
From an application perspective, data mining and
KDD are often used interchangeably. Figure 1 presents a
generic representation of a typical knowledge discovery
process. This figure not only depicts each stage within
the KDD process but also highlights the evolution of
knowledge from data through information in this process
as well as the two major types of data mining, namely,
exploratory and predictive, whereas the last two steps
(i.e., data mining and interpretation/evaluation) in the
KDD process are considered predictive data mining. It is
important to note in Figure 1 that typically in the KDD
process, the knowledge component itself is treated as a
homogeneous block. Given the well-established multifac-
eted nature of the knowledge construct (Boland & Tenkasi,
1995; Malhotra, 2000; Alavi & Leidner, 2001; Schultze &
Leidner, 2002; Wickramasinghe et al., 2003), this would
appear to be a significant limitation or oversimplification
of knowledge creation through data mining as a technique
and of the KDD process in general.
The Psychosocial-Driven Perspective
to Knowledge Creation
Knowledge can exist in essentially two forms: explicit,
or factual, knowledge and tacit, or experiential (i.e.,
know how) (Polyani, 1958, 1966). Of equal signifi-
cance is the fact that organizational knowledge is not
static; rather, it changes and evolves during the lifetime
of an organization (Becerra-Fernandez & Sabherwal,
2001; Bendoly, 2003; Choi & Lee, 2003). Furthermore,
it is possible to change the form of knowledge, that is,
transform existing tacit knowledge into new explicit
knowledge, and existing explicit knowledge into new
tacit knowledge, or to transform the subjective form of
knowledge into the objective form of knowledge (Nonaka
& Nishiguchi, 2001; Nonaka, 1994). This process of
transforming the form of knowledge and thus increasing
the extant knowledge base as well as the amount and
utilization of the knowledge within the organization is
known as the knowledge spiral (Nonaka & Nishiguchi,
2001). In each of these instances, the overall extant
knowledge base of the organization grows to a new
superior knowledge base.
According to Nonaka and Nishiguchi (2001), four
things are true: a) Tacit-to-tacit knowledge transforma-
tion usually occurs through apprenticeship-type rela-
tions, where the teacher or master passes on the skill to the
apprentice, b) Explicit-to-explicit knowledge transforma-
tion usually occurs via formal learning of facts, c) Tacit-
to-explicit knowledge transformation usually occurs when
600
Incorporating the People Perspective into Data Mining
there is an articulation of nuances; for example, as in
health care, if a renowned surgeon is questioned as to why
he does a particular procedure in a certain manner, by his
articulation of the steps, the tacit knowledge becomes
explicit, and d) Explicit-to-tacit knowledge transformation
usually occurs as new explicit knowledge is internalized;
it can then be used to broaden, reframe, and extend ones
tacit knowledge. These transformations are often referred
to as the modes of socialization, combination,
externalization, and internalization, respectively (Nonaka,
1994). Integral to this changing of knowledge through the
knowledge spiral is that new knowledge is created (Nonaka
& Nishiguchi, 2001), which can bring many benefits to
organizations. Specifically, in todays knowledge-centric
economy, processes that effect a positive change to the
existing knowledge base of the organization and facilitate
better use of the organizations intellectual capital, as the
knowledge spiral does, are of paramount importance.
Two other primarily people-driven frameworks that
focus on knowledge creation as a central theme are
Spenders and Blacklers respective frameworks
(Newell, Robertson, Scarbrough, & Swan, 2002; Swan,
Scarbrough, & Preston, 1999). Spender draws a distinc-
tion between individual knowledge and social knowl-
edge, each of which he claims can be implicit or explicit
(Newell et al.). From this framework, you can see that
Spenders definition of implicit knowledge corresponds
to Nonakas tacit knowledge. However, unlike Spender,
Nonaka doesnt differentiate between individual and
social dimensions of knowledge; rather, he focuses on
the nature and types of the knowledge itself. In contrast,
Blackler (Newell et al.) views knowledge creation from
an organizational perspective, noting that knowledge
can exist as encoded, embedded, embodied, encultured,
and/or embrained. In addition, Blackler emphasized that
for different organizational types, different types of
knowledge predominate, and he highlighted the connec-
tion between knowledge and organizational processes
(Newell et al.).
Blacklers types of knowledge can be thought of in
terms of spanning a continuum of tacit (implicit) through
to explicit with embrained being predominantly tacit
(implicit) and encoded being predominantly explicit
while embedded, embodied, and encultured types of
knowledge exhibit varying degrees of a tacit (implicit)/
explicit combination. An integrated view of all three
frameworks is presented in Figure 2. Specifically, from
Figure 2, Spenders and Blacklers perspectives comple-
ment Nonakas conceptualization of knowledge cre-
ation and, more importantly, do not contradict his thesis
of the knowledge spiral, wherein the extant knowledge
base is continually being expanded to a new knowledge
base, be it tacit/explicit (in Nonakas terminology),
implicit/explicit (in Spenders terminology), or
TEAM LinG
Incorporating the People Perspective into Data Mining
Figure 2. People driven knowledge creation map
The Knowledge Continuum
Explicit Other K Types
(embodied, encultured,embedded)
Tacit/Implicit/Embrained
Social
The Knowledge Spiral
(Spenders
Spenders
Actors)
Actors
Individual
embrained/encultured/embodied/embedded/encoded (in
Blacklers terminology).
ENRICHING DATA MINING WITH THE
KNOWLEDGE SPIRAL
To conceive of knowledge as a collection of information
seems to rob the concept of all of its life. . . . Knowledge
resides in the user and not in the collection. It is how the
user reacts to a collection of information that matters
(Churchman, 1971, p. 10).
Churchman is clearly underscoring the importance of
people in the process of knowledge creation. However,
most formulations of information technology (IT) en-
abled knowledge management, and data mining in par-
ticular seems to have not only ignored the ignored the
human element but also taken a very myopic and homog-
enous perspective on the knowledge construct itself.
Recent research that has surveyed the literature on KM
indicates the need for more frameworks for knowledge
management, particularly a metaframework to facilitate
more successful realization of the KM steps
(Wickramasinghe & Mills, 2001; Holsapple & Joshi,
2002; Alavi & Leidner, 2001; Schultze & Leidner, 2002).
From a macro knowledge management perspective, the
knowledge spiral is the cornerstone of knowledge cre-
ation. From a micro data-mining perspective, one of the
key strengths of data mining as a technique is that it
facilitates knowledge creation from data. Therefore, by
integrating the algorithmic approach of knowledge cre-
ation (in particular data mining) with the psychosocial
approach of knowledge creation (i.e., the people-driven
frameworks of knowledge creation, in particular the
knowledge spiral), it is indeed possible to develop a
metaframework for knowledge creation. By so doing, a
richer and more complete approach to knowledge cre- I
ation is realized. Such an approach not only leads to a
deeper understanding of the knowledge creation pro-
cess but also offers a knowledge creation methodology
that is more customizable to specific organizational
contexts, structures, and cultures. Furthermore, it brings
the human factor back into the knowledge creation
process and doesnt oversimplify the complex knowl-
edge construct as a homogenous product.
Specifically, in Figure 3, the knowledge product of
data mining is broken into its constituent components
based on the people-driven perspectives (i.e., Blackler,
Spender, and Nonaka, respectively) of knowledge cre-
ation. On the other hand, the specific modes of trans-
formation of the knowledge spiral discussed by Nonaka
in his Knowledge Spiral should benefit from the algo-
rithmic structured nature of both exploratory and pre-
dictive data-mining techniques. For example, if you
consider socialization, which is described in Nonaka
and Nishiguchi (2001) and Nonaka (1994) as the pro-
cess of creating new tacit knowledge through discus-
sion within groups, more specifically groups of ex-
perts, and you then incorporate the results of data-
mining techniques into this context, this provides a
structured forum and hence a jump start for guiding the
dialogue and, consequently, knowledge creation. Note,
however, that this only enriches the socialization pro-
cess without restricting the actual brainstorming activi-
ties and thus not necessarily leading to the side effect of
truncating divergent thoughts. This also holds for
Nonakas other modes of knowledge transformation.
An Example within a Health Care
Context
An illustration from health care will serve to illustrate
the potential of combining the people perspective with
data mining. Health care is a very information-rich
industry. The collecting of data and information per-
meates most if not all areas of this industry. By incor-
porating a people perspective with data mining, it will
be possible to realize the full potential of these data
assets.
Table 1 details some specific instances of each of
the transformations identified in Figure 3, and Table 2
provides an example of explicit knowledge stored in a
medication repository 2.
Using the association rules data-mining algorithm,
the following patterns can be discovered:
D1 is administered to 60% of the patients (i.e., 3/5).
D1 and D2 are administered together to 40% of
the patients (i.e., 2/5).
601
TEAM LinG
 Incorporating the People Perspective into Data Mining

Figure 3. Integrating data mining with the knowledge spiral

Nonakas/Spenders/Blacklers K-Types

Tacit/Implicit/Embrained Other K Types (embodied, encultured,embedded) Explicit

KNOWLEDGE

The Knowledge Spiral

Social
(Tacit)
1. SOCIALIZATION 2. EXTERNALIZATION

Spenders Exploratory
Actors
DM

Predictive

Individual
(Explicit) 3. INTERNALIZATION 4. COMBINATION

Specific Modes of Transformations and the Role of Data Mining in Realizing Them:
1. Socialization experiential to experiential via practice group interactions on data-mining results
2. Externalization written extensions to data-mining results
3. Internalization enhancing experiential skills by learning from data-mining results
4. Combination gaining insights through applying various data mining visualization techniques

Table 1. Data mining as an enabler of the knowledge spiral

EXPLICIT1 TACIT
TO

FROM

EXPLICIT The performance of exploratory data mining, such as By assimilating and internalizing
summarization and visualization, makes it possible to knowledge discovered through data
upgrade, expand, and/or revise current facts and mining, physicians can turn this
protocols. explicit knowledge into tacit
knowledge, which they can apply to
treating new patients.
TACIT Interpretation of findings from data mining helps reveal Interpreting treatment patterns (for
tacit knowledge, which can then be articulated and stored example, for hip disease) discovered
as explicit cases in the case repository. through data mining enables
interaction among physicians, hence
making it possible to stimulate and
grow their own tacit knowledge.

602

TEAM LinG
Incorporating the People Perspective into Data Mining
Table 2. Drugs administered to patients
Patient ID Drug
1 D1, D2
2 S3, D4, D5
3 D3, D1, D2
4 D3, D5, D1
5 D5, D2
D2 is administered to 67% of the patients who are
given drug D1 (i.e., 2/3).
As the physicians try to understand these findings,
one physician could explain that D2 has to be given with
D1 for patients who had a heart attack at age 40 or less.
Thus, from this observation, the following rule can be
added to the rule repository: If a patients age is <= 40
years, the patient has a heart attack, and D1 is adminis-
tered to the patient, then D2 should also be administered
to that patient. This is an example of existing tacit
knowledge, because it originated from the physicians
head and was transformed into new explicit knowledge
that is now recorded for everyone to use (as Figure 3
demonstrates).
Table 3. Data mining as an enabler of the knowledge spiral from the example data set
FROM TO
EXPLICIT
Patient ID Drug
1 D1, D2
2 S3, D4, D5
3 D3, D1, D2
4 D3, D5, D1
5 D5, D2
TACIT
1
Each entry gives an example of how data mining enables the knowledge transfer from the knowledge type in the cell row to the
knowledge type in the cell column.
As physicians discuss the implications of these find-
ings, tacit knowledge from some of them is transformed I
into tacit knowledge for other physicians. Thus, during
the interaction of physicians from different specialties,
an environment of existing tacit to new tacit knowledge
transformations occur. These knowledge transforma-
tions are summarized in Table 3. Figure 3 and Table 3
illustrate how data mining helps to realize the four modes
or transformations of the knowledge spiral (socializa-
tion, externalization, internalization, and combination).
FUTURE TRENDS
The two significant ways to create knowledge are through
the a) synthesis of new knowledge through socialization
with experts (a primarily people-dominated perspec-
tive) and b) discovery by finding interesting patterns
through observation and combination of explicit data (a
primarily technology-driven perspective) (Becerra-
Fernandez, et al., 2004). In todays knowledge economy,
knowledge creation and the maximization of an
organizations knowledge and data assets are key strate-
EXPLICIT1 TACIT
D1 is administered to 60% of
the patients.
D1 and D2 are administered
together to 40% of the patients.
D2 is administered to 67% of
the patients who are given drug
D1.
If a patients age is <= 40 years,
the patient has a heart attack,
and D1 is
administered to the patient, then
D2 should also be administered
to that patient.
INTERACTION
603
TEAM LinG

gic necessities. Furthermore, more techniques, such as
business intelligence and business analytics, which have
their foundations in traditional data mining, are being
embraced by organizations in order to try to facilitate
the discovery of novel and unique patterns in data that
will lead to new knowledge and maximization of an
organizations data assets. Full maximization of an
organizations data assets, however, will not be realized
until the people perspective is incorporated into these
data-mining techniques to enable the full potential of
knowledge creation to occur. Thus, as organizations
strive to survive and thrive in todays competitive busi-
ness environment, incorporating a people perspective
into their data-mining initiatives will increasingly be-
come a competitive necessity.
CONCLUSION
Sustainable competitive advantage is dependent on build-
ing and exploiting core competencies (Newell et al.,
2002). In order to sustain competitive advantage, re-
sources that are idiosyncratic (and thus scarce) and
difficult to transfer or replicate are required (Grant,
1991). A knowledge-based view of the firm identifies
knowledge as the organizational asset that enables sus-
tainable competitive advantage, especially in
hypercompetitive environments (Wickramasinghe,
2003; Davenport & Prusak, 1998; Zack, 1999). This is
attributed to the fact that barriers exist regarding the
transfer and replication of knowledge (Wickramasinghe,
2003), thus making knowledge and knowledge manage-
ment of strategic significance (Kanter, 1999). The key
to maximizing the knowledge asset is in finding novel
and actionable patterns and in continuously creating new
knowledge, thereby increasing the extant knowledge
base of the organization. By incorporating a people
perspective into data-mining, it becomes truly possible
to support both major types of knowledge creation
scenarios and thereby realize the synergistic effect of
the respective strengths of these approaches in enabling
superior knowledge creation to ensue.
REFERENCES
Alavi, M., & Leidner, D. (2001). Review: Knowledge
management and knowledge management systems: Con-
ceptual foundations and research issues. MIS Quar-
terly, 25(1), 107-136.
Becerra-Fernandez, I., Gonzalez, A., & Sabherwal, R. (2004).
Knowledge management. Upper Saddle River, NJ: Prentice
Hall.
604
Incorporating the People Perspective into Data Mining
Becerra-Fernandez, I., & Sabherwal, R. (2001). Organiza-
tional knowledge management: A contingency perspec-
tive. Journal of Management Information Systems, 18(1),
23-55.
Bendoly, E. (2003). Theory and support for process
frameworks of knowledge discovery and data mining
from ERP systems. Information & Management, 40, 639-
647.
Boland, R., & Tenkasi, R. (1995). Perspective making
perspective taking. Organizational Science, 6, 350-372.
Choi, B., & Lee, H. (2003). An empirical investigation
of KM styles and their effect on corporate perfor-
mance. Information & Management, 40, 403-417.
Churchman, C. (1971). The design of inquiring sys-
tems: Basic concepts of systems and organizations.
New York: Basic Books.
Davenport, T., & Grover, V. (2001). Knowledge man-
agement. Journal of Management Information Sys-
tems, 18(1), 3-4.
Davenport, T., & Prusak, L. (1998). Working knowl-
edge. Boston: Harvard Business School Press.
Fayyad, Piatetsky-Shapiro, & Smyth, (1996). From data
mining to knowledge discovery: An overview. In Fayyad,
Piatetsky-Shapiro, Smyth, & Uthurusamy (Eds.), Ad-
vances in knowledge discovery and data mining. Menlo
Park, CA: AAAI Press/MIT Press.
Grant, R. (1991). The resource-based theory of competi-
tive advantage: Implications for strategy formulation.
California Management Review, 33(3), 114-135.
Holsapple, C., & Joshi, K. (2002). Knowledge manipu-
lation activities: Results of a delphi study. Information
& Management, 39, 477-419.
Kanter, J. (1999). Knowledge management practically
speaking. Information Systems Management.
Malhotra, Y. (2000). Knowledge management and new
organizational form. In Malhotra (Ed.), Knowledge
management and virtual organizations. Hershey, PA:
Idea Group Publishing.
Newell, S., Robertson, M., Scarbrough, H., & Swan, J.
(2002). Managing knowledge work. New York: Palgrave
Macmillan.
Nonaka, I. (1994). A dynamic theory of organizational
knowledge creation. Organizational Science, 5, 14-37.
Nonaka, I., & Nishiguchi, T. (2001). Knowledge emer-
gence. Oxford, UK: Oxford University Press.
TEAM LinG
Incorporating the People Perspective into Data Mining
Polyani, M. (1958). Personal knowledge: Towards a
postcritical philosophy. Chicago: University of Chi-
cago Press.
Polyani, M. (1966). The tacit dimension. London:
Routledge & Kegan Paul.
Schultze, U., & Leidner, D. (2002). Studying knowledge
management in information systems research: Dis-
courses and theoretical assumptions. MIS Quarterly,
26(3), 212-242.
Spiegler, I. (2003). Technology and knowledge: Bridg-
ing a generating gap. Information & Management, 40,
533-539.
Swan, J., Scarbrough, H., & Preston, J. (1999). Knowl-
edge management: The next fad to forget people? Pro-
ceedings of the Seventh European Conference in In-
formation Systems.
Wickramasinghe, N. (2003). Do we practise what we
preach: Are knowledge management systems in practice
truly reflective of knowledge management systems in
theory? Business Process Management Journal, 9(3),
295-316.
Wickramasinghe, N., Fadlalla, A., Geisler, E., & Schaffer,
J. (2003). Knowledge management and data mining: Stra-
tegic imperatives for healthcare. Proceedings of the Third
Hospital of the Future Conference, Warwick, UK.
Wickramasinghe, N., & Mills, G. (2001). MARS: The
electronic medical record system the core of the kaiser
galaxy. International Journal of Healthcare Technol-
ogy Management, 3(5/6), 406-423.
Zack, M. (1999). Knowledge and strategy. Boston:
Butterworth Heinemann.
KEY TERMS
Combination: A knowledge transfer mode that in-
volves new explicit knowledge being derived from ex-
isting explicit knowledge.
Explicit Knowledge: Also known as factual knowl-
edge (i.e., know what) (Cabena et al., 1998), repre-
sents knowledge that is well established and documented.
Externalization: A knowledge transfer mode that in-
volves new explicit knowledge being derived from exist- I
ing tacit knowledge.
Hegelian/Kantian Perspective of Knowledge
Management: Refers to the subjective component of
knowledge management and can be viewed as an ongoing
phenomenon being shaped by social practices of commu-
nities and encouraging discourse and divergence of mean-
ing.
Internalization: A knowledge transfer mode that
involves new tacit knowledge being derived from exist-
ing explicit knowledge.
Knowledge Spiral: The process of transforming
the form of knowledge and thus increasing the extant
knowledge base as well as the amount and utilization of
the knowledge within the organization.
Lockean/Leibnitzian Perspective of Knowledge
Management: Refers to the objective aspects of knowl-
edge management, where the need for knowledge is to
improve effectiveness and efficiency.
Socialization: A knowledge transfer mode that in-
volves new tacit knowledge being derived from existing
tacit knowledge.
Tacit Knowledge: Also known as experiential knowl-
edge (i.e., know how) (Cabena et al., 1998) represents
knowledge that is gained through experience.
ENDNOTES
1
Each entry explains how data mining enables the
knowledge transfer from the type of knowledge in
the cell row to the type of knowledge in the cell
column.
2
The example is kept small and simple for illustrative
purposes; naturally, in large medical databases the
data would be much larger.
3
Each entry gives an example of how data mining
enables the knowledge transfer from the knowl-
edge type in the cell row to the knowledge type in
the cell column.
605
TEAM LinG
606
Incremental Mining from News Streams
Seokkyung Chung
University of Southern California, USA
Jongeun Jun
University of Southern California, USA
Dennis McLeod
University of Southern California, USA
INTRODUCTION
With the rapid growth of the World Wide Web, Internet
users are now experiencing overwhelming quantities of
online information. Since manually analyzing the data
becomes nearly impossible, the analysis would be per-
formed by automatic data mining techniques to fulfill
users information needs quickly.
On most Web pages, vast amounts of useful knowl-
edge are embedded into text. Given such large sizes of text
collection, mining tools, which organize the text datasets
into structured knowledge, would enhance efficient docu-
ment access. This facilitates information search and, at
the same time, provides an efficient framework for docu-
ment repository management as the number of documents
becomes extremely huge.
Given that the Web has become a vehicle for the
distribution of information, many news organizations are
providing newswire services through the Internet. Given
this popularity of the Web news services, text mining on
news datasets has received significant attentions during
the past few years. In particular, as several hundred news
stories are published everyday at a single Web news site,
triggering the whole mining process whenever a docu-
ment is added to the database is computationally imprac-
tical. Therefore, efficient incremental text mining tools
need to be developed.
BACKGROUND
The simplest document access method within Web news
services is keyword-based retrieval. Although this method
seems effective, there exist at least three serious draw-
backs. First, if a user chooses irrelevant keywords, then
retrieval accuracy will be degraded. Second, since key-
word-based retrieval relies on the syntactic properties of
information (e.g., keyword counting), semantic gap can-
not be overcome (Grosky, Sreenath, & Fotouhi, 2002).
Third, only expected information can be retrieved since
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
the specified keywords are generated from users knowl-
edge space. Thus, if users are unaware of the airplane
crash that occurred yesterday, then they cannot issue a
query about that accident even though they might be
interested.
The first two drawbacks stated above have been
addressed by query expansion based on domain-inde-
pendent ontologies. However, it is well known that this
approach leads to a degradation of precision. That is,
given that the words introduced by term expansion may
have more than one meaning, using additional terms can
improve recall, but decrease precision. Exploiting a manu-
ally developed ontology with a controlled vocabulary
would be helpful in this situation (Khan, McLeod, &
Hovy, 2004). However, although ontology-authoring tools
have been developed in the past decades, manually con-
structing ontologies whenever new domains are encoun-
tered is an error-prone and time-consuming process. There-
fore, integration of knowledge acquisition with data min-
ing, which is referred to as ontology learning, becomes
a must (Maedche & Staab, 2001).
To facilitate information navigation and search on a
news database, clustering can be utilized. Since a collec-
tion of documents is easy to skim if similar articles are
grouped together, if the news articles are hierarchically
classified according to their topics, then a query can be
formulated while a user navigates a cluster hierarchy.
Moreover, clustering can be used to identify and deal with
near-duplicate articles. That is, when news feeds repeat
stories with minor changes from hour to hour, presenting
only the most recent articles is probably sufficient. In
particular, a sophisticated incremental hierarchical docu-
ment clustering algorithm can be effectively used to
address high rate of document update. Moreover, in order
to achieve rich semantic information retrieval, an ontol-
ogy-based approach would be provided. However, one of
the main problems with concept-based ontologies is that
topically related concepts and terms are not explicitly
linked. That is, there is no relation between court-attor-
ney, kidnap-police, and etcetera. Thus, concept-based
TEAM LinG
Incremental Mining from News Streams
ontologies have a limitation in supporting a topical search.
In sum, it is essential to develop incremental text mining
methods for intelligent news information presentation.
MAIN THRUST
In the following, we will explore text mining approaches
that are relevant for news streams data.
Requirements of Document Clustering
in News Streams
Data we are considering are high dimensional, large in
size, noisy, and a continuous stream of documents. Many
previously proposed document clustering algorithms did
not perform well on this dataset due to a variety of
reasons. In the following, we define application-depen-
dent (in terms of news streams) constraints that the
clustering algorithm must satisfy.
1. Ability to determine input parameters: Many clus-
tering algorithms require a user to provide input
parameters (e.g., the number of clusters), which is
difficult to be determined in advance, in particular
when we are dealing with incremental datasets.
Thus, we expect the clustering algorithm not to need
such kind of knowledge.
2. Scalability with large number of documents: The
number of documents to be processed is extremely
large. In general, the problem of clustering n objects
into k clusters is NP-hard. Successful clustering
algorithms should be scalable with the number of
documents.
3. Ability to discover clusters with different shapes
and sizes: The shape of document cluster can be of
arbitrary shapes; hence we cannot assume the shape
of document cluster (e.g., hyper-sphere in k-means).
In addition, the sizes of clusters can be of arbitrary
numbers, thus clustering algorithms should iden-
tify the clusters with wide variance in size.
4. Outliers Identification: In news streams, outliers
have a significant importance. For instance, a unique
document in a news stream may imply a new technol-
ogy or event that has not been mentioned in previ-
ous articles. Thus, forming a singleton cluster for
the outlier is important.
5. Efficient incremental clustering: Given different
ordering of a same dataset, many incremental clus-
tering algorithms produce different clusters, which
is an unreliable phenomenon. Thus, the incremental
clustering should be robust to the input sequence.
Moreover, due to the frequent document insertion
into the database, whenever a new document is
inserted it should perform a fast update of the I
existing cluster structure.
6. Meaningful theme of clusters: We expect each
cluster to reflect a meaningful theme. We define
meaningful theme in terms of precision and recall.
That is, if a cluster (C) is about Turkey earth-
quake, then all documents about Turkey earth-
quake should belong to C, and documents that do
not talk about Turkey earthquake should not
belong to C.
7. Interpretability of resulting clusters: A clustering
structure needs to be tied up with a succinct sum-
mary of each cluster. Consequently, clustering re-
sults should be easily comprehensible by users.
Previous Document Clustering
Approaches
The most widely used document clustering algorithms fall
into two categories: partition-based clustering and hier-
archical clustering. In the following, we provide a concise
overview for each of them, and discuss why these ap-
proaches fail to address the requirements discussed above.
Partition-based clustering decomposes a collection of
documents, which is optimal with respect to some pre-
defined function (Duda, Hart, & Stork, 2001; Liu, Gong,
Xu, & Zhu, 2002). Typical methods in this category
include center-based clustering, Gaussian Mixture Model,
and etcetera. Center-based algorithms identify the clus-
ters by partitioning the entire dataset into a pre-deter-
mined number of clusters (e.g., k-means clustering). Al-
though the center-based clustering algorithms have been
widely used in document clustering, there exist at least
five serious drawbacks. First, in many center-based clus-
tering algorithms, the number of clusters needs to be
determined beforehand. Second, the algorithm is sensi-
tive to an initial seed selection. Third, it can model only a
spherical (k-means) or ellipsoidal (k-medoid) shape of
clusters. Furthermore, it is sensitive to outliers since a
small amount of outliers can substantially influence the
mean value. Note that capturing an outlier document and
forming a singleton cluster is important. Finally, due to the
nature of an iterative scheme in producing clustering
results, it is not relevant for incremental datasets.
Hierarchical (agglomerative) clustering (HAC) identi-
fies the clusters by initially assigning each document to
its own cluster and then repeatedly merging pairs of
clusters until a certain stopping condition is met (Zhao &
Karypis, 2002). Consequently, its result is in the form of
a tree, which is referred to as a dendrogram. A dendrogram
is represented as a tree with numeric levels associated to
its branches. The main advantage of HAC lies in its ability
607
TEAM LinG

to provide a view of data at multiple levels of abstraction.
Although HAC can model arbitrary shapes and different
sizes of clusters, and can be extended to the robust version
(in outlier handling sense), it is not relevant for news
streams application due to the following two reasons.
First, since HAC builds a dendrogram, a user should
determine where to cut the dendrogram to produce actual
clusters. This step is usually done by human visual inspec-
tion, which is a time-consuming and subjective process.
Second, the computational complexity of HAC is expen-
sive since pairwise similarities between clusters need to be
computed.
Topic Detection and Tracking
Over the past six years, the information retrieval commu-
nity has developed a new research area, called TDT (Topic
Detection and Tracking) (Makkonen, Ahonen-Myka, &
Salmenkivi, 2004; Allan, 2002). The main goal of TDT is to
detect the occurrence of a novel event in a stream of news
stories, and to track the known event. In particular, there
are three major components in TDT.
1. Story segmentation: It segments a news stream (e.g.,
including transcribed speech) into topically cohe-
sive stories. Since online Web news (in HTML for-
mat) is supplied in segmented form, this task only
applies to audio or TV news.
2. First Story Detection (FSD): It identifies whether a
new document belongs to an existing topic or a new
topic.
3. Topic tracking: It tracks events of interest based on
sample news stories. It associates incoming news
stories with the related stories, which were already
discussed before. It can be also asked to monitor the
news stream for further stories on the same topic.
Event is defined as some unique thing that happens
at some point in time. Hence, an event is different from a
topic. For example, airplane crash is a topic while Chi-
nese airplane crash in Korea in April 2002 is an event.
Note that it is important to identify events as well as topics.
Although a user is not interested in a flood topic, the user
may be interested in the news story on the Texas flood if
the users hometown is from Texas. Thus, a news recom-
mendation system must be able to distinguish different
events within a same topic.
Single-pass document clustering (Chung & McLeod,
2003) has been extensively used in TDT research. How-
ever, the major drawback of this approach lies in order-
sensitive property. Although the order of documents is
already fixed since documents are inserted into the data-
base in chronological order, order-sensitive property im-
plies that the resulting cluster is unreliable. Thus, new
608
Incremental Mining from News Streams
methodology is required in terms of incremental news
article clustering.
Dynamic Topic Mining
Dynamic topic mining is a framework that supports the
identification of meaningful patterns (e.g., events, top-
ics, and topical relations) from news stream data (Chung
& McLeod, 2003). To build a novel paradigm for an
intelligent news database management and navigation
scheme, it utilizes techniques in information retrieval,
data mining, machine learning, and natural language
processing.
In dynamic topic mining, a Web crawler downloads
news articles from a news Web site on a daily basis.
Retrieved news articles are processed by diverse infor-
mation retrieval and data mining tools to produce useful
higher-level knowledge, which is stored in a content
description database. Instead of interacting with a Web
news service directly, by exploiting the knowledge in the
database, an information delivery agent can present an
answer in response to a user request (in terms of topic
detection and tracking, keyword-based retrieval, docu-
ment cluster visualization, etc). Key contributions of the
dynamic topic mining framework are development of a
novel hierarchical incremental document clustering al-
gorithm, and a topic ontology learning framework.
Despite the huge body of research efforts on docu-
ment clustering, previously proposed document cluster-
ing algorithms are limited in that it cannot address special
requirements in a news environment. That is, an algo-
rithm must address the seven application-dependent
constraints discussed before. Toward this end, the dy-
namic topic mining framework presents a sophisticated
incremental hierarchical document clustering algorithm
that utilizes a neighborhood search. The algorithm was
tested to demonstrate the effectiveness in terms of the
seven constraints. The novelty of the algorithm is the
ability to identify meaningful patterns (e.g., news events,
and news topics) while reducing the amount of compu-
tations by maintaining cluster structure incrementally.
In addition, to overcome the lack of topical relations
in conceptual ontologies, a topic ontology learning frame-
work is presented. The proposed topic ontologies pro-
vide interpretations of news topics at different levels of
abstraction. For example, regarding to a Winona Ryder
court trial news topic (T), the dynamic topic mining could
capture winona, ryder, actress, shoplift, beverly as
specific terms describing T (i.e., the specific concept for
T) while attorney, court, defense, evidence, jury, kill,
law, legal, murder, prosecutor, testify, trial as general
terms representing T (i.e., the general concept for T).
There exists research work on extracting hierarchical
relations between terms from a set of documents (Tseng,
TEAM LinG
Incremental Mining from News Streams
2002). However, the dynamic topic mining framework is
unique in that the topical relations are dynamically gen-
erated based on incremental hierarchical clustering rather
than based on human defined topics, such as Yahoo
directory.
FUTURE TRENDS
There are many future research opportunities in news
streams mining. First, although a document hierarchy can
be obtained using unsupervised clustering, as shown in
Aggarwal, Gates, & Yu (2004), the cluster quality can be
enhanced if a pre-existing knowledge base is exploited.
That is, based on this priori knowledge, we can have some
control while building a document hierarchy.
Second, document representation for clustering can
be augmented with phrases by employing different levels
of linguistic analysis (Hatzivassiloglou, Gravano, &
Maganti, 2000). That is, representation model can be
augmented by adding n-gram (Peng & Schuurmans, 2003),
or frequent itemsets using association rule mining (Hand,
Mannila, & Smyth, 2001). Investigating how different
feature selection algorithms affect on the accuracy of
clustering results is an interesting research work.
Third, besides exploiting text data, other information
can be utilized since Web news articles are composed of
text, hyperlinks, and multimedia data. For example, both
terms and hyperlinks (which point to related news articles
or Web pages) can be used for feature selection.
Finally, a topic ontology learning framework can be
extended to accommodating rich semantic information
extraction. For example, topic ontologies can be anno-
tated within Protg (Noy, Sintek, Decker, Crubezy,
Fergerson, & Musen, 2001) WordNet tab. In addition, a
query expansion algorithm based on ontologies needs to
be developed for intelligent news information presenta-
tion.
CONCLUSION
Incremental text mining from news streams is an emerging
technology as many news organizations are providing
newswire services through the Internet. In order to ac-
commodate dynamically changing topics, efficient incre-
mental document clustering algorithms need to be devel-
oped. The algorithms must address the special require-
ments in news clustering, such as high rate of document
update or ability to identify event level clusters, as well as
topic level clusters.
In order to achieve rich semantic information retrieval
within Web news services, an ontology-based approach
would be provided. To overcome the problem of concept-
based ontologies (i.e., topically related concepts and I
terms are not explicitly linked), topic ontologies are pre-
sented to characterize news topics at multiple levels of
abstraction. In sum, coupling with topic ontologies and
concept-based ontologies, supporting a topical search as
well as semantic information retrieval can be achieved.
REFERENCES
Aggarwal, C.C., Gates, S.C., & Yu, P.S. (2004). On using partial
supervision for text categorization. IEEE Transactions on
Knowledge and Data Engineering, 16(2), 245-255.
Allan, J. (2002). Detection as multi-topic tracking. Infor-
mation Retrieval, 5(2-3), 139-157.
Chung, S., & McLeod, D. (2003, November). Dynamic
topic mining from news stream data. In ODBASE03 (pp.
653-670). Catania, Sicily, Italy.
Duda, R.O., Hart, P.E., & Stork D.G. (2001). Pattern clas-
sification. New York: Wiley Interscience.
Grosky, W.I., Sreenath, D.V., & Fotouhi, F. (2002). Emer-
gent semantics and the multimedia semantic web. SIGMOD
Record, 31(4), 54-58.
Hand, D.J., Mannila, H., & Smyth, P. (2001). Principles
of data mining (adaptive computation and machine
learning). Cambridge, MA: The MIT Press.
Hatzivassiloglou, V., Gravano, L., & Maganti, A. (2000,
July). An investigation of linguistic features and cluster-
ing algorithms for topical document clustering. In ACM
SIGIR Conference on Research and Development in Infor-
mation Retrieval (SIGIR00) (pp. 224-231). Athens, Greece.
Khan, L., McLeod, D., & Hovy, E. (2004). Retrieval effec-
tiveness of an ontology-based model for information
selection. The VLDB Journal, 13(1), 71-85.
Liu, X., Gong, Y., Xu, W., & Zhu, S. (2002, August).
Document clustering with cluster refinement and model
selection capabilities. In ACM SIGIR International Con-
ference on Research and Development in Information
Retrieval (SIGIR02) (pp. 91-198). Tampere, Finland.
Maedche, A., & Staab, S. (2001). Ontology learning for the
semantic Web. IEEE Intelligent Systems, 16(2), 72-79.
Makkonen, J., Ahonen-Myka, H., & Salmenkivi, M.
(2004). Simple semantics in topic detection and track-
ing. Information Retrieval, 7(3-4), 347-368.
Noy, N.F., Sintek, M., Decker, S., Crubezy, M.,
Fergerson, R.W., & Musen M.A. (2001). Creating seman-
609
TEAM LinG

tic Web contents with Protg 2000. IEEE Intelligent
Systems, 6(12), 60-71.
Peng, F., & Schuurmans, D. (2003, April). Combining naive
Bayes and n-gram language models for text classification.
European Conference on IR Research (ECIR03) (pp.
335-350). Pisa, Italy.
Tseng, Y. (2002). Automatic thesaurus generation for Chi-
nese documents. Journal of the American Society for Infor-
mation Science and Technology, 53(13), 1130-1138.
Zhao, Y., & Karypis, G. (2002, November). Evaluations of
hierarchical clustering algorithms for document datasets.
In ACM International Conference on Information and
Knowledge Management (CIKM02) (pp. 515-524).
McLean, VA.
KEY TERMS
Clustering: An unsupervised process of dividing
data into meaningful groups such that each identified
cluster can explain the characteristics of underlying data
distribution. Examples include characterization of differ-
ent customer groups based on the customers purchasing
patterns, categorization of documents in the World Wide
Web, or grouping of spatial locations of the earth where
neighbor points in each region have similar short-term/
long-term climate patterns.
Dynamic Topic Mining: A framework that supports
the identification of meaningful patterns (e.g., events,
topics, and topical relations) from news stream data.
610
Incremental Mining from News Streams
First Story Detection: A TDT component that identi-
fies whether a new document belongs to an existing topic
or a new topic.
Ontology: A collection of concepts and inter-relation-
ships.
Text Mining: A process of identifying patterns or
trends in natural language text including document clus-
tering, document classification, ontology learning, and
etcetera.
Topic Detection And Tracking (TDT): Topic Detec-
tion and Tracking (TDT) is a DARPA-sponsored initiative
to investigate the state of the art for news understanding
systems. Specifically, TDT is composed of the following
three major components: (1) segmenting a news stream
(e.g., including transcribed speech) into topically cohe-
sive stories; (2) identifying novel stories that are the first
to discuss a new event; and (3) tracking known events
given sample stories.
Topic Ontology: A collection of terms that character-
ize a topic at multiple levels of abstraction.
Topic Tracking: A TDT component that tracks events
of interest based on sample news stories. It associates
incoming news stories with the related stories, which were
already discussed before or it monitors the news stream
for further stories on the same topic.
TEAM LinG
 611

Inexact Field Learning Approach for Data I

Mining
Honghua Dai
Deakin University, Australia

INTRODUCTION values of each attribute in the training set, rather than on

Inexact fielding learning (IFL) (Ciesieski & Dai, 1994; Dai
& Ciesieski, 1994a, 1994b, 1995, 2004; Dai & Li, 2001) is a
rough-set, theory-based (Pawlak, 1982) machine learning
approach that derives inexact rules from fields of each
attribute. In contrast to a point-learning algorithm (Quinlan,
1986, 1993), which derives rules by examining individual
values of each attribute, a field learning approach (Dai,
1996) derives rules by examining the fields of each at-
tribute. In contrast to exact rule, an inexact rule is a rule
with uncertainty. The advantage of the IFL method is the
capability to discover high-quality rules from low-quality
data, its property of low-quality data tolerant (Dai &
Ciesieski, 1994a, 2004), high efficiency in discovery, and
high accuracy of the discovered rules.
individual point values. The discovered rule is in a form
called b-rule and is somewhat analogous to a decision
tree found by an induction algorithm. The algorithm is
linear in both the number of attributes and the number of
instances (Dai & Ciesieski, 1994a, 2004).
The advantage of this inexact field-learning approach
is its capability of inducing high-quality classification
rules from low-quality data and its high efficiency that
makes it an ideal algorithm to discover reliable knowledge
from large and very large low-quality databases suitable
for data mining, which needs higher discovering capability.
INEXACT FIELD-LEARNING
ALGORITHM

Detailed description and the applications of the algorithm

BACKGROUND can be found from the listed articles (Ciesieski & Dai,
1994a; Dai & Ciesieski, 1994a, 1994b, 1995, 2004; Dai, 1996;
Achieving high prediction accuracy rates is crucial for all Dai & Li, 2001; Dai, 1996). The following is a description
learning algorithms, particularly in real applications. In of the inexact field-learning algorithm, the Fish_net algorithm:
the area of machine learning, a well-recognized problem is
that the derived rules can fit the training data very well, but Input: The input of the algorithm is a training data set with
they fail to achieve a high accuracy rate on new unseen m instances and n attributes as follows:
cases. This is particularly true when the learning is per-
formed on low-quality databases. Such a problem is re-
Instances X1 X2 ... Xn Classes
ferred as the Low Prediction Accuracy (LPA) problem (Dai
Instance1 a11 a12 ... a1n 1
& Ciesieski, 1994b, 2004; Dai & Li, 2001), which could be
caused by several factors. In particular, overfitting low- Instance 2 a21 a22 ... a2 n 2
quality data and being misled by them seem to be the ... ... ... ... ... ...
significant problems that can hamper a learning algorithm Instancem am1 am 2 ... amn m
from achieving high accuracy. Traditional learning meth- (1)
ods derive rules by examining individual values of in-
stances (Quinlan, 1986, 1993). To generate classification Learning Process:
rules, these methods always try to find cut-off points,
such as in well-known decision tree algorithms (Quinlan,
Step 1: Work Out Fields of each attribute {xi |1 i n}
1986, 1993).
with respect to each class.
What we present here is an approach to derive rough
classification rules from large low-quality numerical data-
bases that appear to be able to overcome these two h(j k ) = [h(jlk ) , h(juk ) ]
problems. The algorithm works on the fields of continu- (2)
(k = 1, 2,..., s; j = 1, 2,...n).
ous numeric variables; that is, the intervals of possible

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
 Inexact Field Learning Approach for Data Mining

h(juk ) = max{ aij | i = 1, 2,..., m} hu( + ) = max {( I i ) | i = 1, 2,..., m} (8)

(k)
aij a j ( Ii ), Ii +
(3)
(k = 1, 2,..., s; j = 1, 2,...n) hl( + ) = min {( I i ) | i = 1, 2,..., m} (9)
( Ii ), Ii +

Similarly we can find h =< hl , hu >

h(jlk ) = min( k ){aij | i = 1, 2,..., m}
aij a j
(4) Step 4: Construct Belief Function using the derived
(k = 1, 2,..., s; j = 1, 2,...n).
contribution fields.

Step 2: Construct Contribution Function based on

the fields found in Step 1. 1 Contribution NegativeRegion

Br (C ) = 1 Contribution PositiveRegion
c a (10)
Contribution RoughRegion
s b a
0 x j U i k h(j i ) h(j k )
s
ck ( x j ) = 1 x j h(j k ) U i k h(j i )

x j a x h( k ) ( s h (i ) ) (5) Step 5: Decide Threshold. It could have 6 different
j I Ui k j
b a j
cases to be considered. The simplest case is to take
(k = 1, 2,..., s) the threshold

= midpoint of h + and h (11)

The formula (5) is given on the assumption that
s Step 6: Form the Inexact Rule.
[a, b] = h (j k ) I (U i k h (j i ) ) , and for any small number
s s
> 0, b h(j k ) and a + U i k h(ji ) or a U i k h(ji ) . 1 N

Otherwise, the formula (5) becomes,

If ( I ) = ( xi ) >
N i =1 (12)
Then = 1( Br (c))

s
0 x j U i k h(j i ) h(j k )
s
This algorithm was tested on three large real observa-
ck ( x j ) = 1 x j h (j k ) U i k h (ji ) tional weather data sets containing both high-quality and

x j b x h( k ) ( s h (i ) ) (6) low-quality data. The accuracy rates of the forecasts were
j I Ui k j
a b
j
86.4%, 78%, and 76.8%. These are significantly better
(k = 1, 2,..., s) than the accuracy rates achieved by C4.5 (Quinlan, 1986,
1993), feed forward neural networks, discrimination analy-
sis, K-nearest neighbor classifiers, and human weather
forecasters. The fish-net algorithm exhibited significantly
Step 3: Work Out Contribution Fields by applying less overfitting than the other algorithms. The training
the constructed contribution functions to the train- times were shorter, in some cases by orders of magnitude
ing data set. (Dai & Ciesieski, 1994a, 2004; Dai 1996).
Calculate the contribution of each instance.
FUTURE TRENDS
n
( I i ) = ( ( xij )) / n
j =1
(7) The inexact field-learning approach has led to a success-
(i = 1, 2,..., m) ful algorithm in a domain where there is a high level of
noise. We believe that other algorithms based on fields
also can be developed. The b-rules, produced by the
Work out the contribution field for each class. current FISH-NET algorithm involve linear combinations
h+ =< hl+ , hu+ >
of attributes. Non-linear rules may be even more accurate.

612

TEAM LinG
Inexact Field Learning Approach for Data Mining
While extensive tests have been done on the fish-net
algorithm with large meteorological databases, nothing in
the algorithm is specific to meteorology. It is expected that
the algorithm will perform equally well in other domains. In
parallel to most existing exact machine-learning methods,
the inexact field-learning approaches can be used for large
or very large noisy data mining, particularly where the data
quality is a major problem that may not be dealt with by
other data-mining approaches. Various learning algorithms
can be created, based on the fields derived from a given
training data set. There are several new applications of
inexact field learning, such as Zhuang and Dai (2004) for
Web document clustering and some other inexact learning
approaches (Ishibuchi et al., 2001; Kim et al., 2003). The
major trends of this approach are in the following:
1. Heavy application for all sorts of data-mining tasks
in various domains.
2. Developing new powerful discovery algorithms in
conjunction with IFL and traditional learning ap-
proaches.
3. Extend current IFL approach to deal with high dimen-
sional, non-linear, and continuous problems.
CONCLUSION
The inexact field-learning algorithm: Fish-net is developed
for the purpose of learning rough classification/forecast-
ing rules from large, low-quality numeric databases. It runs
high efficiently and generates robust rules that do not
overfit the training data nor result in low prediction accu-
racy.
The inexact field-learning algorithm, fish-net, is based
on fields of the attributes rather than the individual point
values. The experimental results indicate that:
1. The fish-net algorithm is linear both in the number of
instances and in the number of attributes. Further,
the CPU time grows much more slowly than the other
algorithms we investigated.
2. The Fish-net algorithm achieved the best prediction
accuracy tested on new unseen cases out of all the
methods tested (i.e., C4.5, feed-forward neural net-
work algorithms, a k-nearest neighbor method, the
discrimination analysis algorithm, and human experts.
3. The fish-net algorithm successfully overcame the
LPA problem on two large low-quality data sets
examined. Both the absolute LPA error rate and the
relative LPA error rate (Dai & Ciesieski, 1994b) of the
fish-net were very low on these data sets. They were
significantly lower than that of point-learning ap-
proach, such as C4.5, on all the data sets and lower
than the feed-forward neural network. A reason- I
ably low LPA error rate was achieved by the feed-
forward neural network but with the high time cost
of error back-propagation. The LPA error rate of
the KNN method is comparable to fish-net. This
was achieved after a very high-cost genetic algo-
rithm search.
4. The FISH-NET algorithm obviously was not af-
fected by low-quality data. It performed equally
well on low-quality data and high-quality data.
REFERENCES
Ciesielski, V., & Dai, H. (1994a). FISHERMAN: A compre-
hensive discovery, learning and forecasting systems.
Proceedings of 2nd Singapore International Conference
on Intelligent System, Singapore.
Dai, H. (1994c). Learning of forecasting rules from large
noisy meteorological data [doctoral thesis]. RMIT,
Melbourne, Victoria, Australia.
Dai, H. (1996a). Field learning. Proceedings of the 19th
Australian Computer Science Conference.
Dai, H. (1996b). Machine learning of weather forecasting
rules from large meteorological data bases. Advances in
Atmospheric Science, 13(4), 471-488.
Dai, H. (1997). A survey of machine learning [technical
report]. Monash University, Melbourne, Victoria, Aus-
tralia.
Dai, H. & Ciesielski, V. (1994a). Learning of inexact rules
by the FISH-NET algorithm from low quality data. Pro-
ceedings of the 7 th Australian Joint Conference on
Artificial Intelligence, Brisbane, Australia.
Dai, H. & Ciesielski, V. (1994b). The low prediction
accuracy problem in learning. Proceedings of Second
Australian and New Zealand Conference On Intelligent
Systems, Armidale, NSW, Australia.
Dai, H., & Ciesielski, V. (1995). Inexact field learning
using the FISH-NET algorithm [technical report].
Monash University, Melbourne, Victoria, Australia.
Dai, H., & Ciesielski, V. (2004). Learning of fuzzy classi-
fication rules by inexact field learning approach [tech-
nical report]. Deakin University, Melbourne, Australia.
Dai, H. & Li, G. (2001). Inexact field learning: An approach
to induce high quality rules from low quality data. Pro-
ceedings of the 2001 IEEE International Conference on
Data Mining (ICDM-01), San Jose, California.
613
TEAM LinG

Ishibuchi, H., Yamamoto, T., & Nakashima, T. (2001).
Fuzzy data mining: Effect of fuzzy discretization. Proceed-
ings of IEEE International Conference on Data Mining,
San Jose, California.
Kim, M., Ryu, J., Kim, S., & Lee, J. (2003). Optimization of
fuzzy rules for classification using genetic algorithm.
Proceedings of the 7th Pacific-Asia Conference on Knowl-
edge Discovery and Data Mining, Seoul, Korea.
Pawlak, Z. (1982). Rough sets. International Journal of
Information and Computer Science, 11(5), 145-172.
Quinlan, R. (1986). Induction of decision trees. Machine
Learning, 1, 81-106.
Quinlan, R. (1993). C4.5: Programs for machine learning.
San Mateo, CA: Morgan Kaufmann Publishers.
Zhuang, L. & Dai, H. (2004). Maximal frequent itemset
approach for Web document clustering. Proceedings of
the 2004 International Conference on Computer and
Information Technology (CIT04), Wuhan, China.
614
Inexact Field Learning Approach for Data Mining
KEY TERMS
b-Rule: A type of inexact rule that represent the
uncertainty with contribution functions and belief func-
tions.
Exact Learning: The learning approaches that are
capable of inducing exact rules.
Exact Rules: Rules without uncertainty.
Field Learning: Derives rules by looking at the field
of the values of each attribute in all the instances of the
training data set.
Inexact Learning: The learning by which inexact rules
are induced.
Inexact Rules: Rules with uncertainty.
Low-Quality Data: Data with lots of noise, missing
values, redundant features, mistakes, and so forth.
LPA (Low Prediction Accuracy) Problem: The prob-
lem when derived rules can fit the training data very well
but fail to achieve a high accuracy rate on new unseen
cases.
Point Learning: Derives rules by looking at each
individual point value of the attributes in every instance
of the training data set.
TEAM LinG
 615

Information Extraction in Biomedical I

Literature
Min Song
Drexel University, USA

Il-Yeol Song
Drexel University, USA

Xiaohua Hu
Drexel University, USA

Hyoil Han
Drexel University, USA

INTRODUCTION we review recent advances in applying IE techniques to

Information extraction (IE) technology has been de-
fined and developed through the US DARPA Message
Understanding Conferences (MUCs). IE refers to the
identification of instances of particular events and rela-
tionships from unstructured natural language text docu-
ments into a structured representation or relational
table in databases. It has proved successful at extracting
information from various domains, such as the Latin
American terrorism, to identify patterns related to ter-
rorist activities (MUC-4). Another domain, in the light
of exploiting the wealth of natural language documents,
is to extract the knowledge or information from these
unstructured plain-text files into a structured or rela-
tional form. This form is suitable for sophisticated
query processing, for integration with relational data-
bases, and for data mining. Thus, IE is a crucial step for
fully making text files more easily accessible.
BACKGROUND
biomedical literature.
MAIN THRUST
This article attempts to synthesize the works that have
been done in the field. Taxonomy helps us understand
the accomplishments and challenges in this emerging
field. In this article, we use the following set of criteria
to classify the biomedical literature mining related
studies:
1. What are the target objects that are to be extracted?
2. What techniques are used to extract the target
objects from the biomedical literature?
3. How are the techniques or systems evaluated?
Figure 1. Shows the overview of a typical biomedical
literature mining system.
Curated DB

The advent of large volumes of text databases and search

engines have made them readily available to domain
Genbank
experts and have significantly accelerated research on SwissProt

bioinformatics. With the size of a digital library com- Evaluate the system
monly exceeding millions of documents, rapidly in- Biomedical

creasing, and covering a wide range of topics, efficient Extract entities &
relationships
Text Mining Systems
Build a KB
Knowledge base
(KB)

and automatic extraction of meaningful data and rela-

tions has become a challenging issue. To tackle this Integrated into the
Patient Records
issue, rigorous studies have been carried out recently to system MESH

apply IE to biomedical data. Such research efforts began MEDLINE UMLS

BLAST
to be called biomedical literature mining or text mining SNOMED CT

in bioinformatics (de Bruijn & Martin, 2002; Hirschman

Literature Collections Gene
Ontologies
Ontologies
et al., 2002; Shatkay & Feldman, 2003). In this article,

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
 Information Extraction in Biomedical Literature

4) From what data sources are the target objects ex-
tracted?
Target Objects
In terms of what is to be extracted by the systems, most
studies can be broken into the following two major
areas: (1) named entity extraction such as proteins or
genes; and (2) relation extraction, such as relationships
between proteins. Most of these studies adopt informa-
tion extraction techniques using curated lexicon or
natural language processing for identifying relevant
tokens such as words or phrases in text (Shatkay &
Feldman, 2003).
In the area of named entity extraction, Proux et al.
(2000) use single word names only with selected test
set from 1,200 sentences coming from Flybase. Collier,
et al. (2000) adopt Hidden Markov Models (HMMs) for
10 test classes with small training and test sets.
Krauthammer et al. (2000) use BLAST database with
letters encoded as 4-tuples of DNA. Demetriou and
Gaizuaskas (2002) pipeline the mining processes, in-
cluding hand-crafted components and machine learning
components. For the study, they use large lexicon and
morphology components. Narayanaswamy et al. (2003)
use a part of speech (POS) tagger for tagging the parsed
MEDLINE abstracts. Although Narayanaswamy and his
colleagues (2003) implement an automatic protein name
detection system, the number of words used is 302, and,
thus, it is difficult to see the quality of their system,
since the size of the test data is too small. Yamamoto, et
al. (2003) use morphological analysis techniques for
preprocessing protein name tagging and apply support
vector machine (SVM) for extracting protein names.
They found that increasing training data from 390 ab-
stracts to 1,600 abstracts improved F-value perfor-
mance from 70% to 75%. Lee et al. (2003) combined an SVM
and dictionary lookup for named entity recognition. Their
approach is based on two phases: the first phase is
identification of each entity with an SVM classifier, and
the second phase is post-processing to correct the errors
by the SVM with a simple dictionary lookup. Bunescu, et
al. (2004) studied protein name identification and protein-
protein interaction. Among several approaches used in
their study, the main two ways are one using POS tagging
and the other using the generalized dictionary-based
tagging. Their dictionary-based tagging presents higher
F-value. Table 1 summarizes the works in the areas of
named entity extraction in biomedical literature.
The second target object type of biomedical literature
extraction is relation extraction. Leek (1997) applies HMM
techniques to identify gene names and chromosomes
through heuristics. Blaschke et al. (1999) extract protein-
protein interactions based on co-occurrence of the form
p1I1 p2 within a sentence, where p1, p2 are
proteins, and I1 is an interaction term. Protein names and
interaction terms (e.g., activate, bind, inhibit) are pro-
vided as a dictionary. Proux (2000) extracts an interact
relation for the gene entity from Flybase database.
Pustejovsky (2002) extracts an inhibit relation for the gene
entity from MEDLINE. Jenssen, et al. (2001) extract a gene-
gene relations based on co-occurrence of the form
g1g2 within a MEDLINE abstracts, where g1 and g2
are gene names. Gene names are provided as a dictionary,
harvested from HUGO, LocusLink, and other sources.
Although their study uses 13,712 named human genes
and millions of MEDLINE abstracts, no extensive quan-
titative results are reported and analyzed. Friedman, et
al. (2001) extract a pathway relation for various biologi-
cal entities from a variety of articles. In their work, the
precision of the experiments is high (from 79-96%).
However, the recalls are relatively low (from 21-72%).
Bunescu et al. (2004) conducted protein/protein interac-
tion identification with several learning methods, such as
pattern matching rule induction (RAPIER), boosted wrap-
per induction (BWI), and extraction using longest com-
mon subsequences (ELCS). ELCS automatically learns
rules for extracting protein interactions using a bottom-up

Table 1. A summary of works in biomedical entity extraction

Author Named Entities Database No. of Words Learning F Value

Methods
Collier, et al. (2000) Proteins and MEDLINE 30,000 HMM 73
DNA
Krauthammer, et al. Gene and Protein Review articles 5,000 Character 75
(2000) sequence
mapping
Demetriou and Protein, Species, MEDLINE 30,000 PASTA 83
Gaizauskas (2002) and 10 more template filing
Narayanaswamy Protein MEDLINE 302 Hand-crafted 75.86
(2003) rules and co-
occurrence
Yamamoto, et al. Protein GENIA 1,600 abstracts BaseNP 75
(2003) recognition
Lee, et al. (2003) Protein GENIA 10,000 SVM 77
DNA
RNA
Bunescu (2004) Protein MEDLINE 5,206 RAPIER, 57.86
BWI, TBL,
k-NN , SVMs,
MaxEnt

616

TEAM LinG
Information Extraction in Biomedical Literature
Table 2. A summary of relation extraction for biomedical data
Authors Relation Entity
Leek (1997) Location Gene
Blaschke (1999) Interact Protein
Proux (2000) Interact Gene
Pustejovsky (2001) Inhibit Gene
Jenssen (2001) Location Gene
Friedman (2001) Pathway Many
Bunescu (2004) Interact Protein
approach. They conducted experiments in two ways: one
with manually crafted protein names and the other with the
extracted protein names by their name identification method.
In both experiments, Bunescu, et al. compared their results
with human-written rules and showed that machine learn-
ing methods provide higher precisions than human-writ-
ten rules. Table 2 summarizes the works in the areas of
relation extraction in biomedical literature.
Techniques Used
The most commonly used extraction technique is co-
occurrence based. The basic idea of this technique is that
entities are extracted based on frequency of co-occur-
rence of biomedical named entities such as proteins or
genes within sentences. This technique was introduced
by Blaschke, et al. (1999). Their goal was to extract
information from scientific text about protein interac-
tions among a predetermined set of related programs.
Since Blaschke and his colleagues study, numerous
other co-occurrence-based systems have been proposed
in the literature. All are associated with information
extraction of biomedical entities from the unstructured
text corpus. The common denominator of the co-occur-
rence-based systems is that they are based on co-occur-
rences of names or identifiers of entities, typically
along with activation/dependency terms. These systems
are differentiated one from another by integrating dif-
ferent machine learning techniques such as syntactical
analysis or POS tagging, as well as ontologies and con-
trolled vocabularies (Hahn et al., 2002; Pustejovsky et
al., 2002; Yakushiji et al., 2001). Although these tech-
niques are straightforward and easy to develop, from the
performance standpoint, recall and precision are much
lower than any other machine-learning techniques (Ray
& Craven, 2001).
In parallel with co-occurrence-based systems, the
researchers began to investigate other machine learning
or NLP techniques. One of the earliest studies was done
by Leek (1997), who utilized Hidden Markov Models
(HMMs) to extract sentences discussing gene location of
chromosomes. HMMs are applied to represent sentence
structures for natural language processing, where states
DB Learning Precision Recall I
Methods
OMIM HMM 80% 36%
MEDLINE Co-occurrence n/a n/a
Flybase Co-occurrence 81% 44%
MEDLINE Co-occurrence 90% 57%
MEDLINE Co-occurrence n/a n/a
Articles Co-occurrence 96% 63%
and thesauri
MEDLINE RAPIER, BWI, n/a n/a
ELCS
of an HMM correspond to candidate POS tags, and
probabilistic transitions among states represent pos-
sible parses of the sentence, according to the matches of
the terms occurring in it to the POSs. In the context of
biomedical literature mining, HMM is also used to model
families of biological sequences as a set of different
utterances of the same word generated by an HMM
technique (Baldi et al., 1994).
Ray and Craven (2001) have proposed a more sophis-
ticated HMM-based technique to distinguish fact-bear-
ing sentences from uninteresting sentences. The target
biological entities and relations that they intend to ex-
tract are protein subcellular localizations and gene-dis-
order associations. With a predefined lexicon of loca-
tions and proteins and several hundreds of training
sentences derived from Yeast database, they trained and
tested the classifiers over a manually labeled corpus of
about 3,000 MEDLINE abstracts. There have been sev-
eral studies applying natural language tagging and pars-
ing techniques to biomedical literature mining. Friedman,
et al. (2001) propose methods parsing sentences and using
thesauri to extract facts about genes and proteins from
biomedical documents. They extract interactions among
genes and proteins as part of regulatory pathways.
Evaluation
One of the pivotal issues yet to be explored further in
biomedical literature mining is how to evaluate the
techniques or systems. The focus of the evaluation
conducted in the literature is on extraction accuracy.
The accuracy measures used in IE are precision and
recall ratio. For a set of N items, where N is either
terms, sentences, or documents, and the system needs
to label each of the terms as positive or negative,
according to some criterion (positive, if a term belongs
to a predefined document category or a term class). As
discussed earlier, the extraction accuracy is measured
by precision and recall ratio. Although these evaluation
techniques are straightforward and are well accepted,
recall ratios often are criticized in the field of information
retrieval, when the total number of true positive terms is
not clearly defined.
617
TEAM LinG

In IE, an evaluation forum similar to TREC in informa-
tion retrieval (IR) is the Message Understanding Confer-
ence (MUC). Participants in MUC tested the ability of
their systems to identify entities in text to resolve co-
reference, extract and populate attributes of entities, and
perform various other extraction tasks from written text.
As identified by Shatkay and Feldman (2003), the impor-
tant challenge in biomedical literature mining is the cre-
ation of gold-standards and critical evaluation methods
for systems developed in this very active field. The
framework of evaluating biomedical literature mining sys-
tems was recently proposed by Hirschman, et al. (2002).
According to Hirschman, et al. (2002), the following ele-
ments are needed for a successful evaluation: (1) chal-
lenging problem; (2) task definition; (3) training data; (4)
test data; (5) evaluation methodology and implementa-
tion; (6) evaluator; (7) participants; and (8) funding. In
addition to these elements for evaluation, the existing
biomedical literature mining systems encounter the is-
sues of portability and scalability, and these issues need
to be taken into consideration of the framework for evalu-
ation.
Data Sources
In terms of data sources from which target biomedical
objects are extracted, most of the biomedical data min-
ing systems focus on mining MEDLINE abstracts of
National Library of Medicine. The principal reason for
relying on MEDLINE is related to complexity. Ab-
stracts occasionally are easier to mine, since many
papers contain less precise and less well supported
sections in the text that are difficult to distinguish from
more informative sections by machines (Andrade &
Bork, 2000). The current version of MEDLINE contains
nearly 12 million abstracts stored on approximately
43GB of disk space. A prominent example of methods
that target entire papers is still restricted to a small
number of journals (Friedman et al., 2000; Krauthammer
et al., 2002). The task of unraveling information about
function from MEDLINE abstracts can be approached
from two different viewpoints. One approach is based
on computational techniques for understanding texts
written in natural language with lexical, syntactical, and
semantic analysis. In addition to indexing terms in docu-
ments, natural language processing (NLP) methods ex-
tract and index higher-level semantic structures com-
posed of terms and relationships between terms. How-
ever, this approach is confronted with the variability,
fuzziness, and complexity of human language (Andrade
& Bork, 2000). The Genies system (Friedman et al.,
2000; Krauthammer et al., 2002), for automatically
gathering and processing of knowledge about molecular
pathways, and the Information Finding from Biological
618
Information Extraction in Biomedical Literature
Papers (IFBP) transcription factor database are natural
language processing based systems.
An alternative approach that may be more relevant in
practice is based on the treatment of text with statistical
methods. In this approach, the possible relevance of
words in a text is deduced from the comparison of the
frequency of different words in this text with the fre-
quency of the same words in reference sets of text.
Some of the major methods using the statistical ap-
proach are AbXtract and the automatic pathway discov-
ery tool of Ng and Wong (1999). There are advantages
to each of these approaches (i.e., grammar or pattern
matching). Generally, the less syntax that is used, the
more domain-specific the system is. This allows the
construction of a robust system relatively quickly, but
many subtleties may be lost in the interpretation of
sentences. Recently, GENIA corpus has been used for
extracting biomedical-named entities (Collier et al., 2000;
Yamamoto et al., 2003). The reason for the recent surge
of using GENIA corpus is because GENIA provides anno-
tated corpus that can be used for all areas of NLP and IE
applied to the biomedical domain that employs super-
vised learning. With the explosion of results in molecular
biology, there is an increased need for IE to extract
knowledge to build databases and to search intelligently
for information in online journal collections.
FUTURE TRENDS
With the taxonomy proposed here, we now identify the
research trends of applying IE to mine biomedical literature.
1. A variety of biomedical objects and relations are
to be extracted.
2. Rigorous studies are conducted to apply advanced
IE techniques, such as Random Common Field and
Max Entropy based HMM to biomedical data.
3. Collaborative efforts to standardize the evaluation
methods and the procedures for biomedical litera-
ture mining.
4. Continue to broaden the coverage of curated data-
bases and extend the size of the biomedical data-
bases.
CONCLUSION
The sheer size of biomedical literature triggers an in-
tensive pursuit for effective information extraction
tools. To cope with such demand, the biomedical litera-
ture mining emerges as an interdisciplinary field that
information extraction and machine learning are applied
to the biomedical text corpus.
TEAM LinG
Information Extraction in Biomedical Literature
In this article, we approached the biomedical literature
mining from an IE perspective. We attempted to synthe-
size the research efforts made in this emerging field. In
doing so, we showed how current information extraction
can be used successfully to extract and organize informa-
tion from the literature. We surveyed the prominent meth-
ods used for information extraction and demonstrated
their applications in the context of biomedical literature
mining
The following four aspects were used in classifying
the current works done in the field: (1) what to extract;
(2) what techniques are used; (3) how to evaluate; and
(4) what data sources are used. The taxonomy proposed
in this article should help identify the recent trends and
issues pertinent to the biomedical literature mining.
REFERENCES
Andrade, M.A., & Bork, P. (2000). Automated extrac-
tion of information in molecular biology. FEBS Letters,
476,12-7.
Blaschke, C., Andrade, M.A., Ouzounis, C., & Valencia, A.
(1999). Automatic extraction of biological information
from scientific text: Protein-protein interactions, Pro-
ceedings of the First International Conference on Intel-
ligent Systems for Molecular Biology.
Bunescu, R. et al. (2004). Comparative experiments on
learning information extractors for proteins and their
interactions [to be published]. Journal Artificial Intelli-
gence in Medicine on Summarization and Information
Extraction from Medical Documents.
Collier, N., Nobata,C., & Tsujii, J. (2000). Extracting the
names of genes and gene products with a hidden Markov
model. Proceedings of the 18th International Confer-
ence on Computational Linguistics (COLING2000).
De Bruijn, B., & Martin, J. (2002). Getting to the (c)ore of
knowledge: Mining biomedical literature. International
Journal of Medical Informatics, 67, 7-18.
Demetriou, G., & Gaizauskas, R. (2002). Utilizing text
mining results: The pasta Web system. Proceedings of
the Workshop on Natural Language Processing in the
Biomedical Domain.
Friedman, C., Kra, P., Yu, H., Krauthammer, M., &
Rzhetsky, A. (2001). GENIES: A natural-language pro-
cessing system for the extraction of molecular path-
ways from journal articles. Bioinformatics, 17, S74-82.
Hahn, U., Romacker, M., & Schulz, S. (2002). Creating
knowledge repositories from biomedical reports: The
MEDSYNDIKATE text mining system. Proceedings of
the Pacific Symposium on Biocomputing. I
Hirschman, L., Park, J.C., Tsujii, J., Wong, L., & Wu, C.H.
(2002). Accomplishments and challenges in literature data
mining for biology. Bioinformatics, 18(12), 1553-1561.
Jenssen, T.K., Laegreid, A., Komorowski, J., & Hovig, E.
(2001). A literature network of human genes for high-
throughput analysis of gene expression. Nature Genet-
ics, 28(1), 21-8.
Krauthammer, M., Rzhetsky, A., Morozov P., & Friedman,
C. (2000). Using BLAST for identifying gene and protein
names in journal articles. Gene, 259(1-2), 245-252.
Lee, K., Hwang, Y., & Rim, H. (2003). Two-phase
biomedical NE recognition based on SVMs. Proceed-
ings of the ACL 2003 Workshop on Natural Language
Processing in Biomedicine.
Leek, T.R. (1997). Information extraction using hidden
Markov models [masters theses]. San Diego, CA: De-
partment of Computer Science, University of California.
Narayanaswamy, M., Ravikumar, K.E., & Vijay-Shanker,
K. (2003). A biological named entity recognizer. Pro-
ceedings of the Pacific Symposium on Biocomputing.
Ng, S.K., & Wong, M. (1999). Toward routine automatic
pathway discovery from on-line scientific text abstracts.
Proceedings of the Genome Informatics Series: Work-
shop on Genome Informatics.
Proux, D., Rechenmann, F., & Julliard, L. (2000). A
pragmatic information extraction strategy for gathering
data on genetic interactions. Proceedings of the Inter-
national Conference on Intelligent System for Mo-
lecular Biology.
Pustejovsky, J., Castano, J., Zhang, J., Kotecki, M., &
Cochran, B. (2002). Robust relational parsing over bio-
medical literature: extracting inhibit relations. Pacific
Symposium on Biocomputing (pp. 362-73).
Ray, S., & Craven, M. (2001). Representing sentence struc-
ture in hidden Markov models for information extraction.
Proceedings of the 17th International Joint Conference on
Artificial Intelligence, Seattle, Washington.
Shatkay, H., & Feldman, R. (2003). Mining the biomedical
literature in the genomic era: An overview. Journal of
Computational Biology, 10(6), 821-855.
Yakushiji, A., Tateisi, Y., Miyao,Y., & Tsujii, J. (2001).
Event extraction from biomedical papers using a full
parser. Proceedings of the Pacific Symposium on
Biocomputing.
619
TEAM LinG

Yamamoto, K., Kudo, T., Konagaya, A., & Matsumoto, Y.
(2003). Protein name tagging for biomedical annotation in
text. Proceedings of the ACL 2003 Workshop on Natural
Language Processing in Biomedicine.
KEY TERMS
F-Value: Combines recall and precision in a single
efficiency measure (it is the harmonic mean of preci-
sion and recall): F = 2 * (recall * precision) / (recall +
precision).
Hidden Markov Model (HMM): A statistical model
where the system being modeled is assumed to be a
Markov process with unknown parameters, and the chal-
lenge is to determine the hidden parameters from the
observable parameters, based on this assumption.
Natural Language Processing (NLP): A subfield of
artificial intelligence and linguistics. It studies the prob-
620
Information Extraction in Biomedical Literature
lems inherent in the processing and manipulation of
natural language.
Part of Speech (POS): A classification of words ac-
cording to how they are used in a sentence and the types
of ideas they convey. Traditionally, the parts of speech
are the noun, pronoun, verb, adjective, adverb, preposi-
tion, conjunction, and interjection.
Precision: The ratio of the number of correctly filled
slots to the total number of slots the system filled.
Recall: Denotes the ratio of the number of slots the
system found correctly to the number of slots in the
answer key.
Support Vector Machine (SVM): A learning ma-
chine that can perform binary classification (pattern
recognition) as well as multi-category classification
and real valued function approximation (regression es-
timation) tasks.
TEAM LinG

Instance Selection
Huan Liu
Arizona State University, USA
Lei Yu
Arizona State University, USA
INTRODUCTION
The amounts of data have become increasingly large in
recent years as the capacity of digital data storage world-
wide has significantly increased. As the size of data
grows, the demand for data reduction increases for effec-
tive data mining. Instance selection is one of the effective
means to data reduction. This article introduces the basic
concepts of instance selection and its context, necessity,
and functionality. The article briefly reviews the state-of-
the-art methods for instance selection.
Selection is a necessity in the world surrounding us.
It stems from the sheer fact of limited resources, and
data mining is no exception. Many factors give rise to
data selection: Data is not purely collected for data
mining or for one particular application; there are miss-
ing data, redundant data, and errors during collection and
storage; and data can be too overwhelming to handle.
Instance selection is one effective approach to data
selection. This process entails choosing a subset of data
to achieve the original purpose of a data-mining applica-
tion. The ideal outcome is a model independent, mini-
mum sample of data that can accomplish tasks with little
or no performance deterioration.
BACKGROUND AND MOTIVATION
When we are able to gather as much data as we wish, a
natural question is How do we efficiently use it to our
advantage? Raw data is rarely of direct use, and manual
analysis simply cannot keep pace with the fast accumu-
lation of massive data. Knowledge discovery and data
mining (KDD), an emerging field comprising disci-
plines such as databases, statistics, and machine learn-
ing, comes to the rescue. KDD aims to turn raw data into
nuggets and create special edges in this ever-competi-
tive world for science discovery and business intelli-
gence. The KDD process is defined as the nontrivial
process of identifying valid, novel, potentially useful,
and ultimately understandable patterns in data
(Fayyad, Piatetsky-Shapiro, Smyth, & Uthurusamy,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
621
I
1996). It includes data selection, preprocessing, data
mining, interpretation, and evaluation. The first two
processes (data selection and preprocessing) play a
pivotal role in successful data mining (Han & Kamber,
2001). Facing the mounting challenges of enormous
amounts of data, much of the current research concerns
itself with scaling up data-mining algorithms (Provost
& Kolluri, 1999). Researchers have also worked on
scaling down the data an alternative to the scaling up
of the algorithms. The major issue of scaling down data
is to select the relevant data and then present it to a data-
mining algorithm. This line of work is parallel with the
work on scaling up algorithms, and the combination of
the two is a two-edged sword in mining nuggets from
massive data.
In data mining, data is stored in a flat file and de-
scribed by terms called attributes or features. Each
line in the file consists of attribute-values and forms an
instance, which is also called a record, tuple, or data
point in a multidimensional space defined by the at-
tributes. Data reduction can be achieved in many ways
(Liu & Motoda, 1998; Blum & Langley, 1997; Liu &
Motoda, 2001). By selecting features, we reduce the
number of columns in a data set; by discretizing feature
values, we reduce the number of possible values of
features; and by selecting instances, we reduce the
number of rows in a data set. We focus on instance
selection here.
Instance selection reduces data and enables a data-
mining algorithm to function and work effectively with
huge data. The data can include almost everything re-
lated to a domain (recall that data is not solely collected
for data mining), but one application normally involves
using one aspect of the domain. It is natural and sensible
to focus on the relevant part of the data for the applica-
tion so that the search is more focused and mining is
more efficient. Cleaning data before mining is often
required. By selecting relevant instances, we can usu-
ally remove irrelevant, noise, and redundant data. The
high-quality data will lead to high-quality results and
reduced costs for data mining.
TEAM LinG

MAJOR LINES OF RESEARCH AND
DEVELOPMENT
A spontaneous response to the challenge of instance
selection is, without fail, some form of sampling. Al-
though sampling is an important part of instance selec-
tion, other approaches do not rely on sampling but
resort to search or take advantage of data-mining algo-
rithms. In this section, we start with sampling methods
and proceed to other instance-selection methods asso-
ciated with data-mining tasks, such as classification and
clustering.
Sampling Methods
Sampling methods are useful tools for instance selec-
tion (Gu, Hu, & Liu, 2001).
Simple random sampling is a method of selecting n
instances out of the N such that every one of the (nN )
distinct samples has an equal chance of being drawn. If
an instance that has been drawn is removed from the data
set for all subsequent draws, the method is called ran-
dom sampling without replacement. Random sampling
with replacement is entirely feasible: At any draw, all N
instances of the data set have an equal chance of being
drawn, no matter how often they have already been
drawn.
Stratified random sampling divides the data set of N
instances into subsets of N1, N2,, Nl instances, respec-
tively. These subsets are nonoverlapping, and together
they comprise the whole data set (i.e., N1+N2,,+Nl =N).
The subsets are called strata. When the strata have been
determined, a sample is drawn from each stratum, the
drawings being made independently in different strata. If
a simple random sample is taken in each stratum, the whole
procedure is described as stratified random sampling. It is
often used in applications when one wishes to divide a
heterogeneous data set into subsets, each of which is
internally homogeneous.
Adaptive sampling refers to a sampling procedure
that selects instances depending on results obtained
from the sample. The primary purpose of adaptive sam-
pling is to take advantage of data characteristics in order
to obtain more precise estimates. It takes advantage of
the result of preliminary mining for more effective
sampling, and vice versa.
Selective sampling is another way of exploiting data
characteristics to obtain more precise estimates in sam-
pling. All instances are first divided into partitions
according to some homogeneity criterion, and then
random sampling is performed to select instances from
each partition. Because instances in each partition are
more similar to each other than instances in other
622
Instance Selection
partitions, the resulting sample is more representative
than a randomly generated one. Recent methods can be
found in Liu, Motoda, and Yu (2002), in which samples
selected from partitions based on data variance result in
better performance than samples selected with random
sampling.
Methods for Labeled Data
One key data-mining application is classification
predicting the class of an unseen instance. The data for
this type of application is usually labeled with class
values. Instance selection in the context of classifica-
tion has been attempted by researchers according to the
classifiers being built. In this section, we include five
types of selected instances.
Critical points are the points that matter the most to
a classifier. The issue originated from the learning
method of Nearest Neighbor (NN) (Cover & Thomas,
1991). NN usually does not learn during the training
phase. Only when it is required to classify a new sample
does NN search the data to find the nearest neighbor for
the new sample, using the class label of the nearest
neighbor to predict the class label of the new sample.
During this phase, NN can be very slow if the data are
large and can be extremely sensitive to noise. There-
fore, many suggestions have been made to keep only the
critical points, so that noisy ones are removed and the
data set is reduced. Examples can be found in Yu, Xu,
Ester, and Kriegel (2001) and Zeng, Xing, and Zhou
(2003), in which critical data points are selected to
improve the performance of collaborative filtering.
Boundary points are the instances that lie on bor-
ders between classes. Support vector machines (SVM)
provide a principled way of finding these points through
minimizing structural risk (Burges, 1998). Using a non-
linear function to map data points to a high-dimen-
sional feature space, a nonlinearly separable data set
becomes linearly separable. Data points on the bound-
aries, which maximize the margin band, are the support
vectors. Support vectors are instances in the original
data sets and contain all the information a given classi-
fier needs for constructing the decision function. Bound-
ary points and critical points are different in the ways
they are found.
Prototypes are representatives of groups of instances
via averaging (Chang, 1974). A prototype that repre-
sents the typicality of a class is used in characterizing a
class rather than describing the differences between
classes. Therefore, they are different from critical points
or boundary points.
Tree-based sampling is a method involving deci-
sion trees (Quinlan, 1993), which are commonly used
classification tools in data mining and machine learn-
TEAM LinG
Instance Selection
ing. Instance selection can be done via the decision tree
built. Breiman and Friedman (1984) propose delegate sam-
pling. The basic idea is to construct a decision tree such
that instances at the leaves of the tree are approximately
uniformly distributed. Delegate sampling then samples
instances from the leaves in inverse proportion to the
density at the leaf and assigns weights to the sampled
points that are proportional to the leaf density.
In real-world applications, although large amounts
of data are potentially available, the majority of data are
not labeled. Manually labeling the data is a labor-inten-
sive and costly process. Researchers investigate whether
experts can be asked to label only a small portion of the
data that is most relevant to the task if labeling all data is
too expensive and time-consuming, a process that is
called instance labeling. Usually an expert can be en-
gaged to label a small portion of the selected data at
various stages. So we wish to select as little data as
possible at each stage and use an adaptive algorithm to
guess what else should be selected for labeling in the
next stage. Instance labeling is closely associated with
adaptive sampling, clustering, and active learning.
Methods for Unlabeled Data
When data are unlabeled, methods for labeled data can-
not be directly applied to instance selection. The wide-
spread use of computers results in huge amounts of data
stored without labels, for example, Web pages, transac-
tion data, newspaper articles, and e-mail messages (Baeza-
Yates & Ribeiro-Neto, 1999). Clustering is one ap-
proach to finding regularities from unlabeled data. We
discuss three types of selected instances here.
Prototypes are pseudo data points generated from the
formed clusters. The idea is that after the clusters are
formed, one may just keep the prototypes of the clusters
and discard the rest of the data points. The k-means
clustering algorithm is a good example of this sort.
Given a data set and a constant k, the k-means clustering
algorithm is to partition the data into k subsets such that
instances in each subset are similar under some measure.
The k means are iteratively updated until a stopping
criterion is satisfied. The prototypes in this case are the
k means.
Bradley, Fayyad, and Reina (1998) extend the k-
means algorithm to perform clustering in one scan of the
data. By keeping some points that defy compression plus
some sufficient statistics, they demonstrate a scalable k-
means algorithm. From the viewpoint of instance selec-
tion, prototypes plus sufficient statistics is a method of
representing a cluster by using both defiant points and
pseudo points that can be reconstructed from sufficient
statistics rather than keeping only the k means. I
Squashed data are some pseudo data points gener-
ated from the original data. In this aspect, they are similar
to prototypes, as both may or may not be in the original
data set. Squashed data points are different from proto-
types in that each pseudo data point has a weight, and
the sum of the weights is equal to the number of instances
in the original data set. Presently, two ways of obtaining
squashed data are (a) model free (DuMouchel, Volinsky,
Johnson, Cortes, & Pregibon, 1999) and (b) model depen-
dent, or likelihood based (Madigan, Raghavan,
DuMouchel, Nason, Posse, & Ridgeway, 2002).
FUTURE TRENDS
As shown in this article, instance selection has been
studied and employed in various tasks, such as sam-
pling, classification, and clustering. Each task is very
unique, as each has different information available and
different requirements. Clearly, a universal model of
instance selection is out of the question. This short
article provides some starting points that can hopefully
lead to more concerted study and development of new
methods for instance selection. Instance selection deals
with scaling down data. When we better understand
instance selection, we will naturally investigate whether
this work can be combined with other lines of research,
such as algorithm scaling-up, feature selection, and
construction, to overcome the problem of huge amounts
of data,. Integrating these different techniques to
achieve the common goal effective and efficient
data mining is a big challenge.
CONCLUSION
With the constraints imposed by computer memory and
mining algorithms, we experience selection pressures
more than ever. The central point of instance selection
is approximation. Our task is to achieve as good of
mining results as possible by approximating the whole
data with the selected instances and, hopefully, to do
better in data mining with instance selection, as it is
possible to remove noisy and irrelevant data in the
process. In this short article, we have presented an
initial attempt to review and categorize the methods of
instance selection in terms of sampling, classification,
and clustering.
623
TEAM LinG

REFERENCES
Baeza-Yates, R., & Ribeiro-Neto, B. (1999). Modern infor-
mation retrieval. Addison-Wesley and ACM Press.
Blum, A., & Langley, P. (1997). Selection of relevant
features and examples in machine learning. Artificial
Intelligence, 97, 245-271.
Bradley, P., Fayyad, U., & Reina, C. (1998). Scaling clus-
tering algorithms to large databases. Proceedings of the
Fourth International Conference on Knowledge Discov-
ery and Data Mining (pp. 9-15).
Breiman, L. & Friedman, J. (1984). Tools for large data set
analysis. In E.J. Wegman & J.G. Smith (Eds.), Statistical
signal processing. New York: M. Dekker.
Burges, C. (1998). A tutorial on support vector machines.
Journal of Data Mining and Knowledge Discovery, 2,
121-167.
Chang, C. (1974). Finding prototypes for nearest neigh-
bor classifiers. IEEE Transactions on Computers, C-23.
Cover, T. M., & Thomas, J. A. (1991). Elements of
information theory. Wiley.
DuMouchel, W., Volinsky, C., Johnson, T., Cortes, C.,
& Pregibon, D. (1999). Squashing flat files flatter.
Proceedings of the Fifth ACM Conference on Knowl-
edge Discovery and Data Mining (pp. 6-15).
Fayyad, U., Piatetsky-Shapiro, G., Smyth, P., &
Uthurusamy, R. (1996). From data mining to knowledge
discovery. Advances in Knowledge Discovery and Data
Mining.
Gu, B., Hu, F., & Liu, H. (2001). Sampling: Knowing
whole from its part. In H. Liu & H. Motoda (Eds.),
Instance selection and construction for data mining.
Boston: Kluwer Academic.
Han, J., & Kamber, M. (2001). Data mining: Concepts
and techniques. Morgan Kaufmann.
Liu, H., & Motoda, H., (1998). Feature selection for
knowledge discovery and data mining. Boston: Kluwer
Academic.
Liu, H., & Motoda, H. (Eds.). (2001). Instance selec-
tion and construction for data mining. Boston: Kluwer
Academic.
Liu, H., Motoda, H., & Yu, L. (2002). Feature selection
with selective sampling. Proceedings of the 19th Interna-
tional Conference on Machine Learning (pp. 395-402).
624
Instance Selection
Madigan, D., Raghavan, N., DuMouchel, W., Nason, M.,
Posse, C., & Ridgeway, G. (2002). Likelihood-based data
squashing: A modeling approach to instance construc-
tion. Journal of Data Mining and Knowledge Discovery,
6(2), 173-190.
Provost, F., & Kolluri, V. (1999). A survey of methods for
scaling up inductive algorithms. Journal of Data Mining
and Knowledge Discovery, 3, 131-169.
Quinlan, R. J. (1993). C4.5: Programs for machine learn-
ing. Morgan Kaufmann.
Yu, K., Xu, X., Ester, M., & Kriegel, H. (2001). Select-
ing relevant instances for efficient and accurate col-
laborative filtering. Proceedings of the 10th Interna-
tional Conference on Information and Knowledge Man-
agement (pp.239-46).
Zeng, C., Xing, C., & Zhou, L. (2003). Similarity mea-
sure and instance selection for collaborative filtering.
Proceedings of the 12th International Conference on
World Wide Web (pp. 652-658).
KEY TERMS
Classification: A process of predicting the classes
of unseen instances based on patterns learned from
available instances with predefined classes.
Clustering: A process of grouping instances into
clusters so that instances are similar to one another
within a cluster but dissimilar to instances in other
clusters.
Data Mining: The application of analytical methods
and tools to data for the purpose of discovering patterns,
statistical or predictive models, and relationships among
massive data.
Data Reduction: A process of removing irrelevant
information from data by reducing the number of fea-
tures, instances, or values of the data.
Instance: A vector of attribute values in a multidi-
mensional space defined by the attributes, also called a
record, tuple, or data point.
Instance Selection: A process of choosing a subset
of data to achieve the original purpose of a data-mining
application as if the whole data is used.
Sampling: A procedure that draws a sample, Si, by a
random process in which each Si receives its appropriate
probability, Pi, of being selected.
TEAM LinG

Integration of Data Sources through
Data Mining
Andreas Koeller
Montclair State University, USA
INTRODUCTION
Integration of data sources refers to the task of develop-
ing a common schema as well as data transformation
solutions for a number of data sources with related
content. The large number and size of modern data
sources make manual approaches at integration increas-
ingly impractical. Data mining can help to partially or
fully automate the data integration process.
BACKGROUND
Many fields of business and research show a tremen-
dous need to integrate data from different sources. The
process of data source integration has two major com-
ponents.
Schema matching refers to the task of identifying
related fields across two or more databases (Rahm &
Bernstein, 2001). Complications arise at several levels,
for example
Source databases can be organized by using sev-
eral different models, such as the relational model,
the object-oriented model, or semistructured
models (e.g., XML).
Information stored in a single table in one rela-
tional database can be stored in two or more tables
in another. This problem is common when source
databases show different levels of normalization
and also occurs in nonrelational sources.
A single field in one database, such as Name, could
correspond to multiple fields, such as First Name
and Last Name, in another.
Data transformation (sometimes called instance
matching) is a second step in which data in matching
fields must be translated into a common format. Fre-
quent reasons for mismatched data include data format
(such as 1.6.2004 vs. 6/1/2004), numeric precision
(3.5kg vs. 3.51kg), abbreviations (Corp. vs. Corpora-
tion), or linguistic differences (e.g., using different
synonyms for the same concept across databases).
Todays databases are large both in the number of
records stored and in the number of fields (dimensions)
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
625
I
for each datum object. Database integration or migra-
tion projects often deal with hundreds of tables and
thousands of fields (Dasu, Johnson, Muthukrishnan, &
Shkapenyuk, 2002), with some tables having 100 or
more fields and/or hundreds of thousands of rows. Meth-
ods of improving the efficiency of integration projects,
which still rely mostly on manual work (Kang & Naughton,
2003), are critical for the success of this important task.
MAIN THRUST
In this article, I explore the application of data-mining
methods to the integration of data sources. Although
data transformation tasks can sometimes be performed
through data mining, such techniques are most useful in
the context of schema matching. Therefore, the follow-
ing discussion focuses on the use of data mining in
schema matching, mentioning data transformation where
appropriate.
Schema-Matching Approaches
Two classes of schema-matching solutions exist:
schema-only-based matching and instance-based match-
ing (Rahm & Bernstein, 2001).
Schema-only-based matching identifies related
database fields by taking only the schema of input data-
bases into account. The matching occurs through lin-
guistic means or through constraint matching. Linguis-
tic matching compares field names, finds similarities in
field descriptions (if available), and attempts to match
field names to names in a given hierarchy of terms
(ontology). Constraint matching matches fields based
on their domains (data types) or their key properties
(primary key, foreign key). In both approaches, the data
in the sources are ignored in making decisions on match-
ing. Important projects implementing this approach in-
clude ARTEMIS (Castano, de Antonellis, & de Capitani
di Vemercati, 2001) and Microsofts CUPID (Madhavan,
Bernstein, & Rahm, 2001).
Instance-based matching takes properties of the
data into account as well. A very simple approach is to
conclude that two fields are related if their minimum
and maximum values and/or their average values are
TEAM LinG

equal or similar. More sophisticated approaches consider
the distribution of values in fields. A strong indicator of
a relation between fields is a complete inclusion of the
data of one field in another. I take a closer look at this
pattern in the following section. Important instance-based
matching projects are SemInt (Li & Clifton, 2000) and LSD
(Doan, Domingos, & Halevy, 2001).
Some projects explore a combined approach, in which
both schema-level and instance-level matching is per-
formed. Halevy and Madhavan (2003) present a Corpus-
based schema matcher. It attempts to perform schema
matching by incorporating known schemas and previous
matching results and to improve the matching result by
taking such historical information into account.
Data-mining approaches are most useful in the context
of instance-based matching. However, some mining-re-
lated techniques, such as graph matching, are employed
in schema-only-based matching as well.
Instance-Based Matching through
Inclusion Dependency Mining
An inclusion dependency is a pattern between two
databases, stating that the values in a field (or set of
fields) in one database form a subset of the values in
some field (or set of fields) in another database. Such
subsets are relevant to data integration for two reasons.
First, fields that stand in an inclusion dependency to one
another might represent related data. Second, knowl-
edge of foreign keys is essential in successful schema
matching. Because a foreign key is necessarily a subset
of the corresponding key in another table, foreign keys
can be discovered through inclusion dependency dis-
covery.
The discovery of inclusion dependencies is a very
complex process. In fact, the problem is in general NP-
hard as a function of the number of fields in the largest
inclusion dependency between two tables. However, a
number of practical algorithms have been published.
De Marchi, Lopes, and Petit (2002) present an algo-
rithm that adopts the idea of levelwise discovery used in
the famous Apriori algorithm for association rule min-
ing. Inclusion dependencies are discovered by first com-
paring single fields with one another and then combining
matches into pairs of fields, continuing the process
through triples, then 4-sets of fields, and so on. How-
ever, due to the exponential growth in the number of
inclusion dependencies in larger tables, this approach
does not scale beyond inclusion dependencies with a
size of about eight fields.
A more recent algorithm (Koeller & Rundensteiner,
2003) takes a graph-theoretic approach. It avoids enu-
merating all inclusion dependencies between two tables
and finds candidates for only the largest inclusion de-
626
Integration of Data Sources through Data Mining
pendencies by mapping the discovery problem to a
problem of discovering patterns (specifically cliques)
in graphs. This approach is able to discover inclusion
dependencies with several dozens of attributes in tables
with tens of thousands of rows. Both algorithms rely on
the antimonotonic property of the inclusion depen-
dency discovery problem. This property is also used in
association rule mining and states that patterns of size k
can only exist in the solution of the problem if certain
patterns of sizes smaller than k exist as well. Therefore,
it is meaningful to first discover small patterns (e.g.,
single-attribute inclusion dependency) and use this in-
formation to restrict the search space for larger patterns.
Instance-Based Matching in the
Presence of Data Mismatches
Inclusion dependency discovery captures only part of
the problem of schema matching, because only exact
matches are found. If attributes across two relations are
not exact subsets of each other (e.g., due to entry
errors), then data mismatches requiring data transfor-
mation, or partially overlapping data sets, it becomes
more difficult to perform data-driven mining-based dis-
covery. Both false negatives and false positives are
possible. For example, matching fields might not be
discovered due to different encoding schemes (e.g., use
of a numeric identifier in one table, where text is used to
denote the same values in another table). On the other
hand, purely data-driven discovery relies on the assump-
tion that semantically related values are also syntacti-
cally equal. Consequently, fields that are discovered by
a mining algorithm to be matching might not be seman-
tically related.
Data Mining by Using Database
Statistics
The problem of false negatives in mining for schema
matching can be addressed by more sophisticated min-
ing approaches. If it is known which attributes across
two relations relate to one another, data transforma-
tion solutions can be used. However, automatic discov-
ery of matching attributes is also possible, usually
through the evaluation of statistical patterns in the data
sources. In the classification of Kang and Naughton
(2003), interpreted matching uses artificial intelligence
techniques, such as Bayesian classification or neural
networks, to establish hypotheses about related at-
tributes. In the uninterpreted matching approach, statis-
tical features, such as the unique value count of an
attribute or its frequency distribution, are taken into
consideration. The underlying assumption is that two
TEAM LinG
Integration of Data Sources through Data Mining
attributes showing a similar distribution of unique values
might be related even though the actual data values are not
equal or similar.
Another approach for detecting a semantic relation-
ship between attributes is to use information entropy
measures. In this approach, the concept of mutual infor-
mation, which is based on entropy and conditional en-
tropy of the underlying attributes, measures the reduc-
tion in uncertainty of one attribute due to the knowledge
of the other attribute (Kang & Naughton, 2003, p. 207).
Further Problems in Information
Integration through Data Mining
In addition to the approaches mentioned previously, sev-
eral other data-mining and machine-learning approaches,
in particular classification and rule-mining techniques,
are used to solve special problems in information inte-
gration.
For example, a common problem occurring in real-
world integration projects is related to duplicate records
across two databases, which must be identified. This
problem is usually referred to as the record-linking
problem or the merge/purge problem (Hernandz &
Stolfo, 1998). Similar statistical techniques as the ones
described previously are used to approach this problem.
The Commonwealth Scientific and Industrial Research
Organisation (2003) gives an overview of approaches,
which include decision models, predictive models such
as support vector machines, and a Bayesian decision cost
model.
In a similar context, data-mining and machine-learn-
ing solutions are used to improve the data quality of
existing databases as well. This important process is
sometimes called data scrubbing. Lbbers, Grimmer,
and Jarke (2003) present a study of the use of such
techniques and refer to the use of data mining in data
quality improvement as data auditing.
Recently, the emergence of Web Services such as
XML, SOAP, and UDDI promises to open opportunities
for database integration. Hansen, Madnick, and Siegel
(2002) argue that Web services help to overcome some
of the technical difficulties of data integration, which
mostly stem from the fact that traditional databases are
not built with integration in mind. On the other hand, Web
services by design standardize data exchange protocols
and mechanisms. However, the problem of identifying
semantically related databases and achieving schema
matching remains.
FUTURE TRENDS
I
Increasing amounts of data are being collected at all
levels of business, industry, and science. Integration of
data also becomes more and more important as busi-
nesses merge and research projects increasingly re-
quire interdisciplinary efforts. Evidence for the need
for solutions in this area is provided by the multitude of
partial software solutions for such business applica-
tions as ETL (Pervasive Software, Inc., 2003), and by
the increasing number of integration projects in the
life sciences, such as Genbank by the National Center
for Biotechnology Information (NCBI) or Gramene by
the Cold Spring Harbor Laboratory and Cornell Univer-
sity. Currently, the integration of data sources is a
daunting task, requiring substantial human resources. If
automatic methods for schema matching were more
readily available, data integration projects could be
completed much faster and could incorporate many
more databases than is currently the case.
Furthermore, an emerging trend in data source inte-
gration is the move from batch-style integration, where
a set of given data sources is integrated at one time into
one system, to real-time integration, where data sources
are immediately added to an integration system as they
become available. Solutions to this new challenge can
also benefit tremendously from semiautomatic or au-
tomatic methods of identifying database structure and
relationships.
CONCLUSION
Information integration is an important and difficult
task for businesses and research institutions. Although
data sources can be integrated with each other by manual
means, this approach is not very efficient and does not
scale to the current requirements. Thousands of data-
bases with the potential for integration exist in every
field of business and research, and many of those
databases have a prohibitively high number of fields
and/or records to make manual integration feasible.
Semiautomatic or automatic approaches to integration
are needed.
Data mining provides very useful tools to automatic
data integration. Mining algorithms are used to identify
schema elements in unknown source databases, to re-
late those elements to each other, and to perform
additional tasks, such as data transformation. Essential
627
TEAM LinG

business tasks such as extraction, transformation, and
loading (ETL) and data integration and migration in gen-
eral become more feasible when automatic methods are
used.
Although the underlying algorithmic problems are
difficult and often show exponential complexity, several
interesting solutions to the schema-matching and data
transformation problems in integration have been pro-
posed. This is an active area of research, and more com-
prehensive and beneficial applications of data mining to
integration are likely to emerge in the near future.
REFERENCES
Castano, S., de Antonellis, V., & de Capitani di Vemercati,
S. (2001). Global viewing of heterogeneous data sources.
IEEE Transactions on Knowledge and Data Engineer-
ing, 13(2), 277-297.
Commonwealth Scientific and Industrial Research
Organisation. (2003, April). Record linkage: Current
practice and future directions (CMIS Tech. Rep. No.
03/83). Canberra, Australia: L. Gu, R. Baxter, D. Vickers,
& C. Rainsford. Retrieved July 22, 2004, from http://
www.act.cmis.csiro.au/rohanb/PAPERS/record
_linkage.pdf
Dasu, T., Johnson, T., Muthukrishnan, S., & Shkapenyuk,
V. (2002). Mining database structure; or, how to build a
data quality browser. Proceedings of the 2002 ACM
SIGMOD International Conference on Management
of Data, USA (pp. 240-251).
de Marchi, F., Lopes, S., & Petit, J.-M. (2002). Efficient
algorithms for mining inclusion dependencies. Pro-
ceedings of the Eighth International Conference on
Extending Database Technology, Prague, Czech Repub-
lic, 2287 (pp. 464-476).
Doan, A. H., Domingos, P., & Halevy, A. Y. (2001). Rec-
onciling schemas of disparate data sources: A machine-
learning approach. Proceedings of the ACM SIGMOD
International Conference on Management of Data, USA
(pp. 509-520).
Halevy, A. Y., & Madhavan, J. (2003). Corpus-based
knowledge representation. Proceedings of the 18th
International Joint Conference on Artificial Intelli-
gence, Mexico (pp. 1567-1572).
Hernndez, M. A., & Stolfo, S. J. (1998). Real-world data
is dirty: Data cleansing and the merge/purge problem.
Journal of Data Mining and Knowledge Discovery, 2(1),
9-37.
628
Integration of Data Sources through Data Mining
Kang, J., & Naughton, J. F. (2003). On schema matching
with opaque column names and data values. Proceed-
ings of the ACM SIGMOD International Conference
on Management of Data, USA (pp. 205-216).
Koeller, A., & Rundensteiner, E. A. (2003). Discovery of
high-dimensional inclusion dependencies. Proceedings
of the 19th IEEE International Conference on Data
Engineering, India (pp. 683-685).
Li, W., & Clifton, C. (2000). SemInt: A tool for identifying
attribute correspondences in heterogeneous databases
using neural network. Journal of Data and Knowledge
Engineering, 33(1), 49-84.
Lbbers, D., Grimmer, U., & Jarke, M. (2003). Systematic
development of data mining-based data quality tools.
Proceedings of the 29th International Conference on
Very Large Databases, Germany (pp. 548-559).
Madhavan, J., Bernstein, P. A., & Rahm, E. (2001) Generic
schema matching with CUPID. Proceedings of the 27th
International Conference on Very Large Databases,
Italy (pp. 49-58).
Massachusetts Institute of Technology, Sloan School of
Management. (2002, May). Data integration using
Web services (Working Paper 4406-02). Cambridge, MA.
M. Hansen, S. Madnick, & M. Siegel. Retrieved July 22,
2004, from http://hdl.handle.net/1721.1/1822
Pervasive Software, Inc. (2003). ETL: The secret weapon
in data warehousing and business intelligence.
[Whitepaper]. Austin, TX: Pervasive Software.
Rahm, E., & Bernstein, P. A. (2001). A survey of ap-
proaches to automatic schema matching. VLDB Jour-
nal, 10(4), 334-350.
KEY TERMS
Antimonotonic: A property of some pattern-find-
ing problems stating that patterns of size k can only exist
if certain patterns with sizes smaller than k exist in the
same dataset. This property is used in levelwise algo-
rithms, such as the Apriori algorithm used for associa-
tion rule mining or some algorithms for inclusion de-
pendency mining.
Database Schema: A set of names and conditions
that describe the structure of a database. For example, in
a relational database, the schema includes elements
such as table names, field names, field data types, pri-
mary key constraints, or foreign key constraints.
TEAM LinG
Integration of Data Sources through Data Mining
Domain: The set of permitted values for a field in a
database, defined during database design. The actual data
in a field are a subset of the fields domain.
Extraction, Transformation, and Loading (ETL): De-
scribes the three essential steps in the process of data
source integration: extracting data and schema from the
sources, transforming it into a common format, and load-
ing the data into an integration database.
Foreign Key: A key is a field or set of fields in a
relational database table that has unique values, that is,
no duplicates. A field or set of fields whose values form
a subset of the values in the key of another table is
called a foreign key. Foreign keys express relationships
between fields of different tables.
Inclusion Dependency: A pattern between two da-
tabases, stating that the values in a field (or set of fields)
in one database form a subset of the values in some field
(or set of fields) in another database.
Levelwise Discovery: A class of data-mining algo-
rithms that discovers patterns of a certain size by first I
discovering patterns of size 1, then using information
from that step to discover patterns of size 2, and so on. A
well-known example of a levelwise algorithm is the Apriori
algorithm used to mine association rules.
Merge/Purge: The process of identifying duplicate
records during the integration of data sources. Related
data sources often contain overlapping information ex-
tents, which have to be reconciled to improve the quality
of an integrated database.
Relational Database: A database that stores data in
tables, which are sets of tuples (rows). A set of corre-
sponding values across all rows of a table is called an
attribute, field, or column.
Schema Matching: The process of identifying an
appropriate mapping from the schema of an input data
source to the schema of an integrated database.
629
TEAM LinG
630

Intelligence Density
David Sundaram
The University of Auckland, New Zealand

Victor Portougal
The University of Auckland, New Zealand

INTRODUCTION Figure 1. Steps for increasing intelligence density

(Dhar & Stein, 1997)
The amount of information that decision makers have to
process has been increasing at a tremendous pace. A few Knowledge
years ago it was suggested that information in the world
was doubling every 16 months. The very volume has
prevented this information from being used effectively.
Another problem that compounds the situation is the
fact that the information is neither easily accessible nor
available in an integrated manner. This has led to the oft-
quoted comment that though computers have promised
a fount of wisdom they have swamped us with a flood of
data. Decision Support Systems (DSS) and related deci-
sion support tools like data warehousing and data mining
have been used to glean actionable information and
nuggets from this flood of data. Data

BACKGROUND need to have the ability to scrub or cleanse the data of

Dhar and Stein (1997) define Intelligence Density (ID)
as the amount of useful decision support information
that a decision maker gets from using a system for a
certain amount of time. Alternately ID can be defined as
the amount of time taken to get the essence of the
underlying data from the output. This is done using the
utility concept, initially developed in decision theory
and game theory (Lapin & Whisler, 2002). Numerical
utility values, referred to as utilities (sometimes called
utiles) express the true worth of information. These
values are obtained by constructing a special utility
function. Thus intelligence density can be defined more
formally as follows:
Utilities of decision making power gleaned (quality)
Intelligence Density = -----------------------------------------------------------------
Units of analytic time spent by the decision maker
errors. After scrubbing the data we need to have tools
and technologies that will allow us to integrate data in a
flexible manner. This integration should support not
only data of different formats but also data that are not
of the same type.
Enterprise Systems/Enterprise Resource Planning
(ERP) systems with their integrated databases have pro-
vided clean and integrated view of a large amount of
information within the organization thus supporting the
lower levels of the intelligence density pyramid (Figure 2).
But even in the biggest and best organizations with
massive investments in ERP systems we still find the
need for data warehouses and OLAP even though they
predominantly support the lower levels of the intelli-
gence density pyramid. Once we have an integrated view
of the data we can use data mining and other decision
support tools to transform the data and discover patterns
and nuggets of information from the data.

Increasing the intelligence density of its data en-

ables an organization to be more effective, productive,
and flexible. Key processes that allow one to increase
the ID of data are illustrated in Figure 1. Mechanisms
that will allow us to access different types of data need
to be in place first. Once we have access to the data we
MAIN THRUST
Three key technologies that can be leveraged to over-
come the problems associated with information of low

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
Intelligence Density

Figure 2. ERP and DSS support for increasing homonyms and synonyms. The key steps that need to be
intelligence density (Adapted from Shafiei and undertaken to transform raw data to a form that can be I
Sundaram, 2004) stored in a Data Warehouse for analysis are:

K now ledge
Learn D SS
D iscover
Transform
Integrate
Scrub ER P
A ccess System
D ata
intelligence density are Data Warehousing (DW), Online
Analytical Processing (OLAP), and Data Mining (DM).
These technologies have had a significant impact on the
design and implementation of DSS. A generic decision
support architecture that incorporates these technolo-
gies is illustrated in Figure 3. This architecture high-
lights the complimentary nature of data warehousing,
OLAP, and data mining. The data warehouse and its
related components support the lower end of the intel-
ligence density pyramid by providing tools and tech-
nologies that allow one to extract, load, cleanse, con-
vert, and transform the raw data available in an
organisation into a form that then allows the decision
maker to apply OLAP and data mining tools with ease.
The OLAP and data mining tools in turn support the
middle and upper levels of the intelligence density
pyramid. In the following paragraphs we look at each of
these technologies with a particular focus on their abil-
ity to increase the intelligence density of data.
The extraction and loading of the data into the Data
Warehouse environment from a number of sys-
tems on a periodic basis
Conversion of the data into a format that is appro-
priate to the Data Warehouse
Cleansing of the data to remove inconsistencies,
inappropriate values, errors, etc
Integration of the different data sets into a form
that matches the data model of the Data Warehouse
Transformation of the data through operations such
as summarisation, aggregation, and creation of de-
rived attributes.
Once all these steps have been completed the data is
ready for further processing. While one could use dif-
ferent programs/packages to accomplish the various
steps listed above they could also be conducted within a
single environment. For example, Microsoft SQL Server
(2004) provides the Data Transformation Services by
which raw data from organisational data stores can be
loaded, cleansed, converted, integrated, aggregated, sum-
marized, and transformed in a variety of ways.
Figure 3. DSS architecture incorporating data
warehouses, OLAP, and data mining (Adapted from
Srinivasan et al., 2000)

Data Warehousing

Data warehouses are fundamental to most information

system architectures and are even more crucial in DSS
architectures. A Data Warehouse is not a DSS, but a Data
Warehouse provides data that is integrated, subject-
oriented, time-variant, and non-volatile in a bid to
support decision making (Inmon, 2002). There are a
number of processes that needs to be undertaken before
data can enter a Data Warehouse or be analysed using
OLAP or Data Mining Tools. Most Data Warehouses
reside on relational DBMS like ORACLE, Microsoft
SQL Server, or DB2. The data from which the Data
Warehouses are built can exist on varied hardware and
software platforms. The data quite often also needs to be
extracted from a number of different sources from
within as well as without the organization. This requires
the resolution of many data integration issues such as

631

TEAM LinG

OLAP
OLAP can be defined as the creation, analysis, ad hoc
querying, and management of multidimensional data
(Thomsen, 2002). Predominantly the focus of most OLAP
systems is on the analysis and ad hoc querying of the
multidimensional data. Data warehousing systems are
usually responsible for the creation and management of
the multidimensional data. A superficial understanding
might suggest that there does not seem to be much of a
difference between data warehouses and OLAP. This is
due to the fact, that both are complimentary technologies
with the aim of increasing the intelligence density of data.
OLAP is a logical extension to the data warehouse. OLAP
and related technologies focus on providing support for
the analytical, modelling, and computational requirements
of decision makers. While OLAP systems provide a me-
dium level of analysis capabilities most of the current
crop of OLAP systems do not provide the sophisticated
modeling or analysis functionalities of data mining, math-
ematical programming, or simulation systems.
Data Mining
Data mining can be defined as the process of identifying
valid, novel, useful, and understandable patterns in data
through automatic or semiautomatic means (Berry &
Linoff, 1997). Data mining borrows techniques that origi-
nated from diverse fields such as computer science,
statistics, and artificial intelligence. Data mining is now
being used in a range of industries and for a range of tasks
in a variety of contexts (Wang, 2003). The complexity of
the field of data mining makes it worthwhile to structure
it into goals, tasks, methods, algorithms, and algorithm
implementations. The goals of data mining drive the
tasks that need to be undertaken, and the tasks drive the
methods that will be applied. The methods that will be
applied, drives the selections of algorithms followed by
the choice of algorithm implementations.
The goals of data mining are description, prediction,
and/or verification. Description oriented tasks include
clustering, summarisation, deviation detection, and vi-
sualization. Prediction oriented tasks include classifica-
tion and regression. Statistical analysis techniques are
predominantly used for verification. Methods or tech-
niques to carry out these tasks are many, chief among
them are: neural networks, rule induction, market basket,
cluster detection, link, and statistical analysis. Each
method may have several supporting algorithms and in
turn each algorithm may be implemented in a different
manner. Data mining tools such as Clementine (SPSS,
2004) not only support the discovery of nuggets but also
support the entire intelligence density pyramid by pro-
viding a sophisticated visual interactive environment.
632
Intelligence Density
FUTURE TRENDS
There are two key trends that are evident in the commer-
cial as well as the research realm. The first is the comple-
mentary use of various decision support tools such as
data warehousing, OLAP, and data mining in a synergis-
tic fashion leading to information of high intelligence
density. Another subtle but vital trend is the ubiquitous
inclusion of data warehousing, OLAP, and data mining in
most information technology architectural landscapes.
This is especially true of DSS architectures.
CONCLUSION
In this chapter we first defined intelligence density and
the need for decision support tools that would provide
intelligence of a high density. We then introduced
three emergent technologies integral in the design and
implementation of DSS architectures whose prime pur-
pose is to increase the intelligence density of data. We
introduced and described data warehousing, OLAP, and
data mining briefly from the perspective of their ability
to increase intelligence density of data. We also pro-
posed a generic decision support architecture that
complementarily uses data warehousing, OLAP, and
data mining.
REFERENCES
Berry, M.J.A., & Linoff, G. (1997). Data mining tech-
niques: For marketing, sales, and customer support.
John Wiley & Sons Inc.
Berson, A., & Smith, S.J. (1997). Data warehousing, Data
mining, & OLAP. McGraw-Hill.
Dhar, V., & Stein, R. (1997). Intelligent decision support
methods: The science of knowledge work. Prentice Hall.
Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996).
The KDD process for extracting useful knowledge
from volumes of data. Communications of the ACM,
39(11), 27-34.
Inmon, W.H. (2002). Building the data warehouse. John
Wiley & Sons.
Kimball, R., & Ross, M. (2002). The data warehouse
toolkit: The complete guide to dimensional model-
ing. John Wiley & Sons.
Lapin, L., & Whisler, W.D. (2002). Quantitative decision
making with spreadsheet applications. Belmont, CA:
Duxbury/Thomson Learning.
TEAM LinG
Intelligence Density
Microsoft. (2004). Microsoft SQL Server. Retrieved from
http://www.microsoft.com/
Shafiei, F., & Sundaram, D. (2004, January 5-8). Multi-
enterprise collaborative enterprise resource planning and
decision support systems. Thirty-Seventh Hawaii Inter-
national Conference on System Sciences (CD/ROM).
SPSS. (2004). Clementine. Retrieved from http://
www.spss.com
Srinivasan, A., Sundaram, D., & Davis, J. (2000). Imple-
menting decision support systems. McGraw Hill.
Thomsen, E. (2002). OLAP solutions: Building multidi-
mensional information systems (2nd ed.). New York;
Chichester, UK: Wiley.
Wang, J. (2003). Data mining: Opportunities and chal-
lenges. Hershey, PA: Idea Group Publishing.
Westphal, C., & Blaxton, T. (1998). Data mining solu-
tions: Methods and tools for solving real-world prob-
lems. John Wiley & Sons.
KEY TERMS
Data Mining: can be defined as the process of
identifying valid, novel, useful, and understandable pat-
terns in data through automatic or semiautomatic means
ultimately leading to the increase of the intelligence den-
sity of the raw input data.
Data Warehouses: provide data that are integrated,
subject-oriented, time-variant, and non-volatile thereby I
increasing the intelligence density of the raw input data.
Decision Support Systems/Tools: in a wider sense
can be defined as systems/tools that affect the way
people make decisions. But in our present context could
be defined as systems that increase the intelligence
density of data.
Enterprise Resource Planning /Enterprise Sys-
tems: are integrated information systems that support
most of the business processes and information system
requirements in an organization.
Intelligence Density: is the useful decision sup-
port information that a decision maker gets from using
a system for a certain amount of time or alternately the
amount of time taken to get the essence of the underly-
ing data from the output.
Online Analytical Processing (OLAP): enables the
creation, management, analysis, and ad hoc querying of
multidimensional data thereby increasing the intelligence
density of the data already available in data warehouses.
Utilities (Utiles): are numerical utility values, ex-
pressing the true worth of information. These values are
obtained by constructing a special utility function.
633
TEAM LinG
634
Intelligent Data Analysis
Xiaohui Liu
Brunel University, UK
INTRODUCTION
Intelligent Data Analysis (IDA) is an interdisciplinary
study concerned with the effective analysis of data. IDA
draws the techniques from diverse fields, including arti-
ficial intelligence, databases, high-performance comput-
ing, pattern recognition, and statistics. These fields often
complement each other (e.g., many statistical methods,
particularly those for large data sets, rely on computation,
but brute computing power is no substitute for statistical
knowledge) (Berthold & Hand 2003; Liu, 1999).
BACKGROUND
The job of a data analyst typically involves problem
formulation, advice on data collection (though it is not
uncommon for the analyst to be asked to analyze data that
have already been collected), effective data analysis, and
interpretation and report of the finding. Data analysis is
about the extraction of useful information from data and
is often performed by an iterative process in which explor-
atory analysis and confirmatory analysis are the two
principal components.
Exploratory data analysis, or data exploration, re-
sembles the job of a detective; that is, understanding
evidence collected, looking for clues, applying relevant
background knowledge, and pursuing and checking the
possibilities that clues suggest.
Data exploration is not only useful for data under-
standing but also helpful in generating possibly interest-
ing hypotheses for a later studynormally a more formal
or confirmatory procedure for analyzing data. Such pro-
cedures often assume a potential model structure for the
data and may involve estimating the model parameters
and testing hypotheses about the model.
Over the last 15 years, we have witnessed two phe-
nomena that have affected the work of modern data
analysts more than any others. First, the size and variety
of machine-readable data sets have increased dramati-
cally, and the problem of data explosion has become
apparent. Second, recent developments in computing
have provided the basic infrastructure for fast data access
as well as many advanced computational methods for
extracting information from large quantities of data. These
developments have created a new range of problems and
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
challenges for data analysts as well as new opportunities
for intelligent systems in data analysis, and have led to the
emergence of the field of Intelligent Data Analysis (IDA),
which draws the techniques from diverse fields, including
artificial intelligence (AI), databases, high-performance
computing, pattern recognition, and statistics. What dis-
tinguishes IDA is that it brings together often comple-
mentary methods from these diverse disciplines to solve
challenging problems with which any individual disci-
pline would find difficult to cope, and to explore the most
appropriate strategies and practices for complex data
analysis.
MAIN THRUST
In this paper, we will explore the main disciplines and
associated techniques as well as applications to help
clarify the meaning of intelligent data analysis, followed
by a discussion of several key issues.
Statistics and Computing: Key
Disciplines
IDA has its origins in many disciplines, principally statis-
tics and computing. For many years, statisticians have
studied the science of data analysis and have laid many
of the important foundations. Many of the analysis meth-
ods and principles were established long before comput-
ers were born. Given that statistics are often regarded as
a branch of mathematics, there has been an emphasis on
mathematics rigor, a desire to establish that something is
sensible on theoretical ground before trying it out on
practical problems (Berthold & Hand, 2003). On the other
hand, the computing community, particularly in machine
learning (Mitchell, 1997) and data mining (Wang, 2003) is
much more willing to try something out (e.g., designing
new algorithms) to see how they perform on real-world
datasets, without worrying too much about the theory
behind it.
Statistics is probably the oldest ancestor of IDA, but
what kind of contributions has computing made to the
subject? These may be classified into three categories.
First, the basic computing infrastructure has been put in
place during the last decade or so, which enables large-
scale data analysis (e.g., advances in data warehousing
TEAM LinG
Intelligent Data Analysis
and online analytic processing, computer networks, desk-
top technologies have made it possible to easily organize
and move the data around for the analysis purpose). The
modern computing processing power also has made it
possible to efficiently implement some of the very
computationally-intensive analysis methods such as sta-
tistical resampling, visualizations, large-scale simulation
and neural networks, and stochastic search and optimiza-
tion methods.
Second, there has been much work on extending tra-
ditional statistical and operational research methods to
handle challenging problems arising from modern data
sets. For example, in Bayesian networks (Ramoni et al.,
2002), where the work is based on Bayesian statistics, one
tries to make the ideas work on large-scale practical
problems by making appropriate assumptions and devel-
oping computationally efficient algorithms; in support
vector machines (Cristianini & Shawe-Taylor, 2000), where
one tries to see how the statistical learning theory (Vapnik,
1998) could be utilized to handle very high-dimensional
datasets in linear feature spaces; and in evolutionary
computation (Eiben & Michalewicz, 1999) one tries to
extend the traditional operational research search and
optimization methods.
Third, new kinds of IDA algorithms have been pro-
posed to respond to new challenges. Here are several
examples of the novel methods with distinctive comput-
ing characteristics: powerful three-dimensional virtual
reality visualization systems that allow gigabytes of data
to be visualized interactively by teams of scientists in
different parts of the world (Cruz-Neira, 2003); parallel and
distributed algorithms for different data analysis tasks
(Zaki & Pan, 2002); so-called any-time analysis algorithms
that are designed for real-time tasks, where the system, if
stopped any time from its starting point, would be able to
give some satisfactory (not optimal) solution (of course,
the more time it has, the better solution would be); induc-
tive logic programming extends the deductive power of
classic logic programming methods to induce structures
from data (Mooney, 2004) ; Association rule learning
algorithms were motivated by the need in retail industry
where customers tend to buy related items (Nijssen &
Kok, 2001), while work in inductive databases attempt to
supply users with queries involving inductive capabili-
ties (De Raedt, 2002). Of course, this list is not meant to
be exhaustive, but it gives some ideas about the kind of
IDA work going on within the computing community.
IDA Applications
Data analysis is performed for a variety of reasons by
scientists, engineers, business communities, medical and
government researchers, and so forth. The increasing size
and variety of data as well as new exciting applications
such as bioinformatics and e-science have called for new
ways of analyzing the data. Therefore, it is a very difficult I
task to have a sensible summary of the type of IDA
applications that are possible. The following is a partial list.
Bioinformatics: A huge amount of data has been
generated by genome-sequencing projects and other
experimental efforts to determine the structures and
functions of biological molecules and to under-
stand the evolution of life (Orengo et al., 2003). One
of the most significant developments in
bioinformatics is the use of high-throughput de-
vices such as DNA microarray technology to study
the activities of thousands of genes in a single
experiment and to provide a global view of the
underlying biological process by revealing, for ex-
ample, which genes are responsible for a disease
process, how they interact and are regulated, and
which genes are being co-expressed and participate
in common biological pathways. Major IDA chal-
lenges in this area include the analysis of very high
dimensional but small sample microarray data, the
integration of a variety of data for constructing
biological networks and pathways, and the han-
dling of very noisy microarray image data.
Medicine and Healthcare: With the increasing de-
velopment of electronic patient records and medical
information systems, a large amount of clinical data
is available online. Regularities, trends, and surpris-
ing events extracted from these data by IDA meth-
ods are important in assisting clinicians to make
informed decisions, thereby improving health ser-
vices (Bellazzi et al., 2001). Examples of such appli-
cations include the development of novel methods
to analyze time-stamped data in order to assess the
progression of disease, autonomous agents for moni-
toring and diagnosing intensive care patients, and
intelligent systems for screening early signs of
glaucoma. It is worth noting that research in
bioinformatics can have significant impact on the
understanding of disease and consequently better
therapeutics and treatments. For example, it has
been found using DNA microarray technology that
the current taxonomy of cancer in certain cases
appears to group together molecularly distinct dis-
eases with distinct clinical phenotypes, suggesting
the discovery of subgroups of cancer (Alizadeh et al., 2000).
Science and Engineering: Enormous amounts of
data have been generated in science and engineer-
ing (Cartwight, 2000) (e.g., in cosmology, chemical
engineering, or molecular biology, as discussed
previously). In cosmology, advanced computational
tools are needed to help astronomers understand
the origin of large-scale cosmological structures
635
TEAM LinG

as well as the formation and evolution of their
astrophysical components (i.e., galaxies, quasars,
and clusters). In chemical engineering, mathemati-
cal models have been used to describe interactions
among various chemical processes occurring in-
side a plant. These models are typically very large
systems of nonlinear algebraic or differential equa-
tions. Challenges for IDA in this area include the
development of scalable, approximate, parallel, or
distributed algorithms for large-scale applications.
Business and Finance: There is a wide range of
successful business applications reported, although
the retrieval of technical details is not always easy,
perhaps for obvious reasons. These applications
include fraud detection, customer retention, cross
selling, marketing, and insurance. Fraud is costing
industries billions of pounds, so it is not surprising
to see that systems have been developed to combat
fraudulent activities in such areas as credit card,
health care, stock market dealing, or finance in gen-
eral. Interesting challenges for IDA include timely
integration of information from different resources
and the analysis of local patterns that represent
deviations from a background model (Hand et al., 2002).
IDA Key Issues
In responding to challenges of analyzing complex data
from a variety of applications, particularly emerging ones
such as bioinformatics and e-science, the following issues
are receiving increasing attention in addition to the develop-
ment of novel algorithms to solve new emerging problems.
Strategies: There is a strategic aspect to data analy-
sis beyond the tactical choice of this or that test,
visualization or variable. Analysts often bring exog-
enous knowledge about data to bear when they
decide how to analyze it. The question of how data
analysis may be carried out effectively should lead
us to having a close look not only at those individual
components in the data analysis process but also at
the process as a whole, asking what would consti-
tute a sensible analysis strategy. The strategy should
describe the steps, decisions, and actions that are
taken during the process of analyzing data to build
a model or answer a question.
Data Quality: Real-world data contain errors and are
incomplete and inconsistent. It is commonly ac-
cepted that data cleaning is one of the most difficult
and most costly tasks in large-scale data analysis
and often consumes most of project resources. Re-
search on data quality has attracted a significant
amount of attention from different communities
and includes statistics, computing, and information
636
Intelligent Data Analysis
systems. Important progress has been made, but
further work is needed urgently to come up with
practical and effective methods for managing differ-
ent kinds of data quality problems in large databases.
Scalability: Currently, technical reports analyzing
really big data are still sketchy. Analysis of big,
opportunistic data (i.e., data collected for an unre-
lated purpose) is beset with many statistical pit-
falls. Much research has been done to develop
efficient, heuristic, parallel, and distributed algo-
rithms that are able to scale well. We will be eager
to see more practical experience shared when ana-
lyzing large, complex, real-world datasets in order
to obtain a deep understanding of the IDA process.
Mapping Methods to Applications: Given that there
are so many methods developed in different com-
munities for essentially the same task (e.g., classi-
fication), what are the important factors in choos-
ing the most appropriate method(s) for a given
application? The most commonly used criterion is
the prediction accuracy. However, it is not always
the only, or even the most important, criterion for
evaluating competing methods. Credit scoring is
one of the most quoted applications where
misclassification cost is more important than pre-
dictive accuracy. Other important factors in decid-
ing that one method is preferable to another in-
clude computational efficiency and interpretability
of methods.
Human-Computer Collaboration: Data analysis is
often an iterative, complex process in which both
analyst and computer play an important part. An
interesting issue is how one can have an effective
analysis environment where the computer will per-
form complex and laborious operations and pro-
vide essential assistance, while the analyst is al-
lowed to focus on the more creative part of the data
analysis using knowledge and experience.
FUTURE TRENDS
There is strong evidence that IDA will continue to gen-
erate a lot of interest in both academic and industrial
communities, given the number of related conferences,
journals, working groups, books, and successful case
studies already in existence. It is almost inconceivable
that this topic will fade in the foreseeable future, since
there are so many important and challenging real-world
problems that demand solutions from this area, and there
are still so many unanswered questions. The debate on
what constitutes intelligent or unintelligent data analy-
sis will carry on for a while.
TEAM LinG
Intelligent Data Analysis
More analysis tools and methods inevitably will ap-
pear, but help in their proper use will not be fast enough.
A tool can be used without an essential understanding of
what it can offer and how the results should be inter-
preted, despite the best intentions of the user. Research
will be directed toward the development of more helpful
middle-ground tools, those that are less generic than
current data analysis software tools but more general than
specialized data analysis applications.
Much of the current work in the area is empirical in
nature, and we are still in the process of accumulating
more experience in analyzing large, complex data. A lot of
heuristics and trial and error have been used in exploring
and analyzing these data, especially the data collected
opportunistically. As time goes by, we will see more
theoretical work that attempts to establish a sounder
foundation for analysts of the future.
CONCLUSION
Statistical methods have been the primary analysis tool,
but many new computing developments have been ap-
plied to the analysis of large and challenging real-world
datasets. Intelligent data analysis requires careful think-
ing at every stage of an analysis process, assessment, and
selection of the most appropriate approaches for the
analysis tasks in hand and intelligent application of rel-
evant domain knowledge. This is an area with enormous
potential, as it seeks to answer the following key ques-
tions. How can one perform data analysis most effectively
(intelligently) to gain new scientific insights, to capture
bigger portions of the market, to improve the quality of
life, and so forth? What are the guiding principles to
enable one to do so? How can one reduce the chance of
performing unintelligent data analysis? Modern datasets
are getting larger and more complex, but the number of
trained data analysts is certainly not keeping up at any
rate. This poses a significant challenge for the IDA and
other related communities such as statistics, data mining,
machine learning, and pattern recognition. The quest for
bridging this gap and for crucial insights into the process
of intelligent data analysis will require an interdisciplinary
effort from all these disciplines.
REFERENCES
Alizadeh, A.A. et al. (2000). Distinct types of diffuse large
b-cell lymphoma identified by gene expression profiling.
Nature, 403, 503-511.
Bellazzi, R., Zupan, B., & Liu, X. (Eds). (2001). Intelligent
data analysis in medicine and pharmacology. London.
Berthold, M., & Hand, D.J. (Eds). (2003). Intelligent data
analysis: An introduction. Springer-Verlag. I
Cartwright, H. (Ed). (2000). Intelligent data analysis in
science. Oxford University Press.
Cristianini, N., & Shawe-Taylor, J. (2000). An introduction
to support vector machines. Cambridge University Press
Cruz-Neira, C. (2003). Computational humanities: The new
challenge for virtual reality. IEEE Computer Graphics
and Applications, 23(3), 10-13.
De Raedt, L. (2002). A perspective on inductive data-
bases. ACM SIGKDD Explorations Newsletter, 4(2), 69-77.
Eiben, A.E., & Michalewicz, Z. (Eds). (2003). Evolution-
ary computation. IOS Press.
Hand, D.J., Adams, N., & Bolton, R. (2002). Pattern detec-
tion and discovery. Lecture Notes in Artificial Intelli-
gence, 2447.
Liu, X. (1999). Progress in intelligent data analysis. Inter-
national Journal of Applied Intelligence, 11(3), 235-240.
Mitchell, T. (1997). Machine learning. McGraw Hill.
Mooney, R. et al. (2004). Relational data mining with
inductive logic programming for link discovery. In H.
Kargupta et al. (Eds.), Data mining: Next generation
challenges and future directions. AAAI Press.
Nijssen, S., & Kok, J. (2001). Faster association rules for
multiple relations. Proceedings of the International Joint
Conference on Artificial Intelligence.
Orengo, C., Jones, D., & Thornton, J. (Eds). (2003).
Bioinformatics: Genes, proteins & computers. BIOS Sci-
entific Publishers.
Ramoni M., Sebastiani P., & Cohen, P. (2002). Bayesian
clustering by dynamics. Machine Learning, 47(1), 91-121.
Vapnik, V.N. (1998). Statistical learning theory. Wiley.
Wang, J. (Ed.). (2003). Data mining: Opportunities and
challenges. Hershey, PA: Idea Group Publishing.
Zaki, M., & Pan, Y. (2002). Recent developments in parallel
and distributed data mining. Distributed and Parallel
Databases: An International Journal, 11(2), 123-127.
KEY TERMS
Bioinformatics: The development and application of
computational and mathematical methods for organiz-
ing, analyzing, and interpreting biological data.
637
TEAM LinG

E-Science: The large-scale science that will increas-
ingly be carried out through distributed global collabora-
tions enabled by the Internet.
Intelligent Data Analysis: An interdisciplinary study
concerned with the effective analysis of data, which
draws the techniques from diverse fields including AI,
databases, high-performance computing, pattern recog-
nition, and statistics.
Machine Learning: A study of how computers can be
used automatically to acquire new knowledge from past
cases or experience or from the computers own experi-
ences.
Noisy Data: Real-world data often contain errors due
to the nature of data collection, measurement, or sensing
638
Intelligent Data Analysis
procedures. They can be incomplete, inaccurate, out-of-
date, or inconsistent.
Support Vector Machines (SVM): Learning machines
that can perform difficult classification and regression
estimation tasks. SVM non-linearly map their n-dimen-
sional input space into a high-dimensional feature space.
In this high-dimensional feature space, a linear classifier
is constructed.
Visualization: Visualization tools to graphically dis-
play data in order to facilitate better understanding of
their meanings. Graphical capabilities range from simple
scatter plots to three-dimensional virtual reality systems.
TEAM LinG

Intelligent Query Answering
Zbigniew W. Ras
University of North Carolina, Charlotte, USA
Agnieszka Dardzinska
Bialystok Technical University, Poland
INTRODUCTION
One way to make query answering system (QAS) intel-
ligent is to assume a hierarchical structure of its at-
tributes. Such systems have been investigated by Cuppens
& Demolombe (1988), Gal & Minker (1988), and
Gaasterland et al. (1992), and they are called coopera-
tive. Any attribute value listed in a query, submitted to
cooperative QAS, is seen as a node of the tree represent-
ing that attribute. If QAS retrieves an empty set of
objects, which match query q in a target information
system S, then any attribute value listed in q can be gener-
alized and the same the number of objects that possibly can
match q in S can increase. In cooperative systems, these
generalizations are usually controlled by users.
Another way to make QAS intelligent is to use knowl-
edge discovery methods to increase the number of
queries which QAS can answer: knowledge discovery
module of QAS extracts rules from a local system S and
requests their extraction from remote sites (if system is
distributed). These rules are used to construct new
attributes and/or impute null or hidden values of at-
tributes in S. By enlarging the set of attributes from
which queries are built and by making information sys-
tems less incomplete, we not only increase the number
of queries which QAS can handle but also increase the
number of retrieved objects.
So, QAS based on knowledge discovery has two
classical scenarios that need to be considered:
In a standalone and incomplete system, associa-
tion rules are extracted from that system and used
to predict what values should replace null values
before queries are answered.
When system is distributed with autonomous sites
and user needs to retrieve objects, from one of
these sites (called client), satisfying query q based
on attributes which are not local for that site, we
search for definitions of these non-local attributes
at remote sites and use them to approximate q (Ras,
2002; Ras & Joshi, 1997; Ras & Dardzinska, 2004).
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
639
I
The goal of this article is to provide foundations and
basic results for knowledge discovery-based QAS.
BACKGROUND
Modern query answering systems area of research is
related to enhancements of query answering systems
into intelligent systems. The emphasis is on problems in
users posing queries and systems producing answers.
This becomes more and more relevant as the amount of
information available from local or distributed infor-
mation sources increases. We need systems not only
easy to use but also intelligent in answering the users
needs. A query answering system often replaces human
with expertise in the domain of interest, thus it is
important, from the users point of view, to compare the
system and the human expert as alternative means for
accessing information.
Knowledge systems are defined as information sys-
tems coupled with a knowledge base simplified in Ras
(2002), Ras and Joshi (1997), and Ras and Dardzinska
(1997) to a set of rules treated as definitions of attribute
values. If information system is distributed with autono-
mous sites, these rules can be extracted either from the
information system, which is seen as local (query was
submitted to that system), or from remote sites. Domains
of attributes in the local information system S and the set
of decision values used in rules from the knowledge base
associated with S form the initial alphabet for the local
query answering system. When the knowledge base as-
sociated with S is updated (new rules are added or some
deleted), the alphabet for the local query answering sys-
tem is automatically changed. In this paper we assume
that knowledge bases for all sites are initially empty.
Collaborative information system (Ras, 2002) learns rules
describing values of incomplete attributes and attributes
classified as foreign for its site called a client. These rules
can be extracted at any site but their condition part should
use, if possible, only terms that can be processed by the
query-answering system associated with the client. When
TEAM LinG

the time progresses more and more rules can be added to
the local knowledge base, which means that some at-
tribute values (decision parts of rules) foreign for the
client are also added to its local alphabet. The choice of
which site should be contacted first, in search for defini-
tions of foreign attribute values, is mainly based on the
number of attribute values common for the client and
server sites. The solution to this problem is given in Ras
(2002).
MAIN THRUST
The technology dimension will be explored to help
clarify the meaning of intelligent query answering based
on knowledge discovery and chase.
Intelligent Query Answering for
Standalone Information System
QAS for an information system is concerned with iden-
tifying all objects in the system satisfying a given de-
scription. For example, an information system might
contain information about students in a class and clas-
sify them using four attributes of hair color, eye
color, gender, and size. A simple query might be to Null value replacement by values suggested either by
find all students with brown hair and blue eyes. When an
information system is incomplete, students having brown
hair and unknown eye color can be handled by either
including or excluding them from the answer to the
query. In the first case we talk about optimistic approach
to query evaluation while in the second case we talk
about pessimistic approach. Another option to handle
such a query would be to discover rules for eye color in
terms of the attributes hair color, gender, and size.
These rules could then be applied to students with
unknown eye color to generate values that could be used
in answering the query. Consider that in our example one
of the generated rules said:
(hair, brown) (size, medium) (eye, brown).
Thus, if one of the students having brown hair and
medium size has no value for eye color, then the query
answering system should not include this student in the
list of students with brown hair and blue eyes. Attributes
hair color and size are classification attributes and eye
color is the decision attribute.
We are also interested in how to use this strategy to
build intelligent QAS for incomplete information sys-
tems. If a query is submitted to information system S,
the first step of QAS is to make S as complete as
possible. The approach proposed in Dardzinska & Ras
640
Intelligent Query Answering
(2003b) is to use not only functional dependencies to
chase S (Atzeni & DeAntonellis, 1992) but also use rules
discovered from a complete subsystem of S to do the
chasing.
In the first step, intelligent QAS identifies all in-
complete attributes used in a query. An attribute is
incomplete in S if there is an object in S with incomplete
information on this attribute. The values of all incom-
plete attributes are treated as concepts to be learned (in
a form of rules) from S.
Incomplete information in S is replaced by new data
provided by Chase algorithm based on these rules. When
the process of removing incomplete vales in the local
information system is completed, QAS finds the answer
to query in a usual way.
Intelligent Query Answering for
Distributed Autonomous Information
Systems
Semantic inconsistencies are due to different interpre-
tations of attributes and their values among sites (for
instance one site can interpret the concept young
differently than other sites). Different interpretations
are also due to the way each site is handling null values.
statistical or knowledge discovery methods is quite
common before a user query is processed by QAS.
Ontology (Guarino, 1998; Sowa, 1999, 2000; Van
Heijst et al., 1997) is a set of terms of a particular
information domain and the relationships among them.
Currently, there is a great deal of interest in the devel-
opment of ontologies to facilitate knowledge sharing
among information systems.
Ontologies and inter-ontology relationships between
them are created by experts in the corresponding do-
main, but they can also represent a particular point of
view of the global information system by describing
customized domains. To allow intelligent query pro-
cessing, it is often assumed that an information system
is coupled with some ontology. Inter-ontology relation-
ships can be seen as semantical bridges between ontolo-
gies built for each of the autonomous information sys-
tems so they can collaborate and understand each other.
In Ras and Dardzinska (2004), the notion of optimal
rough semantics and the method of its construction have
been proposed. Rough semantics can be used to model
semantic inconsistencies among sites due to different
interpretations of incomplete values of attributes. Dis-
tributed chase (Ras & Dardzinska, 2004) is a chase-type
algorithm, driven by a client site of a distributed infor-
mation system (DIS), which is similar to chase algo-
rithms based on knowledge discovery and presented in
TEAM LinG
Intelligent Query Answering
Dardzinska & Ras (2003a, 2003b). Distributed chase has
one extra feature in comparison to other chase-type algo-
rithms: the dynamic creation of knowledge bases at all sites
of DIS involved in the process of solving a query submit-
ted to the client site of DIS.
The knowledge base at the client site may contain
rules extracted from the client information system and
also rules extracted from information systems at remote
sites in DIS. These rules are dynamically updated through
the incomplete values replacement process (Ras &
Dardzinska, 2004).
Although the names of attributes are often the same
among sites, their semantics and granularity levels may
differ from site to site. As the result of these differ-
ences, the knowledge bases at the client site and at
remote sites have to satisfy certain properties in order to
be applicable in a distributed chase.
So, assume that system S = (X,A,V), which is a part of
DIS, is queried be user.
Chase algorithm, to be applicable to S, has to be based
on rules from the knowledge base D associated with S,
which satisfies the following conditions:
1. Attribute value used in decision part of a rule from
D has the granularity level either equal to or finer
than the granularity level of the corresponding
attribute in S.
2. The granularity level of any attribute used in the
classification part of a rule from D is either equal
or softer than the granularity level of the corre-
sponding attribute in S.
3. Attribute used in the decision part of a rule from D
either does not belong to A or is incomplete in S.
Assume again that S=(X,A,V) is an information sys-
tem (Pawlak, 1991; Ras & Dardzinska, 2004), where X is
a set of objects, A is a set of attributes (seen as partial
functions from X into 2(V[0,1])) and, V is a set of values of
attributes from A. By [0,1] we mean the set of real
numbers from 0 to 1. Let L(D)={[t vc] D: c In(A)}
be a set of all rules (called a knowledge-base) extracted
initially from the information system S by ERID
(Dardzinska & Ras, 2003c), where In(A) is a set of
incomplete attributes in S.
Assume now that query q(B) is submitted to system
S=(X,A,V), where B is the set of all attributes used in
q(B) and that A B . All attributes in B - [A B] are
called foreign for S. If S is a part of a distributed infor-
mation system, definitions of foreign attributes for S can
be extracted at its remote sites (Ras, 2002). Clearly, all
semantic inconsistencies and differences in granularity
of attribute values among sites have to be resolved first.
In Ras & Dardzinska (2004) only different granularity of
attribute values and different semantics related to differ-
ent interpretations of incomplete attribute values among
sites have been considered. I
In Ras (2002), it was shown that query q(B) can be
processed at site S by discovering definitions of values
of attributes from B - [A B] at the remote sites for S
and next use them to answer q(B).
Foreign attributes for S in B, can be also seen as
attributes entirely incomplete in S, which means values
(either exact or partially incomplete) of such attributes
should be ascribed by chase to all objects in S before
query q(B) is answered. The question remains, if values
discovered by chase are really correct?
Classical approach to this kind of problem is to
build a simple DIS environment (mainly to avoid diffi-
culties related to different granularity and different
semantics of attributes at different sites). As the test-
ing data set (Ras & Dardzinska, 2005) have taken 10,000
tuples randomly selected from a database of some insur-
ance company. This sample table, containing 100 at-
tributes, was randomly partitioned into four subtables of
equal size containing 2,500 tuples each. Next, from each
of these subtables 40 attributes (columns) have been
randomly removed leaving four data tables of the size
2,50060 each. One of these tables was called a client and
the remaining 3 have been called servers. Now, for all
objects at the client site, values of one of the attributes,
which were chosen randomly, have been hidden. This
attribute is denoted by d. At each server site, if attribute
d was listed in its domain schema, descriptions of d using
See5 software (data are complete so it was not necessary
to use ERID) have been learned. All these descriptions,
in the form of rules, have been stored in the knowledge
base of the client. Distributed Chase was applied to
predict what is the real value of the hidden attribute for
each object x at the client site. The threshold value =
0.125 was used to rule out all values predicted by distrib-
uted Chase with confidence below that threshold. Al-
most all hidden values (2476 out of 2500) have been
discovered correctly (assuming = 0.125).
Distributed Chase and Security
Problem of Hidden Attributes
Assume now that an information system S=(X,A,V)
is a part of DIS and attribute bA has to be hidden. For
that purpose, we construct Sb=(X,A,V) to replace S,
where:
1. aS(x) = aSb(x), for any a A-{b}, x X,
2. bSb(x) is undefined, for any x X,
3. bS(x) Vb.
641
TEAM LinG

Users are allowed to submit queries to S b and not to S.
What about the information system Chase(Sb)? How it
differs from S?
If bS(x) = bChase(Sb)(x), where x X, then values of
additional attributes for object x have to be hidden in Sb
to guarantee that value bS(x) can not be reconstructed by
Chase. In Ras and Dardzinska (2005) it was shown how to
identify the minimal number of such values.
FUTURE TRENDS
One of the main problems related to semantics of an
incomplete information system S is the freedom of how
new values are constructed to replace incomplete values
in S, before any rule extraction process begins. This
replacement of incomplete attribute values in some of
the slots in S can be done either by chase or/and by a
number of available statistical methods (Giudici, 2003).
This implies that semantics of queries submitted to S
and driven (defined) by query answering system QAS
based on chase may often differ. Although rough seman-
tics can be used by QAS to handle this problem, we still
have to look for new alternate methods.
Assuming different semantics of attributes among
sites in DIS, the use of global ontology or local ontolo-
gies built jointly with inter-ontology relationships among
them seems to be necessary for solving queries in DIS
using knowledge discovery and chase. Still a lot of
research has to be done in this area.
CONCLUSION
Assume that the client site in DIS is represented by
partially incomplete information system S. When a
query is submitted to S, its query answering system QAS
will replace S by Chase(S) and next will solve the query
using, for instance, the strategy proposed in Ras & Joshi
(1997). Rules used by Chase can be extracted from S or
from its remote sites in DIS assuming that all differ-
ences in semantics of attributes and differences in
granularity levels of attributes are resolved first. We
can argue here why the resulting information system
obtained by Chase can not be stored aside and reused
when a new query is submitted to S? If system S is not
frequently updated, we can do that by keeping a copy of
Chase(S) and next reusing that copy when a new query is
submitted to S. But, the original information system S
still has to be kept so when user wants to enter new data
to S, they can be stored in the original system. System
Chase(S), if stored aside, can not be reused by QAS
when the number of updates in the original S exceeds a
642
Intelligent Query Answering
given threshold value. It means that the new updated
information system S has to be chased again before any
query is answered by QAS.
REFERENCES
Atzeni, P., & DeAntonellis, V. (1992). Relational data-
base theory. The Benjamin Cummings Publishing Com-
pany.
Cuppens, F., & Demolombe, R. (1988). Cooperative
answering: A methodology to provide intelligent access
to databases. In Proceedings of the Second Interna-
tional Conference on Expert Database Systems (pp.
333-353).
Dardzinska, A., & Ras, Z.W. (2003a). Rule-based Chase
algorithm for partially incomplete information sys-
tems. In Proceedings of the Second International
Workshop on Active Mining, Maebashi City, Japan (pp.
42-51).
Dardzinska, A., & Ras, Z.W. (2003b). Chasing unknown
values in incomplete information systems. In Proceed-
ings of ICDM03 Workshop on Foundations and New
Directions of Data Mining, Melbourne, Florida (pp. 24-
30). IEEE Computer Society.
Dardzinska, A., & Ras, Z.W. (2003c). On rule discovery
from incomplete information systems. In Proceedings
of ICDM03 Workshop on Foundations and New Di-
rections of Data Mining. Melbourne, Florida (pp. 31-35).
IEEE Computer Society.
Gaasterland, T., Godfrey, P., & Minker, J. (1992). Relax-
ation as a platform for cooperative answering. Journal of
Intelligent Information Systems, 1(3), 293-321.
Gal, A., & Minker, J. (1988). Informative and cooperative
answers in databases using integrity constraints. In natu-
ral language understanding and logic programming
(pp. 288-300). North Holland.
Giudici, P. (2003). Applied data mining: Statistical meth-
ods for business and industry. West Sussex, UK: Wiley.
Guarino, N. (Ed.). (1998). Formal ontology in information
systems. Amsterdam: IOS Press.
Pawlak, Z. (1991). Rough sets-theoretical aspects of
reasoning about data. Kluwer.
Ras, Z. (2002). Reducts-driven query answering for
distributed knowledge systems. International Journal
of Intelligent Systems, 17(2), 113-124.
TEAM LinG
Intelligent Query Answering
Ras, Z., & Dardzinska, A. (2004). Ontology based distrib-
uted autonomous knowledge systems. Information Sys-
tems International Journal, 29(1), 47-58.
Ras, Z., & Dardzinska, A. (2005). Data security and null
value imputation in distributed information systems. In
Advances in Soft Computing, Proceedings of MSRAS04
Symposium (pp. 133-146). Poland: Springer-Verlag.
Ras, Z., & Joshi, S. (1997). Query approximate answering
system for an incomplete DKBS. Fundamenta Informaticae,
30(3), 313-324.
Sowa, J.F. (1999). Ontological categories. In L.
Albertazzi (Ed.), Shapes of forms: From Gestalt psychol-
ogy and phenomenology to ontology and mathematics
(pp. 307-340). Kluwer.
Sowa, J.F. (2000). Knowledge representation: Logi-
cal, philosophical, and computational foundations.
Pacific Grove, CA: Brooks/Cole Publishing.
Van Heijst, G., Schreiber, A., & Wielinga, B. (1997).
Using explicit ontologies in KBS development. Inter-
national Journal of Human and Computer Studies, 46
(2/3), 183-292.
KEY TERMS
Autonomous Information System: Information
system existing as an independent entity.
Chase: Kind of a recursive strategy applied to a
database V, based on functional dependencies or rules
extracted from V, by which a null value or an incomplete
value in V is replaced by a new more complete value.
Distributed Chase: Kind of a recursive strategy ap-
plied to a database V, based on functional dependencies I
or rules extracted both from V and other autonomous
databases, by which a null value or an incomplete value
in V is replaced by a new more complete value. Any
differences in semantics among attributes in the involved
databases have to be resolved first.
Intelligent Query Answering: Enhancements of query
answering systems into sort of intelligent systems (ca-
pable or being adapted or molded). Such systems should
be able to interpret incorrectly posed questions and
compose an answer not necessarily reflecting precisely
what is directly referred to by the question, but rather
reflecting what the intermediary understands to be the
intention linked with the question.
Knowledge Base: A collection of rules defined as
expressions written in predicate calculus. These rules
have a form of associations between conjuncts of values
of attributes.
Ontology: An explicit formal specification of how
to represent objects, concepts and other entities that are
assumed to exist in some area of interest and relation-
ships holding among them. Systems that share the same
ontology are able to communicate about domain of
discourse without necessarily operating on a globally
shared theory. System commits to ontology if its ob-
servable actions are consistent with the definitions in
the ontology.
Query Semantics: The meaning of a query with an
information system as its domain of interpretation.
Application of knowledge discovery and Chase in query
evaluation makes semantics operational.
Semantics: The meaning of expressions written in
some language, as opposed to their syntax, which de-
scribes how symbols may be combined independently
of their meaning.
643
TEAM LinG
644
Interactive Visual Data Mining
Shouhong Wang
University of Massachusetts Dartmouth, USA
Hai Wang
Saint Marys University, Canada
INTRODUCTION
In the data mining field, people have no doubt that high
level information (or knowledge) can be extracted from the
database through the use of algorithms. However, a one-
shot knowledge deduction is based on the assumption
that the model developer knows the structure of knowl-
edge to be deducted. This assumption may not be invalid
in general. Hence, a general proposition for data mining is
that, without human-computer interaction, any knowl-
edge discovery algorithm (or program) will fail to meet the
needs from a data miner who has a novel goal (Wang, S.
& Wang, H., 2002). Recently, interactive visual data
mining techniques have opened new avenues in the data
mining field (Chen, Zhu, & Chen, 2001; de Oliveira &
Levkowitz, 2003; Han, Hu & Cercone, 2003; Shneiderman,
2002; Yang, 2003).
Interactive visual data mining differs from traditional
data mining, standalone knowledge deduction algorithms,
and one-way data visualization in many ways. Briefly,
interactive visual data mining is human centered, and is
implemented through knowledge discovery loops coupled
with human-computer interaction and visual representa-
tions. Interactive visual data mining attempts to extract
unsuspected and potentially useful patterns from the data
for the data miners with novel goals, rather than to use the
data to derive certain information based on a priori human
knowledge structure.
BACKGROUND
A single generic knowledge deduction algorithm is insuf-
ficient to handle a variety of goals of data mining since a
goal of data mining is often related to its specific problem
domain. In fact, knowledge discovery in databases is the
nontrivial process of identifying valid, novel, potentially
useful, and ultimately understandable patterns of data
mining (Fayyad, Piatetsky-Shapiro, & Smyth, 1996). By
this definition, two aspects of knowledge discovery are
important to meaningful data mining. First, the criteria of
validity, novelty, usefulness of knowledge to be discov-
ered could be subjective. That is, the usefulness of a data
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
pattern depends on the data miner and does not solely
depend on the statistical strength of the pattern. Second,
heuristic search in combinatorial spaces built on com-
puter and human interaction is useful for effective knowl-
edge discovery. One strategy for effective knowledge
discovery is the use of human-computer collaboration.
One technique used for human-computer collabora-
tion in the business information systems field is data
visualization (Bell, 1991; Montazami & Wang, 1988) which
is particularly relevant to data mining (Keim & Kriegel,
1996; Wang, 2002). From the human side of data visualiza-
tion, graphics cognition and problem solving are the two
major concepts of data visualization. It is a commonly
accepted principle that visual perception is compounded
out of processes in a way which is adaptive to the visual
presentation and the particular problem to be solved
(Kosslyn, 1980; Newell & Simon, 1972).
MAIN THRUST
Major components of interactive visual data mining and
their functions that make data mining more effective are
the current research theme in this field. Wang, S. and
Wang, H. (2002) have developed a model of interactive
visual data mining for human-computer collaboration
knowledge discovery. According to this model, an inter-
active visual data mining system has three components
on the computer side, besides the database: data visual-
ization instrument, data and model assembly, and human-
computer interface.
Data Visualization Instrument
Data visualization instruments are tools for presenting
data in human understandable graphics, images, or anima-
tion. While there have been many techniques for data
visualization, such as various statistical charts with col-
ors and animations, the self-organizing maps (SOM)
method based on Kohonen neural network (Kohonen,
1989) has become one of the promising techniques of data
visualization in data mining. SOM is a dynamic system
that can learn the topological relations and abstract struc-
TEAM LinG
Interactive Visual Data Mining
tures in the high-dimensional input vectors using low
dimensional space for representation. These low-dimen-
sional presentations can be viewed and interpreted by
human in discovering knowledge (Wang, 2000).
Data and Model Assembly
The data and model assembly is a set of query functions
that assemble the data and data visualization instruments
for data mining. Query tools are characterized by struc-
tured query language (SQL), the standard query language
for relational database systems. To support human-com-
puter collaboration effectively, query processing is nec-
essary in data mining. As the ultimate objective of data
retrieval and presentation is the formulation of knowl-
edge, it is difficult to create a single standard query
language for all purposes of data mining. Nevertheless,
the following functionalities can be implemented through
the design of query that support the examination of the
relevancy, usefulness, interestingness, and novelty of
extracted knowledge.
1. Schematics Examination: Through this query func-
tion, the data miner is allowed to set different values
for the parameters of the data visualization instru-
ment to perceive various schematic visual presen-
tations.
2. Consistency Examination: To cross-check the data
mining results, the data miner may choose different
sets of data of the database to check if the conclu-
sion from one set of data is consistent with others.
This query function allows the data miner to make
such consistency examination.
3. Relevancy Examination: It is a fundamental law
that, to validate a data mining result, one must use
external data, which are not used in generating this
result but are relevant to the problem being inves-
tigated. For instance, the data of customer at-
tributes can be used for clustering to identify
significant market segments for the company.
However, whether the market segments relevant
to a particular product, one must use separate
product survey data. This query function allows
the data miner to use various external data to
examine the data mining results.
4. Dependability Examination: The concept of de-
pendability examination in interactive visual data
mining is similar to that of factor analysis in tradi-
tional statistical analysis, but the dependability
examination query function is more comprehensive
in determining whether a variable contributes the
data mining results in a certain way.
5. Homogeneousness Examination: Knowledge formula-
tion often needs to identify the ranges of values of a I
determinant variable so that observations with values
of a certain range in this variable have a homogeneous
behavior. This query function provides interactive
mechanism for the data miner to decompose variables
for homogeneousness examination.
Human-Computer Interface
Human-computer interface allows the data miner to dialog
with the computer. It integrates the data base, data visu-
alization instruments, and data and model assembly into
a single computing environment. Through the human-
computer interface, the data miner is able to access the
data visualization instruments, select data sets, invoke
the query process, organize the screen, set colors and
animation speed, and manage the intermediate data min-
ing results.
FUTURE TRENDS
Interactive visual data mining techniques will become key
components of any data mining instruments. More theo-
ries and techniques of interactive visual data mining will
be developed in the near future, followed by comprehen-
sive comparisons of these theories and techniques. Query
systems along with data visualization functions on large-
scale database systems for data mining will be available
for data mining practitioners.
CONCLUSION
Given the fact that a one-shot knowledge deduction may
not provide an alternative result if it fails, we must provide
an integrated computing environment for the data miner
through interactive visual data mining. An interactive
visual data mining system consists of three intertwined
components, besides the database: data visualization
instrument, data and model assembly instrument, and
human-computer interface. In interactive visual data min-
ing, the human-computer interaction and effective visual
presentations of multivariate data allow the data miner to
interpret the data mining results based on the particular
problem domain, his/her perception, specialty, and the
creativity. The ultimate objective of interactive visual
data mining is to allow the data miner to conduct the
experimental process and examination simultaneously
through the human-computer collaboration in order to
obtain a satisfactory result.
645
TEAM LinG

REFERENCES
Bell, P. C. (1991). Visual interactive modelling: The past, the
present, and the prospects. European Journal of Opera-
tional Research, 54(3), 274-286.
Chen, M., Zhu, Q., & Chen, Z. (2001). An integrated
interactive environment for knowledge discovery from
heterogeneous data resources. Information and Software
Technology, 43(8), 487-496.
de Oliveira, M. C. F., & Levkowitz, H. (2003). From visual
data exploration to visual data mining: A survey. IEEE
Transactions on Visualization and Computer Graphics,
9(3), 378-394.
Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). The
KDD process for extracting useful knowledge from vol-
umes of data. Communications of the ACM , 39(11), 27-34.
Han, J., Hu, X., & Cercone, N. (2003). A visualization model
of interactive knowledge discovery systems and its imple-
mentations. Information Visualization, 2(2), 105-112.
Keim, D. A., & Kriegel, H. P. (1996). Visualization tech-
niques for mining large databases: A comparison. IEEE
Transactions on Knowledge & Data Engineering, 8(6),
923-938.
Kohonen, T. (1989). Self-organization and associative
memory (3rd ed.). Berlin: Springer-Verlag.
Kosslyn, S. M. (1980). Image and mind. Cambridge, MA:
Harvard University Press.
Montazemi, A., & Wang, S. (1988). The impact of informa-
tion presentation modes on decision making: A meta-
analysis. Journal of Management Information Systems,
5(3), 101-127.
Newell, A., & Simon, H. A. (1972). Human problem solving.
Englewood Cliffs, NJ: Prentice Hall.
Shneiderman, B. (2002). Inventing discovery tools: Com-
bining information visualization with data mining. Infor-
mation Visualization, 1, 5-12.
Wang, S. (2000) Neural networks. In M. Zeleny (Ed.), IEBM
Handbook of IT in Business (pp. 382-391). London, UK:
International Thomson Business Press.
Wang, S. (2002). Nonlinear pattern hypothesis generation
for data mining. Data & Knowledge Engineering, 40(3),
273-283.
646
Interactive Visual Data Mining
Wang, S., & Wang, H. (2002). Knowledge discovery
through self-organizing maps: Data visualization and
query processing. Knowledge and Information Systems,
4(1), 31-45.
Yang, L. (2003). Visual exploration of large relational data
sets through 3D projections and footprint splatting.
IEEE Transactions on Knowledge and Data Engineer-
ing, 15(6), 1460-1471.
KEY TERMS
Data and Model Assembly: A set of query functions
that assemble the data and data visualization instru-
ments for data mining.
Data Visualization: Presentation of data in human
understandable graphics, images, or animation.
Human-Computer Interface: Integrated computing
environment that allows the data miner to access the data
visualization instruments, select data sets, invoke the
query process, organize the screen, set colors and anima-
tion speed, and manage the intermediate data mining
results.
Interactive Data Mining: Human-computer collabo-
ration knowledge discovery process through the inter-
action between the data miner and the computer to extract
novel, plausible, useful, relevant, and interesting knowl-
edge from the data base.
Query Tool: Structured query language that sup-
ports the examination of the relevancy, usefulness, inter-
estingness, and novelty of extracted knowledge for inter-
active data mining.
Self-Organizing Map (SOM): Two layer neural net-
work that maps the high-dimensional data onto low-
dimensional pictures through unsupervised learning or
competitive learning process. It allows the data miner to
view the clusters on the output maps.
Visual Data Mining: Data mining process through
data visualization. The fundamental concept of visual
data mining is the interaction between data visual pre-
sentation, human graphics cognition, and problem solv-
ing.
TEAM LinG

Interscheme Properties Role in Data
Warehouses
Pasquale De Meo
Universit Mediterranea di Reggio Calabria, Italy
Giorgio Terracina
Universit della Calabria, Italy
Domenico Ursino
Universit Mediterranea di Reggio Calabria, Italy
INTRODUCTION
In this article, we illustrate a general approach for the
semi-automatic construction and management of data
warehouses. Our approach is particularly suited when
the number or the size of involved sources is large and/
or when it changes quite frequently over time. Our
approach is based mainly on the semi-automatic deriva-
tion of interschema properties (i.e., terminological and
structural relationships holding among concepts be-
longing to different input schemas). It consists of the
following steps: (1) enrichment of schema descriptions
obtained by the semi-automatic extraction of
interschema properties; (2) exploitation of derived
interschema properties for obtaining in a data reposi-
tory an integrated and abstracted view of available data;
and (3) design of a three-level data warehouse having as
its core the derived data repository.
BACKGROUND
In the last years, an enormous increase of data available
in electronic form has been witnessed, as well as a
corresponding proliferation of query languages, data
models, and data management systems. Traditional ap-
proaches to data management do not seem to guarantee,
in these cases, the needed level of access transparency
to stored data while preserving the autonomy of local
data sources. This situation contributed to push the
development of new architectures for data source
interoperability, allowing users to query preexisting
autonomous data sources in a way that guarantees model
language and location transparency.
In all the architectures for data source
interoperability, components handling the reconcilia-
tion of involved information sources play a relevant
role. In the construction of these components, schema
integration (Chua, Chiang, & Lim, 2003; dos Santos
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
647
I
Mello, Castano & Heuser, 2002; McBrien &
Poulovassilis, 2003) plays a key role.
However, when involved systems are numerous and/
or large, schema integration alone typically ends up
producing a too complex global schema that may, in
fact, fail to supply a satisfactory and convenient de-
scription of available data. In these cases, schema inte-
gration steps must be completed by executing schema
abstraction steps (Palopoli, Pontieri, Terracina &
Ursino, 2000). Carrying out a schema abstraction activ-
ity amounts to clustering objects belonging to a schema
into homogeneous subsets and producing an abstracted
schema obtained by substituting each subset with one
single object representing it.
In order for schema integration and abstraction to be
correctly carried out, the designer has to understand
clearly the semantics of involved information sources.
One of the most common ways for deriving and repre-
senting schema semantics consists in detecting the so-
called interschema properties (Castano, De Antonellis
& De Capitani di Vimercati, 2001; Doan, Madhavan,
Dhamankar, Domingos & Levy, 2003; Gal, Anaby-Tavor,
Trombetta & Montesi, 2004; Madhavan, Bernstein &
Rahm, 2001; Melnik, Garcia-Molina & Rahm, 2002;
Palopoli, Sacc, Terracina & Ursino, 2003; Palopoli,
Terracina & Ursino, 2001; Rahm & Bernstein, 2001).
These are terminological and structural properties re-
lating to concepts belonging to different schemas.
In the literature, several manual methods for deriv-
ing interschema properties have been proposed (see
Batini, Lenzerini, and Navathe (1986) for a survey about
this argument). These methods can produce very precise
and satisfactory results. However, since they require a
great amount of work, to the human expert, they are
difficult to be applied when involved sources are numer-
ous and large.
To handle large amounts of data, various semi-auto-
matic methods also have been proposed. These are much
less resource consuming than manual ones; moreover,
TEAM LinG

interschema properties obtained by semi-automatic tech-
niques can be updated and maintained more simply. In the
past, semi-automatic methods were based on considering
only structural similarities among objects belonging to
different schemas. Presently, all interschema property
derivation techniques also take into account the context
in which schema concepts have been defined (Rahm &
Bernstein, 2001).
The dramatic increase of available data sources led
also to a large variety of both structured and semi-
structured data formats; in order to uniformly manage
them, it is necessary to exploit a unified paradigm. In
this context, one of the most promising solutions is
XML. Due to its semi-structured nature, XML can be
exploited as a unifying formalism for handling the
interoperability of information sources characterized
by heterogeneous data representation formats.
MAIN THRUST
Overview of the Approach
In this article we define a new framework for uniformly
and semi-automatically constructing a data warehouse
from numerous and large information sources charac-
terized by heterogeneous data representation formats.
In more detail, the proposed framework consists of the
following steps:
Translation of involved information sources into
XML ones.
Application to the XML sources derived in the
previous step of almost automatic techniques for
detecting interschema properties, specifically
conceived to operate on XML environments.
Exploitation of derived interschema properties
for constructing an integrated and uniform repre-
sentation of involved information sources.
Exploitation of this representation as the core of
the reconciled data level of a data warehouse1.
In the following subsections, we will illustrate the
last three steps of this framework. The translation step
is not discussed here, because it is performed by apply-
ing the translation rules from the involved source for-
mats to XML already proposed in the literature.
Extraction of Interschema Properties
A possible taxonomy classifies interschema properties
into terminological properties, subschema similarities,
and structural properties.
648
Interscheme Properties Role in Data Warehouses
Terminological properties are synonymies,
homonymies, hyponymies, overlappings, and type con-
flicts. A synonymy between two concepts A and B indi-
cates that they have the same meaning. A homonymy
between two concepts A and B indicates that they have
the same name but different meanings. Concept A is said
to be a hyponym of concept B (which, in turn, is a
hypernym of A), if A has a more specific meaning than
B. An overlapping exists between concepts A and B, if
they are neither synonyms nor hyponyms of the other
but share a significant set of properties; more formally,
there exists an overlapping between A and B, if there
exist non-empty sets of properties {pA1, pA2, , pAn} of
A and {pB1, pB2, , p Bn} of B such that, for 1i n, pAi is
a synonym of pBi. A type conflict indicates that the same
concept is represented by different constructs (e.g., an
element and an attribute in an XML source) in different
schemas.
A subschema similarity represents a similitude be-
tween fragments of different schemas.
Structural properties are inclusions and assertions
between knowledge patterns. An inclusion between two
concepts A and B indicates that the instances of A are a
subset of the instances of B. An assertion between
knowledge patterns indicates either a subsumption or an
equivalence between knowledge patterns. Roughly speak-
ing, knowledge patterns can be seen as views on involved
information sources.
Our interschema property extraction approach is
characterized by the following features: (1) it is XML-
based; (2) it is almost automatic; and (3) it is semantic.
Given two concepts belonging to different informa-
tion sources, one of the most common ways for deter-
mining their semantics consists of examining their
neighborhoods, since the concepts and the relationships
in which they are involved contribute to define their
meaning. In addition, our approach exploits two further
indicators for defining in a more precise fashion the
semantics of involved data sources. These indicators are
the types and the cardinalities of the elements, taking
the attributes belonging to the XML schemas into con-
sideration.
It is clear from this reasoning that concept neighbor-
hood plays a crucial role in the interschema property
computation. In XML schemas, concepts are expressed
by elements or attributes. Since, for the interschema
property extraction, it is not important to distinguish
concepts represented by elements from concepts repre-
sented by attributes, we introduce the term x-compo-
nent for denoting an element or an attribute in an XML
schema.
In order to compute the neighborhood of an x-com-
ponent, it is necessary to define a semantic distance2
between two x-components of the same schema; it takes
TEAM LinG
Interscheme Properties Role in Data Warehouses
into account how much they are related. The formulas for
computing this distance are quite complex; due to space
limitations we cannot show them here. However, the inter-
ested reader can refer to De Meo, Quattrone, Terracina,
and Ursino (2003) for a detailed illustration of them.
We can define now the neighborhood of an x-compo-
nent. In particular, given an x-component xS of an XML
schema S, the neighborhood of level j of xS consists of all
x-components of S whose semantic distance from xS is
less than or equal to j.
In order to verify if two x-components x1j, belonging
to an XML schema S1, and x2k, belonging to an XML
Schema S2, are synonymous, it is necessary to examine
their neighborhoods. More specifically, first, it is nec-
essary to verify if their nearest neighborhoods (i.e., the
neighborhoods of level 0) are similar. This decision is
made by computing a suitable objective function associ-
ated with the maximum weight matching on a bipartite
graph constructed from the x-components of the neigh-
borhoods into consideration and the lexical synonymies
stored in a thesaurus (e.g., WordNet) 3. If these two
neighborhoods are similar, then x1j and x 2k are assumed to
be synonymous.
However, observe that the neighborhoods of level 0
of x 1j and x2k provide quite a limited vision of their
contexts. If a higher certainty on the synonymy between
x1j and x 2k is required, it is necessary to verify the simi-
larity, not only of their neighborhoods of level 0, but also
of the other neighborhoods. As a consequence, it is
possible to introduce a severity level u against which
interschema properties can be determined, and to say
that x1j and x2k are synonymous with a severity level u, if
all neighborhoods of x1j and x 2k of a level lesser than or
equal to u are similar.
After all synonymies of S1 and S2 have been extracted,
homonymies can be derived. In particular, there exist a
homonymy between two x-components x1j and x2k with a
severity level u if: (1) x1j and x2k have the same name; (2)
both of them are elements or both of them are attributes;
and (3) they are not synonymous with a severity level u.
In other words, a homonymy indicates that two concepts
having the same name represent different meanings.
Due to space constraints, we cannot describe in this
article the derivation of all the other interschema prop-
erties mentioned; however, it follows the same philoso-
phy as the detection of synonymies and homonymies.
The interested reader can find a detailed description of it
in Ursino (2002).
Construction of a Uniform
Representation
Detected interschema properties can be exploited for
constructing a global representation of involved infor-
mation sources; this becomes the core of the reconciled
data level in a three-level DW. I
Generally, in classical approaches, this global rep-
resentation is obtained by integrating all involved data
sources into a unique one. However, when involved
sources are numerous and large, a unique global schema
presumably encodes an enormous number and variety
of objects and becomes far too complex to be used
effectively.
In order to overcome the drawbacks mentioned
previously, our approach does not directly integrate
involved source schemas to construct a global flat
schema. Rather, it first groups them into homogeneous
clusters and then integrates schemas on a cluster-by-
cluster basis. Each integrated schema thus obtained is
then abstracted to construct a global schema represent-
ing the cluster. The aforementioned process is iterated
over the set of obtained cluster schemas, until one
schema is left. In this way, a hierarchical structure is
obtained, which is called Data Repository (DR).
Each cluster of a DR represents a group of homoge-
neous schemas and is, in turn, represented by a schema
(hereafter called C-schema). Clusters of level n of the
hierarchy are obtained by grouping some C-schemas of
level n-1; clusters of level 0 are obtained by grouping
input source schemas. Therefore, each cluster Cl is
characterized by (1) its identifier C-id; (2) its C-schema;
(3) the group of identifiers of clusters whose C-schemas
originated the C-schema of Cl (hereafter called O-
identifiers); (4) the set of interschema properties in-
volving objects belonging to the C-schemas that origi-
nated the C-schema of Cl; and (5) a level index.
It is clear from this reasoning that the three funda-
mental operations for obtaining a DR are (1) schema
clustering (Han & Kumber, 2001), which takes a set of
schemas as input and groups them into semantically
homogeneous clusters; (2) schema integration, which
produces a global schema from a set of heterogeneous
input schemas; and (3) schema abstraction, which
groups concepts of a schema into homogeneous clus-
ters and, in the abstracted schema, represents each
cluster with only one concept.
Exploitation of the Uniform
Representation for Constructing a DW
The Data Repository can be exploited as the core struc-
ture of the reconciled level of a new three-level DW
architecture. Indeed, different from classical three-level
architectures, in order to reconcile data, we do not di-
rectly integrate involved schemas to construct a flat glo-
bal schema. Rather, we first collect subsets of involved
schemas into homogeneous clusters and construct a DR
that is used as the core of the reconciled data level.
649
TEAM LinG

In order to pinpoint the differences between classical
three-level DW architectures and ours, the following
observations can be drawn:
A classical three-level DW architecture is a par-
ticular case of the one proposed here, since it
corresponds to a case where involved sources are
all grouped into one cluster and no abstraction is
carried out over the associated C-schema.
The architecture we propose here is naturally con-
ducive to an incremental DW construction.
In a classical three-level architecture designed
over a large number of sources, presumably hun-
dreds of concepts are represented in the global
schema. In particular, the global schema can be
seen as partitioned into subschemas loosely re-
lated to each other, whereas each subschema con-
tains objects tightly related to each other. Such a
difficulty does not characterize our architecture,
where a source cluster would have been associated
to a precise semantics.
In our architecture, data mart design is presumably
simpler than with classical architectures, since
each data mart will insist over a bunch of data
sources spanned by a subtree of our core DR
rooted at some C-schema of level k, for some k.
In our architecture, reconciled data are (virtually)
represented at various abstraction levels within
the core DR.
It follows from these observations that by paying a
limited price in terms of required space and computa-
tion time, we obtain an architecture that retains all
worths of classical three-level architectures but over-
comes some of their limitations arising when involved
data sources are numerous and large.
FUTURE TRENDS
In the next years, interschema properties presumably
will play a relevant role in various applications involving
heterogeneous data sources. Among them we cite ontol-
ogy matching, semantic Web, e-services, semantic query
processing, Web-based financial services, and biologi-
cal data management.
As an example, in this last application field, the high-
throughput techniques for data collection developed in
the last years have led to an enormous increase of
available biological databases; for instance, the largest
public DNA database contains much more than 20 GB of
data (Hunt, Atkinson & Irving, 2002). Interschema prop-
erties can play a relevant role in this context; indeed,
they can allow the manipulation of biological data sources
650
Interscheme Properties Role in Data Warehouses
possibly relative to different, yet related and comple-
mentary, application contexts.
CONCLUSION
In this article we have illustrated an approach for the
semi-automatic construction and management of data
warehouses. We have shown that our approach is par-
ticularly suited when the number or the size of involved
sources is large and/or when they change quite fre-
quently over time.
Various experiments have been conducted for veri-
fying the applicability of our approach. The interested
reader can find many of them in Ursino (2002), as far as
their application to Italian Central Government Offices
databases is concerned, and in De Meo, Quattrone,
Terracina, and Ursino (2003) for their application to
semantically heterogeneous XML sources.
In the future, we plan to extend our studies on the
role of interschema properties to the various applica-
tion fields mentioned in the previous section.
REFERENCES
Batini, C., Lenzerini, M., & Navathe, S.B. (1986). A
comparative analysis of methodologies for database
scheme integration. ACM Computing Surveys, 15(4),
323-364.
Castano, S., De Antonellis, V., & De Capitani di Vimercati,
S. (2001). Global viewing of heterogeneous data sources.
IEEE Transactions on Data and Knowledge Engineer-
ing, 13(2), 277-297.
Chua, C.E.H, Chiang, R.H.L., & Lim, E.P. (2003). Instance-
based attribute identification in database integration. The
International Journal on Very Large Databases, 12(3),
228-243.
De Meo, P., Quattrone, G., Terracina, G., & Ursino, D.
(2003). Almost automatic and semantic integration of
XML schemas at various severity levels. Proceedings
of the International Conference on Cooperative In-
formation Systems (CoopIS 2003), Taormina, Italy.
Doan, A., Madhavan, J., Dhamankar, R., Domingos, P., &
Halevy, A. (2003). Learning to match ontologies on the
semantic Web. The International Journal on Very
Large Databases, 12(4), 303-319.
dos Santos Mello, R., Castano, S., & Heuser, C.A. (2002).
A method for the unification of XML schemata. Infor-
mation & Software Technology, 44(4), 241-249.
TEAM LinG
Interscheme Properties Role in Data Warehouses
Gal, A., Anaby-Tavor, A., Trombetta, A., & Montesi, D.
(2004). A framework for modeling and evaluating auto-
matic semantic reconciliation. The International Jour-
nal on Very Large Databases [forthcoming].
Han, J. & Kamber, M. (2001). Data mining: Concepts and
techniques. Morgan Kaufmann Publishers.
Hunt, E., Atkinson, M.P., & Irving, R.W. (2002). Database
indexing for large DNA and protein sequence collections.
The International Journal on Very Large Databases,
11(3), 256-271.
Madhavan, J., Bernstein, P.A., & Rahm, E. (2001). Generic
schema matching with cupid. Proceedings of the Interna-
tional Conference on Very Large Data Bases (VLDB
2001), Rome, Italy.
McBrien, P., & Poulovassilis, A. (2003). Data integration
by bi-directional schema transformation rules. Proceed-
ings of the International Conference on Data Engineer-
ing (ICDE 2003), Bangalore, India.
Melnik, S., Garcia-Molina, H., & Rahm, E. (2002). Similarity
flooding: A versatile graph matching algorithm and its
application to schema matching. Proceedings of the Inter-
national Conference on Data Engineering (ICDE 2002),
San Jos, California.
Palopoli, L., Pontieri, L., Terracina, G., & Ursino, D. (2000).
Intensional and extensional integration and abstraction
of heterogeneous databases. Data & Knowledge Engi-
neering, 35(3), 201-237.
Palopoli, L., Sacc, D., Terracina, G., & Ursino, D. (2003).
Uniform techniques for deriving similarities of objects
and subschemas in heterogeneous databases. IEEE Trans-
actions on Knowledge and Data Engineering, 15(2),
271-294.
Palopoli, L., Terracina, G., & Ursino, D. (2001). A graph-
based approach for extracting terminological properties
of elements of XML documents. Proceedings of the
International Conference on Data Engineering (ICDE
2001), Heidelberg, Germany.
Rahm, E., & Bernstein, P.A. (2001). A survey of approaches to
automatic schema matching. The International Journal on
Very Large Databases, 10(4), 334-350.
Ursino, D. (2002). Extraction and exploitation of inten-
sional knowledge from heterogeneous information
sources: Semi-automatic approaches and tools. Springer.
KEY TERMS
I
Assertion Between Knowledge Patterns: A particular
interschema property. It indicates either a subsumption or
an equivalence between knowledge patterns. Roughly
speaking, knowledge patterns can be seen as views on
involved information sources.
Data Repository: A complex catalogue of a set of
sources organizing both their description and all associ-
ated information at various abstraction levels.
Homonymy: A particular interschema property. An
homonymy between two concepts A and B indicates that
they have the same name but different meanings.
Hyponymy/Hypernymy: A particular interschema
property. Concept A is said to be a hyponym of a concept
B (which, in turn, is a hypernym of A), if A has a more
specific meaning than B.
Interschema Properties: Terminological and struc-
tural relationships involving concepts belonging to dif-
ferent sources.
Overlapping: A particular interschema property.
An overlapping exists between two concepts A and B, if
they are neither synonyms nor hyponyms of the other
but share a significant set of properties; more formally,
there exists an overlapping between A and B, if there
exist non-empty sets of properties {pA1, pA2, , pAn} of
A and {pB1, pB2, , pBn} of B such that, for 1i n, pAi is
a synonym of pBi..
Schema Abstraction: The activity that clusters ob-
jects belonging to a schema into homogeneous groups
and produces an abstracted schema obtained by substi-
tuting each group with one single object representing it.
Schema Integration: The activity by which differ-
ent input source schemas are merged into a global
structure representing all of them.
Subschema Similarity: A particular interschema
property. It represents a similitude between fragments
of different schemas.
Synonymy: A particular interschema property. A
synonymy between two concepts A and B indicates that
they have the same meaning.
Type Conflict: A particular interschema property.
It indicates that the same concept is represented by
different constructs (e.g., an element and an attribute in
an XML source) in different schemas.
651
TEAM LinG
 Interscheme Properties Role in Data Warehouses

ENDNOTES 3
Clearly, if necessary, a more specific thesaurus,
possibly constructed with the support of a human
1
Here and in the following, we shall consider a expert, might be used.
three-level data warehouse architecture.
2
Semantic distance is often called connection cost
in the literature.

652

TEAM LinG

Inter-Transactional Association Analysis for
Prediction
Ling Feng
University of Twente, The Netherlands
Tharam Dillon
University of Technology Sydney, Australia
INTRODUCTION
The discovery of association rules from large amounts of
structured or semi-structured data is an important data-
mining problem (Agrawal et al., 1993; Agrawal & Srikant,
1994; Braga et al., 2002, 2003; Cong et al., 2002; Miyahara
et al., 2001; Termier et al., 2002; Xiao et al., 2003). It has
crucial applications in decision support and marketing
strategy. The most prototypical application of associa-
tion rules is market-basket analysis using transaction
databases from supermarkets. These databases contain
sales transaction records, each of which details items
bought by a customer in the transaction. Mining associa-
tion rules is the process of discovering knowledge such
as, 80% of customers who bought diapers also bought
beer, and 35% of customers bought both diapers and beer,
which can be expressed as diaper beer (35%, 80%),
where 80% is the confidence level of the rule, and 35% is
the support level of the rule indicating how frequently the
customers bought both diapers and beer. In general, an
association rule takes the form X Y (s, c), where X and
Y are sets of items, and s and c are support and confidence,
respectively.
BACKGROUND
While the traditional association rules have demonstrated
strong potential in areas such as improving marketing
strategies for the retail industry (Dunham, 2003; Han &
Kamer, 2001), their emphasis is on description rather than
prediction. Such a limitation comes from the fact that
traditional association rules only look at association
relationships among items within the same transactions,
whereas the notion of the transaction could be the items
bought by the same customer, the atmospheric events
that happened at the same time, and so on. To overcome
this limitation, we extend the scope of mining association
rules from such traditional intra-transactional associa-
tions to intertransactional associations for prediction
(Feng et al., 1999, 2001; Lu et al., 2000). Compared to
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
653
I
intratransactional associations, an intertransactional as-
sociation describes the association relationships across
different transactions, such as, if (company) As stock
goes up on day one, Bs stock will go down on day two
but go up on day four. In this case, whether we treat
company or day as the unit of transaction, the associated
items belong to different transactions.
MAIN TRUSTS
Extensions from Intratransaction to
Intertransaction Associations
We extend a series of concepts and terminologies for
intertransactional association analysis. Throughout the
discussion, we assume that the following notation is
used.
A finite set of literals called items I = {i 1, i2, , i n}.
A finite set of transaction records T = {t1, t2, , tl},
where for ti T, t i I.
A finite set of attributes called dimensional at-
tributes A = {a1, a2, , am}, whose domains are finite
subsets of nonnegative integers.
An Enhanced Transactional Database
Model
In classical association analysis, records in a transac-
tional database contain only items. Although transac-
tions occur under certain contexts, such as time, place,
customers, and so forth, such contextual information has
been ignored in classical association rule mining, due to
the fact that such rule mining was intratransactional in
nature. However, when we talk about intertransactional
associations across multiple transactions, the contexts of
occurrence of transactions become important and must be
taken into account.
Here, we enhance the traditional transactional data-
base model by associating each transaction record with
TEAM LinG

a number of attributes that describe the context within
which the transaction happens. We call them dimen-
sional attributes, because, together, these attributes
constitute a multi-dimensional space, and each transac-
tion can be mapped to a certain point in this space.
Basically, dimensional attributes can be of any kind, as
long as they are meaningful to applications. Time, dis-
tance, temperature, latitude, and so forth are typical di-
mensional attributes.
Multidimensional Contexts
An m-dimensional mining context can be defined through
m dimensional attributes a1, a2, , am, each of which
represents a dimension. When m=1, we have a single-
dimensional mining context. Let ni = (ni.a1, ni.a2, , ni.am)
and nj = (nj.a1, nj.a2, , nj.am) be two points in an m-
dimensional space, whose values on the m dimensions are
represented as ni.a1, ni.a2, , ni.am and nj.a1, nj.a2, , nj.am,
respectively. Two points ni and nj are equal, if and only if
for k (1 k m), ni.ak = nj.ak. A relative distance between
ni and nj is defined as ni, nj = (nj.a1-ni.a1, nj.a2-ni.a2, ,
nj.am-ni.am). We also use the notation (d1, d2, , dm), where
dk = nj.ak-ni.a k (1 k m), to represent the relative distance
between two points ni and nj in the m-dimensional space.
Besides, the absolute representation (ni.a1, ni.a2, ,
ni.a m) for point ni, we also can represent it by indicating its
relative distance n0, ni from a certain reference point n0,
(i.e., n0+n0, ni, where ni = n0+n0, ni). Note that ni, n0,
ni, and (ni.a1-n0.a1, ni.a2-n0.a2, , ni.am-n0.am) can be used inter-
changeably, since each of them refers to the same point
ni in the space. Let N = {n1, n 2, , nu} be a set of points in
an m-dimensional space. We construct the smallest refer-
ence point of N, n*, where for k (1 k m), n*.ak = min
(n1.ak, n 2.ak, , nu.ak).
Extended Items (Transactions)
The traditional concepts regarding item and transaction
can be extended accordingly under an m-dimensional
context. We call an item i kI happening at the point (d1,
, (i.e., at the point (n0.a1+d1, n0.a2+d2, , n0.am+dm)),
d2, , dm)
an extended item and denote it as (d1, d2, , dm)(ik). In a
similar fashion, we call a transaction tkT happening at the
point (d1, d2, , dm) an extended transaction and denote it as
D(d1, d2, , dm)(t k). The set of all possible extended items, IE,
is defined as a set of (d1, d2, , dm)(i k) for any ikI at all
possible points (d1, d2, , dm) in the m-dimensional space. TE
is the set of all extended transactions, each of which
contains a set of extended items, in the mining context.
654
Inter-Transactional Association Analysis for Prediction
Normalized Extended Item (Transaction)
Sets
We call an extended itemset a normalized extended itemset,
if all its extended items are positioned with respect to the
smallest reference point of the set. In other words, the
extended items in the set have the minimal relative dis-
tance 0 for each dimension. Formally, let Ie = {(d1,1, d1,2,
(i ), (d2,1, d2,2, , d2,m)(i 2), , (dk,1, dk,2, , dk,m)(ik)} be an
,~d1,m) 1
extended itemset. Ie is a normalized extended itemset, if
and only if for j (1 j k) i (1 i m), min (dj, i) = 0.
The normalization concept can be applied to an ex-
tended transaction set as well. We call an extended trans-
action set a normalized extended transaction set, if all its
extended transactions are positioned with respect to the
smallest reference point of the set. Any non-normalized
extended item (transaction) set can be transformed into a
normalized one through a normalization process, where
the intention is to reposition all the involved extended
items (transactions) based on the smallest reference point
of this set. We use INE and TNE to denote the set of all
possible normalized extended itemsets and normalized
extended transaction sets, respectively. According to the
above definitions, any superset of a normalized extended
item (transaction) set is also a normalized extended item
(transaction) set.
Multidimensional Intertransactional
Association Rule Framework
With the above extensions, we are now in a position to
formally define intertransactional association rules and
related measurements.
Definition 1
A multidimensional intertransactional association rule is
an implication of the form X Y, where
(1) X INE and Y IE;
(2) The extended items in X and Y are positioned with
respect to the same reference point;
(3) For (x1, x2, , xm)(i x) X, (y1, y2, , ym)(ix) Y, xj yj
(1 j m);
(4) X Y = .
Different from classical intratransactional association
rules, the intertransactional association rules capture the
occurrence contexts of associated items. The first clause
TEAM LinG
Inter-Transactional Association Analysis for Prediction
of the definition requires the precedent and antecedent
of an intertransactional association rule to be a normal-
ized extended itemset and an extended itemset, respec-
tively. The second clause of the definition ensures that
items in X and Y are comparable in terms of their contex-
tual positions. For prediction, each of the consequent
items in Y takes place in a context later than any of its
precedent items in X, as stated by the third clause.
Definition 2
Given a normalized extended itemset X and an extended
itemset Y, let |Txy| be the total number of minimal extended
transaction sets that contain XY, |Tx| and the total number
of minimal extended transaction sets that contain X, and |Te|
be the total number of extended transactions in the data-
base. The support and confidence of an intertransactional
association rule X Y are support(X Y) = |T xy|/|T e| and
confidence(X Y) = |Txy| / |T x|.
Discovery of Intertransactional
Association Rules
To investigate the feasibility of mining intertransaction
rules, we extended the classical a priori algorithm and
applied it to weather forecasting. To limit the search space,
we set a maximum span threshold, maxspan, to define a
sliding window along time dimensions. Only the associa-
tions among the items that co-occurred within the window
are of interest. Basically, the mining process of
intertransactional association rules can be divided into
two steps: frequent extended itemset discovery and asso-
ciation rule generation.
1. Frequent Extended Itemset Discovery
In this phase, we find the set of all frequent extended
itemsets. For simplicity, in the following, we use itemset
and extended itemset, transaction and extended transac-
tion interchangeably. Let Lk represent the set of frequent
k-itemsets and Ck the set of candidate k-itemsets. The
algorithm makes multiple passes over the database. Each
pass consists of two phases. First, the set of all frequent
(k-1)-itemsets Lk-1 found in the (k-1)th pass is used to
generate the candidate itemset Ck. The candidate genera-
tion procedure ensures that Ck is a superset of the set of
all frequent k-itemsets. The algorithm now scans the data-
base. For each list of consecutive transactions, it deter-
mines which candidates in Ck are contained and increments
their counts. At the end of the pass, Ck is examined to check
which of the candidates actually are frequent, yielding Lk.
The algorithm terminates when L k becomes empty. In the
following, we detail the procedures for candidate genera-
tion and support counting.
Candidate Generation
I
Pass 1: Let I={1, 2, ..., m} be a set of items in a
database. To generate the candidate set C1 of 1-
itemsets, we need to associate all possible inter-
vals with each item. That is,
C1 = { 0(1), 1(1), , maxspan(1), 0(2), 1(2), ,
maxspan(2),
0(m), 1(m), , maxspan(m) }
Starting from transaction t at the reference point s
(i.e., extended transaction s(t)), the transaction t at the
point s+x in the dimensional space (i.e., extended trans-
action s+x(t)) is scanned to determine whether item i n
exists. If so, the count of x(i n) increases by one. One
scan of the database will deliver the frequent set L1.
Pass 2: We generate a candidate 2-itemset {0(i m),
x(in)} from any two frequent 1-itemsets in L1, 0(im)
and x(i n), and obtain C2 = {{0(i m), x(i n)} | (x=0
im<in) (x
0)}.
Pass k>2: Given Lk-1, the set of all frequent (k-1)-
itemsets, the candidate generation procedure re-
turns a superset of the set of all frequent k-itemsets.
This procedure has two parts. In the join phase, we
join two frequent (k-1)-itemsets in Lk-1 to derive a
candidate k-itemset. Let p={u1 (i 1), u2 (i2),,uk-
(i )} and q={v1 (j 1), v2 (j2),, vk-1 (jk-1)}, where
1 k-1
p, q Lk-1, we have
insert into Ck
select p.u1 (i1),, , p. uk-1 (i k-1), q. vk-1 (jk-1)
from p in Lk-1, q in Lk-1
where (i1=j1 u1=v1) (i k-2=j k-2 uk-2=vk-2) (uk-1<vk-
1
(uk-1=vk-1 ik-1<jk-1))
Next, in the prune phase, we delete all those extended
itemsets in Ck that have some (k-1)-subsets with sup-
ports less than the support threshold.
Support Counting
To facilitate the efficient support counting process, a
candidate Ck of k-itemsets is divided into k groups, with
each group Go containing o number of items whose
intervals are 0 (1 o k). For example, a candidate set of
3-itemsets
655
TEAM LinG

C3={ {0(a), 1(a), 2(b)}, {0(c), (d), 2(d)}, {0(a),
0(b), 3(h)}, {0(l), 0(m), 0(n)}, {0(p), v0(q), 0(r)} ]
is divided into three groups:
G1={{0(a), 1(a), 2(b)}},G2={{0(c), 0(d), 2(d)},
{0(a), 0(b), 3(h)}}, G3={{0(l), 0(m), 0(n)},
{0(p), 0(q), 0(r)} }
Each group is stored in a modified hash-tree. Only
those items with interval 0 participate the construction of
this hash tree (e.g., in G2, only {0(c), 0(d)} and {0(a),
D0(b)} enter the hash-tree). The construction process is
similar to that of a priori. The rest items, 2(d) and 3(h),
are simply attached to the corresponding itemsets, {0(c),
0(d)} and {0(a), 0(b)}, respectively, in the leaves of the
tree.
Upon reading one transaction of the database, every
hash tree is tested. If one itemset is contained, its attached
itemsets whose intervals are larger than 0 will be checked
against the consecutive transactions. In the previous
example, if {0(a), 0(b)} exists in the current extended
transaction at the point s, then the extended transaction
s+3(t) will be scanned to see whether it contains item h.
If so, the support of 3-itemset {0(a), 0(b), 3(h)} will
increase by 1.
Association Rule Generation
Using sets of frequent itemsets, we can find the desired
intertransactional association rules. The generation of
intertransaction association rules is similar to the genera-
tion of the classical association rules.
Application of Intertransactional
Association Rules to Weather
Prediction
We apply the previous algorithm to studying Hong Kong
meterological data. The database records wind direction,
Table 1. Some significant intertransactional association rules found from meteorological data
Extended Association Rules
0(2), 3(13) 4(13)
(If there is an east wind direction and no rain in 18 hours, then
there also will be no rain in 24 hours.)
0(2), 0(25), 2(25) 3(25)
(If it is currently warm with an east wind direction, and still
warm 12 hours later, then it will be continuously warm until 18
hours later.)
656
Inter-Transactional Association Analysis for Prediction
wind speed, temperature, relative humidity, rainfall, and
mean sea level pressure, and so forth every six hours each
day. In some data records, certain atmospheric observa-
tions, such as relative humidity, are missing. Since the
context constituted by the time dimension is valid for the
whole set of meteorological records (transactions), we fill
in these empty fields by averaging their nearby values. In
this way, the data to be mined contains no missing fields;
in addition, no database holes (i.e., meaningless contexts)
exist in the mining space. Essentially, there is one dimen-
sion in this case; namely, time. After preprocessing the
data set, we discover intertransactional association rules
from the 1995 meteorological records and then examine
their prediction accuracy using the 1996 meteorological
data from the same area in Hong Kong. Considering
seasonal changes of weather, we extract records from
May to October for our experiments, totaling 736 records
(total_days * record_num_per_day = (31+30+31+
31+30+31) * 4 = 736) for each year. These raw data sets,
containing continuous data, are further converted into
appropriate formats with which the algorithms can work.
Each record has six meteorological elements (items). The
interval of every two consecutive records is six hours. We
set maxspan=11 in order to detect the association rules for
a three-day horizon (i.e., (11+1) / 4 = 3).
At support=45% and confidence=92%, from the 1995
meteorological data, we found only one classical associa-
tion rule: if the humidity is medium wet, then there is no
rain at the same time (which is quite obvious), but we
found 580 intertransactional association rules. Note that
the number of intertransactional association rules re-
turned depends on the maxspan parameter setting. Table
1 lists some significant intertransactional association
rules found from the single-station meteorological data.
We measure their predictive capabilities using the 1996
meteorological data recorded by the same station through
Prediction-Rate (XY) = sup(XY) / sup(X), which can
achieve more than 90% prediction rate. From the test
results, we find that, with intertransactional association
rules, more comprehensive and interesting knowledge
can be discovered from the databases.
Sup. Conf. Pred. Rate
46% 92% 90%
45% 95% 91.8%
TEAM LinG
Inter-Transactional Association Analysis for Prediction
FUTURE TRENDS
Intertransactional association rule framework provides a
general view of associations among events (Feng et al.,
2001).
Traditional Association Rule Mining: If the dimen-
sional attributes are not present in the knowledge to
be discovered, or those attributes are not interre-
lated, then they can be ignored in analysis, and the
problem becomes the traditional transaction-based
association rule problem.
Mining in Multidimensional Databases or Data
Warehouse: Multidimensional database organizes
data into data cubes with dimensional attributes
and measures, on which multidimensional
intertransactional association mining can be per-
formed.
Mining Spatial Association Rules: Existence of
spatial objects can be viewed as events. The coor-
dinates are dimensional attributes. Related proper-
ties can be added as other dimensional attributes.
CONCLUSION
While the classical association rules have demonstrated
strong potential values, such as improving market strat-
egies for retail industry, they are limited to finding asso-
ciations among items within a transaction. In this article,
we propose a more general form of association rules,
named multi-dimensional intertransaction association
rules. These rules can represent not only the associations
of items within transactions but also associations of items
among different transactions. The classical association
rule can be viewed as a special case of multi-dimensional
intertransaction association rules.
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
associations between sets of items in massive databases.
Proceedings of the ACM SIGMOD International Confer-
ence on Management of Data, Washington, D.C.
Agrawal, R., & Srikant, R. (1994). Fast algorithms for
mining association rules. Proceedings of the 20 th Confer-
ence on Very Large Data Bases, Santiago de Chile, Chile.
Braga, D. et al. (2002). Mining association rules from XML
data. Proceedings of the 4th International Conference
on Data Warehousing and Knowledge Discovery, Aix-en
Provence, France.
Braga, D. et al. (2003). Discovering interesting information
in XML data with association rules. Proceedings of the I
18th Symposium on Applied Computing.
Cong, G., Yi, L., Liu, B., & Wang, K. (2002). Discovering
frequent substructures from hierarchical semi-structured
data. Proceedings of the SIAM DM02, Arlington, Vir-
ginia.
Dunham, M.H. (2003). Data mining: Introductory and
advanced topics. Upper Saddle River, NJ: Prentice Hall.
Feng, L. et al. (1999). Mining multi-dimensional inter-
transaction association rules with templates. Proceed-
ings of the International ACM Conference on Informa-
tion and Knowledge Management, Kansas City, USA.
Feng, L. et al. (2002). A template model for multidimen-
sional inter-transactional association rules. International
Journal of VLDB, 11(2), 153-175.
Feng, L., Dillon, T., & Liu, J. (2001). Inter-transactional
association rules for multidimensional contexts for pre-
diction and their applications to studying meteorological
data. International Journal of Data and Knowledge
Engineering, 37, 85-115.
Feng, L., Li, Q., & Wong, A.K.Y. (2001). Mining inter-
transactional association rules: Generalization and em-
pirical evaluation. Proceedings of the 3rd International
Conference on Data Warehousing and Knowledge Dis-
covery, Munich, Germany.
Han, J., & Kamer, M. (2001). Data mining: Concepts and
techniques. San Diego: Morgan Kaufmann Publishers.
Lu, H., Feng, L., & Han, J. (2000). Beyond intra-transaction
association analysis: Mining multi-dimensional inter-
transaction association rules. ACM Transactions on In-
formation Systems, 18(4), 423-454.
Miyahara, T. et al. (2001). Discovery of frequent tree
structured patterns in semistructured Web documents.
Proceedings of the 5th PAKDD, Hong Kong, China.
Termier, A., Rousset, M., & Sebag, M. (2002). Treefinder:
A first step towards XML data mining. Proceedings of the
IEEE International Conference on Data Mining,
Maebashi City, Japan.
Xiao, Y., Yao, J., Li, Z., & Dunham, M. (2003). Efficicent
data mining for maximal frequent subtrees. Proceedings of
the 3rd IEEE International Conference on Data Mining,
Melbourne, Florida.
657
TEAM LinG
 Inter-Transactional Association Analysis for Prediction
KEY TERMS
Dimensional Attribute: One dimension of an m-di-
mensional context.
Extended Item: An item associated with the context
where it happens.
Extended Transaction: A transaction associated with
the context where it happens.
Intertransactional Association: Correlations among
items not only within the same transactions but also
across different transactions.
658
Intratransactional Association: Correlations among
items within the same transactions.
Multidimensional Context: The context where data-
base transactions, and thus the association relation-
ships, happen.
Normalized Extended Itemset: A set of extended items
whose contextual positions have been positioned with
respect to the smallest reference position of the set.
Normalized Extended Transaction Set: A set of ex-
tended transactions whose contextual positions have
been positioned with respect to the largest reference
position of the set.
TEAM LinG

Interval Set Representations of Clusters
Pawan Lingras
Saint Marys University, Canada
Rui Yan
Saint Marys University, Canada
Mofreh Hogo
Czech Technical University, Czech Republic
Chad West
IBM Canada Limited, Canada
INTRODUCTION
The amount of information that is available in the new
information age has made it necessary to consider vari-
ous summarization techniques. Classification, cluster-
ing, and association are three important data-mining
features. Association is concerned with finding the
likelihood of co-occurrence of two different concepts.
For example, the likelihood of a banana purchase given
that a shopper has bought a cake. Classification and
clustering both involve categorization of objects. Clas-
sification processes a previously known categorization
of objects from a training sample so that it can be
applied to other objects whose categorization is un-
known. This process is called supervised learning.
Clustering groups objects with similar characteristics.
As opposed to classification, the grouping process in
clustering is unsupervised. The actual categorization of
objects, even for a sample, is unknown. Clustering is an
important step in establishing object profiles.
Clustering in data mining faces several additional
challenges compared to conventional clustering appli-
cations (Krishnapuram, Joshi, Nasraoui, & Yi, 2001).
The clusters tend to have fuzzy boundaries. There is a
likelihood that an object may be a candidate for more
than one cluster. Krishnapuram et al. argued that clus-
tering operations in data mining involve modeling an
unknown number of overlapping sets. This article de-
scribes fuzzy and interval set clustering as alternatives
to conventional crisp clustering.
BACKGROUND
Conventional clustering assigns various objects to pre-
cisely one cluster. Figure 1 shows a possible clustering
of 12 objects. In order to assign all the objects to
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
659
I
precisely one cluster, we had to assign objects 4 and 9
to Cluster B. However, the dotted circle seems to rep-
resent a more natural representation of Cluster B. An
ability to specify that Object 9 may either belong to
Cluster B or Cluster C, and Object 4 may belong to
Cluster A or Cluster B, will provide a better representa-
tion of the clustering of the 12 objects in Figure 1.
Rough-set theory provides such an ability.
The notion of rough sets was proposed by Pawlak
(1992). Let U denote the universe (a finite ordinary set),
and let R be an equivalence (indiscernibility) relation on
U. The pair A = (U , R) is called an approximation space.
The equivalence relation R partitions the set U into
disjoint subsets. Such a partition of the universe is
denoted by U / R = {E1 , E 2 , L , Em } , where Ei is an
equivalence class of R. If two elements u, v U belong
to the same equivalence class, we say that u and v are
indistinguishable. The equivalence classes of R are called
the elementary or atomic sets in the approximation
space A = (U , R ) . Because it is not possible to differen-
tiate the elements within the same equivalence class,
one may not be able to obtain a precise representation
for an arbitrary set X U in terms of elementary sets
in A. Instead, its lower and upper bounds may represent
the set X. The lower bound A( X ) is the union of all the
elementary sets, which are subsets of X. The upper
bound A( X ) is the union of all the elementary sets that
have a nonempty intersection with X. The pair
( A( X ), A( X )) is the representation of an ordinary set X
in the approximation space A = (U , R ) , or simply the
rough set of X. The elements in the lower bound of X
definitely belong to X, while elements in the upper
bound of X may or may not belong to X. The pair
TEAM LinG

Figure 1. Conventional clustering
Cluster C
Cluster A
Figure 2. Rough set approximation
( A( X ), A( X )) also provides a set theoretic interval for
the set X. Figure 2 illustrates the lower and upper
approximation of a set.
MAIN THRUST
Interval Set Clustering
Rough sets were originally used for supervised learning.
There are an increasing number of research efforts on
clustering in relation to rough-set theory (do Prado,
Engel, & Filho, 2002; Hirano & Tsumoto, 2003; Peters,
Skowron, Suraj, Rzasa, & Borkowski, 2002). Lingras
(2001) developed rough-set representation of clusters.
Figure 3 shows how the 12 objects from Figure 1 could
be clustered by using rough sets. Instead of Object 9
belonging to any one cluster, it belongs to the upper
bounds of Clusters B and C. Similarly, Object 4 belongs
to the upper bounds of Clusters A and B.
Lingras (2001; Lingras & West, 2004; Lingras,
Hogo, & Snorek, 2004) proposed three different ap-
proaches for unsupervised creation of rough or interval
set representations of clusters: evolutionary, statisti-
cal, and neural. Lingras (2001) described how a rough-
set theoretic clustering scheme could be represented by
using a rough-set genome. The rough-set genome con-
660
Interval Set Representations of Clusters
Cluster B
An equivalence class
Lower approximation
Actual set
Upper approximation
sists of one gene per object. The gene for an object is a
string of bits that describes which lower and upper
approximations the object belongs to. The string for a
gene can be partitioned into two parts, lower and upper,
as shown in Figure 4 for three clusters. Both lower and
upper parts of the string consist of three bits each. The
ith bit in the lower/upper string tells whether the object
is in the lower/upper approximation of the ith cluster.
Figure 4 shows examples of all the valid genes for three
clusters. An object represented by g1 belongs to the
upper bounds of the first and second clusters. An object
represented by g6 belongs to the lower and upper bounds
of the second cluster. Any other value not given by g1 to
g7 is not valid. The objective of the genetic algorithms
(GAs) is to minimize the within-group-error. Lingras
provided a formulation of within-group-error for rough-
set based clustering. The resulting GAs were used to
evolve interval clustering of highway sections. Lingras
(2002) applied the unsupervised rough-set clustering
based on GAs for grouping Web users. However, the
clustering process based on GAs seemed
computationally expensive for scaling to larger datasets.
The K-means algorithm is one of the most popular
statistical techniques for conventional clustering
(Hartigan & Wong, 1979). Lingras and West (2004)
provided a theoretical and experimental analysis of a
modified K-means clustering based on the properties of
rough sets. It was used to create interval set representa-
TEAM LinG
Interval Set Representations of Clusters
Figure 3. Rough set based clustering
Upper bound of Group C
12
10
Lower bound of Group A
1
3 8
4
2 5
Upper bound of Group A
Boundary area of Groups A and B
Figure 4. Rough set gene
tions of clusters of Web users as well as a large set of
supermarket customers. The Kohonen (1988) neural
network or self-organizing map is another popular clus-
tering technique. It is desirable in some applications due
to its adaptive capabilities. Lingras et al. (2004) intro-
duced interval set clustering, using a modification of the
Kohonen self-organizing maps, based on rough-set
theory. Both the modified statistical- and neural-net-
work-based approaches used the following principle for
assigning objects to lower and upper bounds of clusters.
For each object, v, let d ( v, x i ) be the distance between
v and the centroid of cluster i. The ratio d ( v, x i ) / d ( v, x j ) ,
1 j k , is used to determine the membership of v.
I
11
Boundary area of Groups B and C
9
Lower bound of Group B
7
6
Upper bound of Group B
d ( v, x i ) = min d ( v, x j )
Let 1 j k
and
T = { j : d ( v, x i ) / d ( v, x j ) threshold and i j}.
1. If T , v exists in the upper bounds of clusters i
and all the clusters j T .
2. Otherwise, v exists in the lower and upper bounds
of cluster i. Moreover, v does not belong to any
other lower or upper bounds.
It was shown (Lingras & West, 2004) that the above
method of assignment preserves important rough-set
properties (Pawlak, 1992; Skowron, Stepaniuk, & Peters,
2003; Szmigielski & Polkowski, 2003; Yao, 2001).
Fuzzy Set Clustering
A fuzzy generalization of clustering uses a fuzzy member-
ship function to describe the degree of membership
(between [0, 1]) of an object in a given cluster. The sum
of fuzzy memberships of an object to all clusters must be
equal to 1.
Fuzzy C-means (FCMs) is a landmark algorithm in
the area of fuzzy C-partition clustering. It was first
proposed by Bezdek (1981). Rhee and Hwang (2001)
proposed a type-2 fuzzy C-means algorithm to solve
membership typicality. They point out that because the
memberships generated are relative numbers, they may
not be suitable for applications in which the member-
ships are supposed to represent typicality. Moreover,
the conventional fuzzy C-means process suffers from
noisy data, that is, when a noise point is located far from
all the clusters, an undesirable clustering result may
661
TEAM LinG

occur. The algorithm of Rhee and Hwang is based on the
fact that when updating the cluster centers, higher mem-
bership values should contribute more than memberships
with smaller values.
Rough-set theory and fuzzy-set theory complement
each other (Shi, Shen, & Liu, 2003). It is possible to create
interval clusters based on the fuzzy memberships ob-
tained by using the fuzzy C-means algorithm described in
the previous section. Let 1 0 . An object v aries of interval clusters, it may be possible to get early
belongs in the lower bound of cluster i if its membership
in the cluster is more than . Similarly, if its membership
in cluster i is greater than , the object belongs in the
upper bound of cluster i. Because 1 0 , if an REFERENCES
object belongs to the lower bound of a cluster, it will also
belong to its upper bound. Lingras and Yan (2004) de-
scribe further conditions on and that ensure satis-
faction of other important properties of rough sets.
FUTURE TRENDS
Temporal data mining is an application of data-mining
techniques to the data that takes the time dimension into
account. Temporal data mining is assuming increasing
importance. Much of the temporal data-mining tasks are
related to the use and analysis of temporal sequences of
raw data. There is little work that analyzes the results of
data mining over a period of time. Changes in cluster
characteristics of objects, such as supermarket customers
or Web users, over a period of time can be useful in data
mining. Such an analysis can be useful for formulating
marketing strategies. Marketing managers may want to
focus on specific groups of customers. Therefore, they
may need to understand the migrations of the customers
from one group to another group. The marketing strate-
gies may depend on desirability of these cluster migra-
tions. The overlapping clusters created by using interval
or fuzzy set clustering can be especially useful in such
studies. Overlapping clusters make it possible for an
object to transition from the core of a cluster to the core
of another cluster through the overlapping region. When
an object moves from the core of a cluster to an overlap-
ping region, it may be possible to provide an early warning
that can trigger an appropriate marketing campaign.
CONCLUSION
Clusters in data mining tend to have fuzzy and rough
boundaries. An object may potentially belong to more
than one cluster. Interval or fuzzy set representation
enables modeling of such overlapping clusters. This
662
Interval Set Representations of Clusters
article briefly describes several approaches to creating
interval and fuzzy set representations of clusters. The
interval set clustering is based on the theory of rough
sets. Changes in clusters can provide important clues
about the changing nature of the usage of a facility as
well as the changing nature of its users. Use of fuzzy and
interval set clustering also adds an interesting dimen-
sion to cluster migration studies. Due to rough bound-
warnings of potential significant changes in clustering
patterns.
Bezdek, J. C. (1981). Pattern recognition with fuzzy
objective function. New York: Plenum Press.
Do Prado, H. A., Engel, P. M., & Filho, H. C. (2002).
Rough clustering: An alternative to finding meaningful
clusters by using the reducts from a dataset. In J. Alpigini,
J. F. Peters, A. Skowron, & N. Zhong, (Eds.), Proceed-
ings of the Symposium on Rough Sets and Current
Trends in Computing: Vol. 2475. Springer-Verlag.
Hartigan, J. A., & Wong, M. A. (1979). Algorithm AS136:
A k-means clustering algorithm. Applied Statistics, 28,
100-108.
Hirano, S., & Tsumoto, S. (2003). Dealing with rela-
tively proximity by rough clustering. Proceedings of
the 22nd International Conference of the North Ameri-
can Fuzzy Information Processing Society (pp. 260-265).
Kohonen, T. (1988). Self-organization and associa-
tive memory. Berlin: Springer-Verlag.
Krishnapuram, R., Joshi, A., Nasraoui, O., & Yi, L.
(2001). Low-complexity fuzzy relational clustering al-
gorithms for Web mining. IEEE Transactions on Fuzzy
Systems, 9(4), 595-607.
Lingras, P. (2001). Unsupervised rough set classifica-
tion using GAs. Journal of Intelligent Information
Systems, 16(3), 215-228.
Lingras, P. (2002). Rough set clustering for Web min-
ing. Proceedings of the IEEE International Confer-
ence on Fuzzy Systems.
Lingras, P., Hogo, M., & Snorek, M. (2004). Interval set
clustering of Web users using modified Kohonen self-
organizing maps based on the properties of rough sets.
Web Intelligence and Agent Systems: An International
Journal.
TEAM LinG
Interval Set Representations of Clusters
Lingras, P., & West, C. (2004). Interval set clustering of
Web users with rough k-means. Journal of Intelligent
Information Systems, 23(1), 5-16.
Lingras, P., & Yan, R. (2004). Interval clustering using
fuzzy and rough set theory. Proceedings of the 23rd
International Conference of the North American Fuzzy
Information Processing Society (pp. 780-784), Canada.
Pawlak, Z. (1992). Rough sets: Theoretical aspects of
reasoning about data. New York: Kluwer Academic.
Peters, J. F., Skowron, A., Suraj, Z., Rzasa, W., &
Borkowski, M. (2002). Clustering: A rough set ap-
proach to constructing information granules, soft com-
puting and distributed processing. Proceedings of the
Sixth International Conference (pp. 57-61).
Rhee, F. C. H., & Hwang, C. (2001). A type-2 fuzzy c-
means clustering algorithm. Proceedings of the IFSA
World Congress and the 20th International Confer-
ence of the North American Fuzzy Information Pro-
cessing Society (Vol. 4, pp. 1926-1929).
Shi, H., Shen, Y., & Liu, Z. (2003). Hyperspectral bands
reduction based on rough sets and fuzzy C-means clus-
tering. Proceedings of the 20th IEEE Instrumentation
and Measurement Technology Conference (Vol. 2, pp.
1053-1056).
Skowron, A., Stepaniuk, J., & Peters, J. F. (2003).
Rough sets and infomorphisms: Towards approximation
of relations in distributed environments. Fundamental
Informatica, 54(2-3), 263-277.
Szmigielski, A., & Polkowski, L. (2003). Computing
from words via rough mereology in mobile robot navi-
gation. Proceedings of the IEE/RSJ International Con-
ference on Intelligent Robots and Systems (Vol. 4, pp.
3498-3503).
Yao, Y. Y. (2001). Information granulation and rough set
approximation. International Journal of Intelligent Sys-
tems, 16(1), 87-104.
KEY TERMS
I
Clustering: A form of unsupervised learning that
divides a data set so that records with similar content are
in the same group and groups are as different from each
other as possible.
Evolutionary Computation: A solution approach
guided by biological evolution that begins with potential
solution models, then iteratively applies algorithms to
find the fittest models from the set to serve as inputs to
the next iteration, ultimately leading to a model that best
represents the data.
Fuzzy C-Means Algorithms: Clustering algorithms
that assign a fuzzy membership in various clusters to an
object instead of assigning the object precisely to a
cluster.
Fuzzy Membership: Instead of specifying whether
an object precisely belongs to a set, fuzzy membership
specifies a degree of membership between [0,1].
Interval Set Clustering Algorithms: Clustering
algorithms that assign objects to lower and upper bounds
of a cluster, making it possible for an object to belong
to more than one cluster.
Interval Sets: If a set cannot be precisely defined,
one can describe it in terms of a lower bound and an
upper bound. The set will contain its lower bound and
will be contained in its upper bound.
Rough Sets: Rough sets are special types of interval
sets created by using equivalence relations.
Self-Organization: A system structure that often
appears without explicit pressure or involvement from
outside the system.
Supervised Learning: A learning process in which
the exemplar set consists of pairs of inputs and desired
outputs. The process learns to produce the desired
outputs from the given inputs.
Temporal Data Mining: An application of data-
mining techniques to the data that takes the time dimen-
sion into account.
Unsupervised Learning: Learning in the absence
of external information on outputs.
663
TEAM LinG
664
Kernel Methods in Chemoinformatics
Huma Lodhi
Imperial College London, UK
INTRODUCTION
Millions of people are suffering from fatal diseases
such as cancer, AIDS, and many other bacterial and viral
illnesses. The key issue is now how to design lifesaving
and cost-effective drugs so that the diseases can be
cured and prevented. It would also enable the provision
of medicines in developing countries, where approxi-
mately 80% of the world population lives. Drug design
is a discipline of extreme importance in
chemoinformatics. Structure-activity relationship (SAR)
and quantitative SAR (QSAR) are key drug discovery
tasks.
During recent years great interest has been shown in
kernel methods (KMs) that give state-of-the-art perfor-
mance. The support vector machine (SVM) (Vapnik,
1995; Cristianini and Shawe-Taylor, 2000) is a well-
known example. The building block of these methods is
an entity known as the kernel. The nondependence of
KMs on the dimensionality of the feature space and the
flexibility of using any kernel function make them an
optimal choice for different tasks, especially modeling
SAR relationships and predicting biological activity or
toxicity of compounds. KMs have been successfully
applied for classification and regression in pharmaceu-
tical data analysis and drug design.
BACKGROUND
Recently, I have seen a swift increase in the interest and
development of data mining and learning methods for
problems in chemoinformatics. There exist a number of
challenges for these techniques for chemometric prob-
lems. High dimensionality is one of them. There are
datasets with a large number of dimensions and a few
data points. Label and feature noise are other important
problems. Despite these challenges, learning methods
are a common choice for applications in the domain of
chemistry. This is due to automated building of predic-
tors that have strong theoretical foundations. Learning
techniques, including neural networks, decision trees,
inductive logic programming, and kernel methods such
as SVMs, kernel principal component analysis, and ker-
nel partial least square, have been applied to chemometric
problems with great success. Among these methods,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
KMs are new to such tasks. They have been applied for
applications in chemoinformatics since the late 1990s.
KMs and their utility for applications in
chemoinformatics are the focus of the research pre-
sented in this article. These methods possess special
characteristics that make them very attractive for tasks
such as the induction of SAR/QSAR. KMs such as SVMs
map the data into some higher dimensional feature
space and train a linear predictor in this higher dimen-
sional space. The kernel trick offers an effective way to
construct such a predictor by providing an efficient
method of computing the inner product between mapped
instances in the feature space. One does not need to
represent the instances explicitly in the feature space.
The kernel function computes the inner product by
implicitly mapping the instances to the feature space.
These methods can handle very high-dimensional noisy
data and can avoid overfitting. SVMs suffer from a
drawback that is the difficulty of interpretation of the
models for nonlinear kernel functions.
MAIN THRUST
I now present basic principles for the construction of
SVMs and also explore empirical findings.
Support Vector Machines and Kernels
The support vector machine was proposed in 1992
(Boser, Guyon, & Vapnik, 1992). A detailed analysis of
SVMs can be found in Vapnik (1995) and Cristianini and
Shawe-Taylor (2000). An SVM works by embedding the
input data, d i ,K , d n , into a Hilbert space through a non-
linear mapping, , and constructing the linear function
in this space. Mapping may not be known explicitly
but be accessed via the kernel function described in the
later section on kernels. The kernel function returns the
inner product between the mapped instances d i and d j in
a higher dimensional space that is for any mapping
: D F , k (d i , d j ) = (d i ), (d j ) . I now briefly de-
scribe SVMs for classification, regression, and kernel
functions.
TEAM LinG
Kernel Methods in Chemoinformatics
Support Vector Classification
The support vector machine for classification (SVC)
(Vapnik, 1995) is based on the idea of constructing the
maximal margin hyperplane in feature space. This unique
hyperplane separates the data into two categories with
maximum margin (hard margin). The maximal margin
hyperplane fails to generalize well when there is a high
level of noise in the data. In order to handle noise, data
margin errors are allowed, hence achieving a soft mar-
gin instead of a hard margin (no margin errors). The
generalization performance is improved by maintaining
the right balance between the margin maximization and
error. To find a hyperplane, one has to solve a convex
quadratic optimization problem. The support vector clas-
sifier is constructed by using only the inner products
between the mapped instances. The classification func-
tion for a new example d is given by
n polynomial kernel includes all products of at most p
f = sgn i ci k (d i , d ) + b , where i are Lagrange mul-
i =1
tiplies, ci { 1,+1} are categories, and b R .
Support Vector Regression
Support vector machines for regression (SVR) (Vapnik,
1995) inherit all the main properties that characterize
SVC. SVR embeds the input data into a Hilbert space
through a nonlinear mapping and constructs a linear
regression function in this space. In order to apply
support vector technique to regression, a reasonable
loss function is used. Vapniks insensitive loss func-
tion is a popular choice that is defined by
c f (d ) = max (0, c f (d ) ) , where c R . This loss
function allows errors below some > 0 and controls
the width of insensitive band. Regression estimation is
performed by solving an optimization problem. The
corresponding regression function f is given by
n
f = (~i i )k (d i , d ) + b , where ~i , i are Lagrange
i =1
multipliers, and b R .
Kernels
A kernel function calculates an inner product between
mapped instances in a feature space and allows implicit
feature extraction. The mathematical foundation of such
a function was established during the first decade of the
20th century (Mercer, 1909). A kernel function is a
symmetric function k (d i , d j ) = k (d j , d i ) for all d i , d j and
n
satisfies positive (semi)definiteness, a a k (d d ) 0
i , j =1
i j i j K
for ai , a j R . The n n matrix with entries of the form
K ij = k (d i , d j ) is known as the kernel matrix, or the
Gram matrix, that is a symmetric positive definite ma-
trix.
Linear, polynomial, and Gaussian Radial Basis Func-
tion (RBF) kernels are well-known examples of general
purpose kernel functions. Linear kernel function is
given by k linear (d i , d j ) = k (d i , d j ) = d id j . Given a kernel k ,
the polynomial construction is given by
k poly (d i , d j ) = (k (d i , d j ) + c ) . Here, p is a positive inte-
p
ger, and c is a nonnegative constant. Clearly, this incurs
a small computational cost to define a new feature
space. The feature space corresponding to a degree p
input features. Note that for p = 1 , I get the linear
construction. Furthermore, RBF kernel defines a fea-
ture space with an infinite number of dimensions. Given
a set of instances, the RBF kernel is given by
2
d d
K RBF (d i , d j ) = exp
i j
2 2 .
New kernels can be designed by keeping in mind that
kernel functions are closed under addition and multipli-
cation. Kernel functions can be defined over general
sets (Watkins, 2000; Haussler, 1999). This important
fact has allowed successful exploration of novel kernels
for discrete spaces such as strings, graphs, and trees
(Lodhi, Saunders, Shawe-Taylor, Cristianini, & Watkins,
2002; Kashima, Tsuda, & Inokuchi, 2003; Mahe, Ueda,
Akutsu, Perret, & Vert, 2004).
Applications
Given that I have presented basic concepts of SVMs, I
now describe the applications of SVMs in chemical
domains.
SAR/QSAR analysis plays a crucial role in the design
and development of drugs. It is based on the assumption
that the chemical structure and activity of compounds
are correlated. The aim of mining molecular databases
is to select a set of important compounds, hence form-
ing a small collection of useful molecules. The predic-
tion of a new compound with low error probability is an
important factor, as the false prediction can be costly
665
TEAM LinG

and result in a loss of information. Furthermore, accurate
prediction can speed up the drug design process.
In order to apply learning techniques such as KMs to
chemometric data, the compounds are transformed into a
form amenable to these techniques. Modeling SAR/QSAR
analysis can be viewed as comprising two stages. In the
first stage, descriptors (features) are extracted or com-
puted, and molecules are transformed into vectors. The
dimensionality of the vector space can be very high,
where, generally, molecular datasets comprise several
tens to hundred compounds. In the second stage, an
induction algorithm (SVC or SVR) is applied to learn an
SAR/QSAR model. The similarity between two com-
pounds is measured by the inner product between two
vectors. The similarity between compounds is inversely
proportional to the cosine of the angle between the
vectors. It is worth noting that kernel methods can be
applied not only for the induction of model but also for
the feature extraction through specialized kernels.
I now describe the application of kernel methods to
inducing structure activity relationship models. I first
focus on the classification task for chemometric data.
Trotter, Buxton, and Holden (2001) studied the efficacy
of an SVC to separating the compounds that will cross
the blood/brain barrier from those that will not cross the
barrier. SVC substantially improved other machine-learn-
ing techniques, including neural networks and decision
trees. In another study, a classification problem has been
formulated in order to predict the reduction of
dihydrofolate reductase by pyrimidines (Burbidge, Trot-
ter, Holden, & Buxton, 2001). An SVC in conjunction
with an RBF kernel is used to conduct experiments.
Experimental results show that SVC outperforms the
other classification techniques, including artificial neu-
ral networks, C5.0 decision tree, and nearest neighbor.
Structural feature extraction is an important prob-
lem in chemoinformatics. Structural feature extraction
refers to the problem of computing the features that
describe the chemical structure. In order to compute
structural features, Kramer, Frank, and Helma (2000)
viewed compounds as labeled graphs where vertices de-
pict atoms and edges describe bonds between them. The
method performs automated construction of structural
features of two-dimensional represented compounds.
Features are constructed by retrieving sequences of lin-
early connected atoms and bonds on the basis of some
statistical criterion. The authors argue that SVMs abil-
ity to handle high-dimensional data makes them attrac-
tive in this scenario, as the dimensionality of the feature
space may be very large. The authors applied an SVC to
predict mutageneicity and carcinogenicity of compounds
with promising results. In another study, an SVC in
conjunction with structural features is applied to model
mutageneicity structure activity relationships from
666
Kernel Methods in Chemoinformatics
noncongeneric datasets (Helma, Cramer, Kramer, & De
Raedt, in press). The performance of SVC is compared
with decision trees C4.5 and a rule learner. SVC and rule
learner showed excellent results.
In chemoinformatics, extracted features play a key
role in SAR/QSAR analysis. The feature extraction
module is constructed in such a way so as the loss of
information is at a minimum. This can make feature
extraction tasks as complex and expensive as solving
the entire problem. Kernel methods are an effective
alternative to explicit feature extraction. Kernels that
transform the compounds into feature vectors without
explicitly representing them can be constructed. This
can be achieved by viewing compounds as graphs
(Kashima, Tsuda, & Inokuchi, 2003; Mahe, Ueda, Akutsu,
Perret, & Vert, 2004). In this way, kernel methods can be
utilized to generate or extract feature for SAR/QSAR
analysis. Kashima et al. proposed a kernel function that
computes the similarity between compounds (labeled
graphs). The compounds are implicitly transformed into
feature vectors, where each entry of feature vector rep-
resents the number of label paths. Label paths are gen-
erated by random walks on graphs. In order to perform
binary classification of compounds, voted kernel
perceptron (Freund & Shapire, 1999) is employed with
promising results. Mahe, et al (2004) have improved the
graph kernels in terms of efficiency (computational time)
and classification accuracy. An SVC in conjunction with
graph kernels is used to predict mutageneicity of aro-
matic and hetroaromatics nitro compounds. SVC in con-
junction with graph kernels validates the efficacy of KMs
for feature extraction and induction of SAR models.
I now focus on the regression task for chemometric
data. In a chemical domain, datasets are generally char-
acterized as having high dimensionality and few data
points. Partial least square (PLS) (a regression method)
is very useful for such scenarios as compared to linear
least square regression. Demiriz, Bemmett, Breneman,
and Embrechts (2001) have applied an SVR for QSAR
analysis. The authors performed feature selection by
removing high-variance variables and induced a model
by using a linear programming SVR. The experimental
results show that SVR in conjunction with RBF kernel
outperforms PLS for chemometric datasets. PLSs
ability of handling high-dimensional data makes it an
optimal choice to combine it with KMs. There exist
two techniques to kernelize PLS (Bennett & Embrechts,
2003; Rosipal, Trejo, Matthews, & Wheeler, 2003).
One technique is based on mapping the data into a
higher dimensional space and constructing a linear
regression function in the space (Rosipal, Trejo,
Matthews, & Wheeler, 2003). Alternatively, PLS is
kernelized by obtaining a low-rank approximation of
kernel matrix and computing a regression function based
TEAM LinG
Kernel Methods in Chemoinformatics
on this approximation (Bennett & Embrechts). Kernel
partial least square is used to predict the binding affinities
of molecules to human serum albumin. Experimental re-
sults show that kernelized versions of PLS outperform
other methods, including PLS and SVR (Bennett &
Embrechts). Different principal component techniques,
including power method, singular value decomposition,
and eigen-value decomposition, have also been kernelized
to show the efficiency of KMs for datasets comprising few
points and a large number of variables (Wu, Massarat, &
de Jong, 1997). I conclude this section with a case study.
Gram-Schmidt Kernels for Prediction of
Activity of Compounds
As an example, I now present the results of a novel
approach for modeling structure-activity relationships
(Lodhi & Guo, 2002) based on the use of a special
kernel namely Gram-Schmidt kernel (GSK) (Cristianini,
Shawe-Taylor, & Lodhi, 2002). The kernel efficiently
performs Gram-Schmidt orthogonalisation in a kernel-
induced feature space. It is based on the idea of building
a more informative kernel matrix, as compared to Gram
matrices, which are constructed by standard kernels. An
SVC in conjunction with GSK is used to perform QSAR
analysis. The learning process of an SVC in conjunction
with GSK comprises two stages. In the first stage, highly
informative features are extracted in a kernel-induced
feature space. In the next stage, a soft margin classifier
is trained. In order to perform the analysis, the GSK
algorithm requires a set of compounds an underlying
kernel function and the number T , which specifies the
dimensionality of the feature space. For the underlying
kernel function, an RBF kernel is employed. GSK in
conjunction with SVC is applied on a benchmark dataset
(King, Muggleton, Lewis, & Sternberg, 1992) to predict
the inhibition of dihydrofolate reductase by pyrim-
idines. The dataset contains 55 compounds that are
divided into 5-fold cross-validation series. For each
drug there are three positions of possible substitution,
and the number of attributes for each substitution is
nine. The experimental results show that SVC in con-
junction with GSK achieves lower classification error
than the best reported results (Burbidge, Trotter, Holden,
& Buxton, 2001). Burbidge et al. performed SAR analysis
on this dataset by using a number of learning techniques
with SVC in conjunction with RBF kernels, achieving a
classification error of 0.1269. An SVC in conjunction with
GSK improved these results, achieving a classification
error of 0.1120 (Lodhi & Guo, 2002).
FUTURE TRENDS
K
Researchers using kernel methods have handled a prin-
cipal area in chemoinformatics; however, there exist
other important application areas, such as structure
elucidation, that require the utilization of such methods.
There is a need to develop new methods for challenging
tasks that are difficult to solve from existing methods.
Interest in and development of KMs for applications in
chemoinformatics are swiftly increasing, and I believe
they will lead to progress in both fields.
CONCLUSION
Kernel methods have strong theoretical foundations.
These methods combine the principles of statistical
learning theory and functional analysis. SVMs and other
kernel methods have been applied in different domains,
including text mining, face recognition, protein homol-
ogy detection, analyses and classification of gene ex-
pression data, and many others with great success. They
have shown impressive performance in
chemoinformatics. I believe that the growing popularity
of machine-learning techniques, especially kernel meth-
ods in chemoinformatics, will lead to significant devel-
opment in both disciplines.
REFERENCES
Bennett, K. P., & Embrechts, M. J. (2003). Advances in
learning theory: Methods, models and applications. Nato
Science Series III: Computer & Systems Science, 190,
227-250.
Boser, B. E., Guyon, I. M., & Vapnik, V. (1992). A
training algorithm for optimal margin classifier. Pro-
ceedings of the Fifth Annual ACM workshop on Com-
putational Learning Theory (pp. 144-152).
Burbidge, R., Trotter, M., Holden, S., & Buxton, B.
(2001). Drug design by machine learning: Support vec-
tor machines for pharmaceutical data. Computers and
Chemistry, 26(1), 4-15.
Cristianini, N., & Shawe-Taylor, J. (2000). An intro-
duction to support vector machines. Cambridge, MA:
Cambridge University Press.
Cristianini, N., Shawe-Taylor, J., & Lodhi, H. (2002).
Latent semantic kernels. Journal of Intelligent Infor-
mation Systems, 18(2/3), 127-152.
667
TEAM LinG

Demiriz, A., Bemmett, K. P., Breneman, C. M., & Embrechts,
M. J. (2001). Support vector machine regression in
chemometrics. Computing Science and Statistics.
Freund, Y., & Shapire, R. (1999). Large margin classifica-
tion using perceptron algorithm. Machine Learning, 37(3),
277-296.
Haussler, D. (1999). Convolution kernels on discrete
structures (Tech. Rep. No. UCSC-CRL-99-10). Santa
Cruz: University of California, Computer Science De-
partment.
Helma, C., Cramer, T., Kramer, S., & De Raedt, L. (in
press). Data mining and machine learning techniques for
identification of mutageneicity inducing substructures
and structure activity relationships of noncongeneric
compounds. Journal of Chemical Information and
Computer Systems.
Kashima, H., Tsuda, K., & Inokuchi, A. (2003).
Marginalized kernels between labeled graphs. Proceed-
ings of the 20th International Conference on Machine
Learning.
King, R. D., Muggleton, S., Lewis, R. A., & Sternberg,
M. J. E. (1992). Drug design by machine learning: The
use of inductive logic programming to model the struc-
ture activity relationships of timethoprim analogues
binding to dihydrofolate reeducates. Proceedings of
the National Academy of Sciences, USA, 89 (pp. 11322-
11326).
Kramer, S., Frank, E., & Helma, C. (2002). Fragment genera-
tion and support vector machines for inducing SARs. SAR
and QSAR in Environmental Research, 13(5), 509-523.
Lodhi, H., & Guo, Y. (2002). Gram-Schmidt kernels applied
to structure activity analysis for drug design. Proceed-
ings of the Second ACM SIGKDD Workshop on Data
Mining in Bioinformatics (pp. 37-42).
Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini,
N., & Watkins, C. (2002). Text classification using
string kernels. Journal of Machine Learning Research,
2, 419-444.
Mahe, P., Ueda, N., Akutsu, T., Perret, J.-L., & Vert, J.-
P. (2004). Extension of marginalized graph kernels.
Proceedings of the 21st International Conference on
Machine Learning.
Mercer, J. (1909). Functions of positive and negative
type and their connection with the theory of integral
equations. Philosophical Transactions of the Royal
Society London (A), 209, 415-446.
668
Kernel Methods in Chemoinformatics
Rosipal, R., Trejo, L., Matthews, B., & Wheeler, K. (2003).
Nonlinear kernel-based chemometric tools: A machine
learning approach. Proceedings of the Third Interna-
tional Symposium on PLS and Related Methods (pp. 249-
260).
Trotter, M., Buxton, B., & Holden, S. (2001). Support
vector machines in combinatorial chemistry. Measure-
ment and Control, 34(8), 235-239.
Vapnik, V. (1995). The nature of statistical learning
theory. Springer-Verlag.
Watkins, C. (2000). Dynamic alignment kernels. In P. J.
Bartlett, B. Schlkopf, D. Schuurmans, & A. J. Smola,
Advances in large-margin classifiers (pp. 39-50). Cam-
bridge, MA: MIT Press.
Wu, W., Massarat, D. L., & de Jong, S. (1997). The
kernel PCA algorithm for wide data. Part I: Theory and
algorithms. Chemometrics and Intelligent Laboratory
Systems, 36, 165-172.
KEY TERMS
Chemoinformatics: Storage, analysis, and drawing
inferences from chemical information (obtained from
chemical data) by using computational methods for drug
discovery.
Kernel Function: A function that computes the
inner product between mapped instances in a feature
space. It is a symmetric, positive definite function.
Kernel Matrix: A matrix that contains almost all
the information required by kernel methods. It is ob-
tained by computing the inner product between n in-
stances.
Machine Learning: A discipline that comprises the
study of how machines learn from experience.
Margin: A real-valued function. The sign and mag-
nitude of the margin give insight into the prediction of
an instance. Positive margin indicates correct predic-
tion, whereas negative margin shows incorrect predic-
tion.
Quantitative Structure-Activity Relationship
(QSAR): Illustrates quantitative relationships between
chemical structures and the biological and pharmaco-
logical activity of chemical compounds.
Support Vector Machines (SVMs): SVMs (im-
plicitly) map input examples into a higher dimensional
feature space via a kernel function and construct a linear
function in this space.
TEAM LinG

Knowledge Discovery with Artificial Neural
Networks
Juan R. Rabual Dopico
University of A Corua, Spain
Daniel Rivero Cebrin
University of A Corua, Spain
Julin Dorado de la Calle
University of A Corua, Spain
Nieves Pedreira Souto
University of A Corua, Spain
INTRODUCTION
The world of Data Mining (Cios, Pedrycz & Swiniarrski,
1998) is in constant expansion. New information is ob-
tained from databases thanks to a wide range of tech-
niques, which are all applicable to a determined set of
domains and count with a series of advantages and
inconveniences. The Artificial Neural Networks (ANNs)
technique (Haykin, 1999; McCulloch & Pitts, 1943; Orchad,
1993) allows us to resolve complex problems in many
disciplines (classification, clustering, regression, etc.),
and presents a series of advantages that convert it into a
very powerful technique that is easily adapted to any
environment. The main inconvenience of ANNs, how-
ever, is that they can not explain what they learn and what
reasoning was followed to obtain the outputs. This im-
plies that they can not be used in many environments in
which this reasoning is essential.
This article presents a hybrid technique that not only
benefits from the advantages of ANNs in the data-mining
field, but also counteracts their inconveniences by using
other knowledge extraction techniques. Firstly we extract
the requested information by applying an ANN, then we
apply other Data Mining techniques to the ANN in order
to explain the information that is contained inside the
network. We thus obtain a two-levelled system that offers
the advantages of the ANNs and compensates for its
shortcomings with other Data Mining techniques.
BACKGROUND
Ever since artificial intelligence appeared, ANNs have
been widely studied. Their generalisation capacity, and
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
669
K
their inductive learning convert them into a very robust
technique that can be used in almost any domain. An ANN
is an information processing technique that is inspired on
neuronal biology and consists of a large amount of inter-
connected computational units (neurons), usually in dif-
ferent layers. When an input vector is presented to the
input neurons, this vector is propagated and processed in
the network until it becomes an output vector in the
output neurons. Figure 1 shows an example of neural
network with 3 inputs, 2 outputs and three layers with 3,
4 and two neurons in each one.
The ANNs have proven to be a very powerful tool in
a lot of applications, but they present a big problem: their
reasoning process cannot be explained, i.e. there is no
clear relationship between the inputs that are presented
to the network and the outputs it produces. This means
that ANNs cannot be used in certain domains, even
though several approaches and attempts to explain their
behaviour have tried to solve this problem.
The reasoning of ANNs, and the rules extraction that
explains their functioning, were explained in various ways.
One of the first attempts established an equivalence
between ANNs and fuzzy rules (Bentez, Castro, &
Requena, 1997; Buckley, Hayashi, & Czogala, 1993; Jang
& Sun, 1992), obtaining only theoretical solutions. Other
works were based on the individual analysis of each
Figure 1. Example of Artificial Neural Network
TEAM LinG

neuron in the network, and its connection with the con-
secutive neurons. Towell and Shavlik (1994) in particular
see the connections between neurons as rules, and
Andrews and Geva (1994) uses networks with functions
that allow a clear identification of the dominant inputs.
Other approaches are the RULENEG (Rule-Extraction
from Neural Networks by Step-wise Negation) (Pop, Hay-
ward, & Diederich, 1994) and TREPAN (Craven, 1996)
algorithms. The first approach however modifies the train-
ing set and therefore loses the generalisation capacity of
the ANNs. The TREPAN approach is similar to decision
tree algorithms such as CART (Classification and Regres-
sion Trees) or C4.5, which turns the ANN into a MofN (M-
of-N) decision tree.
The DEDEC (Decision Detection) algorithm (Tickle,
Andrews, Golea, & Tickle, 1998) extracts rules by finding
minimal information sufficient to distinguish, from the
neural network point of view, between a given pattern and
all other patterns. The DEDEC algorithm uses the trained
ANN to create examples from which rules can be extracted.
Unlike other approaches, it also uses the weight vectors
of the network to obtain an additional analysis that
improves the extraction of rules. This information is then
used to direct the strategy for generating a (minimal) set
of examples for the learning phase. It also uses an efficient
algorithm for the rule extraction phase. Based on these
and other already mentioned techniques, Chalup, Hay-
ward and Diedrich (1998); Visser, Tickle, Hayward and
Andrews (1996); and Tickle, Andrews, Golea and Tickle
(1998) also presented their solutions.
The methods for the extraction of logical rules, devel-
oped by Duch, Adamczak and Grabczewski (2001), are
based on multilayer perceptron networks with the MLP2LN
(Multi-Layer Perceptron converted to Logical Network)
method and its constructive version C-MLP2LN. MLP2LN
consists in taking a multilayer and already trained
perceptron and simplify it in order to obtain a network with
weights 0, +1 or -1. C-MLP2LN acts in a similar way. After
this process, the dominant rules are easily extracted, and
the weights of the input layer allow us to deduct which
parameters are relevant.
More recently, genetic algorithms (GAs) have been
used to discover rules in ANNs. Keedwell, Narayanan and
Savic (2000) use a GA in which the chromosomes are rules
based on value intervals or ranges applied to the inputs
of the ANN. The values are obtained from the training
patterns.
The most recent works in rules extraction from ANNs
are presented by Rivero, Rabual, Dorado, Pazos and
Pedreira (2004) and Rabual, Dorado, Pazos and Rivero
(2003). They extract rules by applying a symbolic regres-
sion system, based on Genetic Programming (GP)
(Engelbrecht, Rouwhorst & Schoeman, 2001; Koza, Keane,
670
Knowledge Discovery with Artificial Neural Networks
Streeter, Mydlowec, Yu, & Lanza, 2003; Wong & Leung,
2000), to a set of inputs / outputs produced by the ANN.
The set of network inputs / produced outputs is dynami-
cally modified, as explained on this paper.
ARCHITECTURE
This article presents an architecture in two levels for the
extraction of knowledge from databases. In a first level, we
apply an ANN as Data Mining technique; in the second
level, we apply a knowledge extraction technique to this
network.
Data Mining with ANNs
Artificial Neural Networks constitute a Data Mining
technique that has been widely used as a technique for
the extraction of knowledge from databases. Their train-
ing process is based on examples, and presents several
advantages that other models do not offer:
A high generalisation level. Once ANNs are trained
with a training set, they produce outputs (close to
desired or supposed outputs) for inputs that were
never presented to them before.
A high error tolerance. Since ANNs are based on the
successive and parallel interconnection between
many processing elements (neurons), the output of
the system is not significantly affected if one of
them fails.
A high noise tolerance.
All these advantages turn ANNs into the ideal tech-
nique for the extraction of knowledge in almost any
domain. They are trained with many different training
algorithms. The most famous one is the backpropagation
algorithm (Rumelhart, Hinton & Williams, 1986), but many
other training algorithms are applied according to the
topology of the network and the use that is given to it. In
the course of recent years, Evolutionary Computation
techniques such as Genetic Algorithms (Holland, 75)
(Goldberg, 89) (Rabual, Dorado, Pazos, Gestal, Rivero &
Pedreira, 2004a; Rabual, Dorado, Pazos, Pereira & Rivero,
2004b) are gaining ground, because they correct the
defects of other training algorithms, such as the tendency
to generate local minimums or to overtrain the network.
Even so, and in spite of these algorithms that train the
network automatically (and even search for the topology
of the network), ANNs present a series of defects that
make them useless in many application fields. As we
already said, their main defect is the fact that in general
they are not interpretable: once an input is applied to the
TEAM LinG
Knowledge Discovery with Artificial Neural Networks

network and an output obtained, we cannot follow the 5

steps of the reasoning that was used by the network to
produce this output. This is why ANNs are not applicable
in those domains in which we need to know why an output
is the result of a certain input. The best example of this type
of domain is the medical field: if we use an ANN to obtain
the diagnosis for a certain disease, we need to know why
that specific diagnosis or output value is produced.
The purpose of this article is to correct this defect
through the use of another knowledge extraction tech-
nique. This technique is applied to the ANN in order to
extract its internal knowledge and express it in terms that
are proper of the used technique (i.e. decision trees, se-
mantic networks, IF-THEN-ELSE rules, etc.).
Extracting Knowledge from ANNs
The second level of the architecture is based on the
application of a knowledge extraction technique to the
ANN that was used for Data Mining. This technique
depends on how we wish to express the internal knowledge
of the network: if we want decision trees, we could use the
CART algorithm, if we want rules, we could use Genetic
Programming, etc.
In general, the second knowledge extraction technique
is applied to the network to explain its behaviour when it
produces a set of outputs from a set of inputs, without
considering its internal functioning. This means that the
network is treated as a black box: we obtain information
on its behaviour by generating a set of inputs / obtained
outputs to which we apply the knowledge extraction tech-
nique. Obtaining this set is complicated, because it has to
be so representative that we can extract from it all the
knowledge that a network really contains, not just part of
the knowledge.
A first option could be to use the inputs of the training
set that was used to create the network, and the outputs
generated from this inputs by the network. This method
supposes that the training set is already sufficiently ample
and representative of the search space in which the net-
work will be applied. However, our aim is not to explain the
behaviour of the network only in those patterns in which
it was trained or in patterns that are close (i.e., the search
area in which the network was trained); we want to explain
the functioning of the network as a whole, including in
areas in which it was not trained.
(1 0)
0.1
+ 1 = 115 = 161051
K
patterns, an amount that is excessive for any Data Mining
tool.
The problem of choosing the input set that will be
applied to the network is solved with a hybrid solution:
the inputs of the training set are multiplied by duplicating
some elements of the set with certain modifications.
These inputs are presented to the network so that for
each of them an output vector is obtained, and as such
we shape a set of patterns that is applied to the selected
algorithm for the extraction of knowledge.
It is important to note that the duplicated inputs were
modified so that the knowledge extraction algorithm
already starts to explore the areas that lie beyond those
that were used to train the network. These first dupli-
cated inputs are nevertheless insufficient to explore
these areas, which is why the knowledge extraction
algorithm is controlled by a system for the generation of
new patterns.
This new patterns generation system takes those
patterns whose adjustment is sufficiently good in the
knowledge extraction algorithm, replaces them with new
patterns that are created from existing ones (either those
that will be replaced, or others from the training set), and
applies to them modifications that create patterns in
areas that were not explored before. When an area is
already explored by the algorithm and is already repre-
sented (by trees, rules, etc.), it is deleted and the system
continues exploring new areas.
The whole process is illustrated in Figure 2.
Figure 2. ANN knowledge extraction process
Pattern Generation System




Input  Output
set  set


Knowledge
Extraction Patterns

We discard the option of elaborating an exhaustive

inputs set by taking all the combinations of all the possible Discard
patterns with
values in each network input with small intervals, because Knowledge Extraction
Algorithm good adjustment
the combinatorial explosion would make this set too large.
For instance, for a network with 5 inputs that each takes
values between 0 and 1, and with intervals of 0.1, we would
ANN Knowledge
obtain

671

TEAM LinG

In this way, we create a closed system in which the
patterns set is constantly updated and the search continues
for new areas that are not yet covered by the knowledge that
we have of the network. A detailed description of the
method and of its application to concrete problems can be
found in Rabual, Dorado, Pazos and Rivero (2003);
Rabual, Dorado, Pazos, Gestal, Rivero and Pedreira (2004a);
and Rabual, Dorado, Pazos, Pereira & Rivero (2004b).
FUTURE TRENDS
The proposed architecture is based on the application of
a knowledge extraction technique to an ANN, whereas the
technique that will be used depends on the type of
knowledge that we wish to obtain. Since there is a great
variety of techniques and algorithms that generate infor-
mation of the same kind (IF-THEN rules, trees, etc.), we
need to study them and carry out experiments to test their
functioning in the proposed system, and in particular their
adequacy for the new patterns generation system.
Also, this system for the generation of new patterns
involves a large number of parameters, such as the per-
centage of patterns change, the number of new patterns,
or the maximum error with which we consider that a pattern
is represented by a rule. Since there are so many param-
eters, we need to study not only each parameter sepa-
rately but also the influence of all the parameters on the
final result.
CONCLUSION
This article proposes a system architecture that makes
good use of the advantages of the ANNs, such as Data
Mining, and avoids its inconveniences. On a first level, we
apply an ANN to extract and model a set of data. The
resulting model offers all the advantages of ANNs, such
as noise tolerance and generalisation capacity. On a
second level, we apply another knowledge extraction
technique to the ANN, and thus obtain the knowledge of
the ANN, which is the generalisation of the knowledge
that was used for its learning; this knowledge is expressed
in the shape that is decided by the user. It is obvious that
the union of various techniques in a hybrid system con-
veys a series of advantages that are associated to them.
REFERENCES
Andrews, R., & Geva, S. (1994). Rule extraction from a
constrained error backpropagation MLP. Proceedings of
672
Knowledge Discovery with Artificial Neural Networks
the Australian Conference on Neural Networks,
Brisbane, Queensland (pp. 9-12).
Bentez, J.M., Castro, J.L., & Requena, I. (1997). Are
artificial neural networks black boxes? IEEE Transactions
on Neural Networks, 8, 1156-1164.
Buckley, J.J., Hayashi, Y., & Czogala, E. (1993). On the
equivalence of neural nets and fuzzy expert systems.
Fuzzy Sets Systems, 53, 129-134.
Chalup, S., Hayward, R., & Diedrich, J. (1998). Rule extrac-
tion from artificial neural networks trained on elemen-
tary number classification task. Queensland University
of Technology, Neurocomputing Research Centre. QUT
NRC technical report.
Cios, K., Pedrycz, W., & Swiniarski, R. (1998). Data mining
methods for knowledge discovery. The 1st Edition, Kluver
International Series in Engineering and Computer Sci-
ence (pp. 495). Boston: Kluwer Academic Publishers.
Craven, M. W. (1996). Extracting comprehensible mod-
els from trained neural networks. PhD Thesis, University
of Wisconsin, Madison.
Duch, W., Adamczak, R., & Grabczewski, K. (2001). A new
methodology of extraction, optimisation and application
of crisp and fuzzy logical rules. IEEE Transactions on
Neural Networks, 12, 277-306.
Engelbrecht, A.P., Rouwhorst, S.E., & Schoeman L. (2001).
A building block approach to genetic programming for
rule discovery. In Abbass, R. Sarkar, & C. Newton (Eds.),
Data mining: A heuristic approach (pp. 175-189). Hershey:
Idea Group Publishing.
Goldberg, D. E. (1989). Genetic algorithms in search,
optimization and machine learning. Reading, MA:
Addison-Wesley.
Haykin, S. (1999). Neural networks (2nd ed.). Englewood
Cliffs, NJ: Prentice Hall.
Holland, J. H. (1975). Adaptation in natural and artificial
systems. University of Michigan Press.
Jang, J., & Sun, C. (1992). Functional equivalence between
radial basis function networks and fuzzy inference sys-
tems. IEEE Transactions on Neural Networks, 4, 156-158.
Keedwell E., Narayanan A., & Savic D. (2000). Creating
rules from trained neural networks using genetic algo-
rithms. Proceedings of the International Journal of
Computers, Systems and Signals (IJCSS) (Vol. 1, pp. 30-
42).
Koza, J. R., Keane, M. A., Streeter, M. J., Mydlowec, W.,
Yu, J., & Lanza, G. (Eds.). (2003). Genetic programming
TEAM LinG
Knowledge Discovery with Artificial Neural Networks
IV: Routine human-competitive machine intelligence.
Dordrecht, The Netherlands: Kluwer Academic Publishers.
McCulloch, W.S., & Pitts, W. (1943). A logical calculus of
ideas immanent in nervous activity. Bulletin of Math-
ematical Biophysics, (5), 115-133.
Orchard, G. (Ed.) (1993). Neural computing. Research and
Applications. Ed. London: Institute of Physics Publish-
ing, Londres.
Pop, E., Hayward, R., & Diederich, J. (1994). RULENEG:
Extracting rules from a trained ANN by stepwise nega-
tion. Queensland University of Technology,
Neurocomputing Research Centre. QUT NRC technical
report.
Rabual, J.R., Dorado, J., Pazos, A., & Rivero, D. (2003).
Rules and generalization capacity extraction from ANN
with GP. Lecture notes in Computer Science, 606-613.
Rabual, J.R., Dorado, J., Pazos, A., Gestal, M., Rivero, D.,
& Pedreira, N. (2004a). Search the optimal RANN architec-
ture, reduce the training set and make the training process
by a distribute genetic algorithm. Artificial Intelligence
and Applications, 1, 415-420.
Rabual, J.R., Dorado, J., Pazos, A., Pereira, J., & Rivero,
D. (2004 b). A new approach to the extraction of ANN rules
and to their generalization capacity through GP. Neural
computation, 7(16), 1483-1523.
Rivero, D., Rabual, J.R., Dorado, J., Pazos, A., & Pedreira,
N. (2004). Extracting knowledge from databases and ANNs
with genetic programming: Iris flower classification prob-
lem. Intelligent agents for data mining and information
retrieval (pp. 136-152).
Rumelhart, D., Hinton, G., & Williams, R. (1986). Learning
representations by back-propagating errors. Nature, 323,
533-536.
Tickle, A.B., Andrews, R., Golea, M., & Diederich, J.
(1998). The truth will come to light: Directions and chal-
lenges in extracting the knowledge embedded within
trained artificial neural networks. IEEE Transaction on
Neural Networks. (9), 1057-1068.
Towell G., & Shavlik J.W. (1994). Knowledge-based arti-
ficial neural networks. Artificial Intelligence, 70, 119-165.
Visser, U., Tickle, A., Hayward, R., & Andrews, R. (1996).
Rule-extraction from trained neural networks: Different
techniques for the determination of herbicides for the
plant protection advisory system PRO_PLANT. Proceed-
ings of the Rule Extraction from Trained Artificial Neu-
ral Networks Workshop (pp. 133-139), Brighton, UK.
Wong, M.L., & Leung, K.S. (2000). Data mining using
grammar based genetic programming and applications. K
Series in Genetic Programming, 3, 232. Boston: Kluwer
Academic Publishers.
KEY TERMS
Area of the Search Space: Set of specific ranges or
values of the input variables that constitute a subset of
the search space.
Artificial Neural Networks: A network of many simple
processors (units or neurons) that imitates a biologi-
cal neural network. The units are connected by unidirec-
tional communication channels, which carry numeric data.
Neural networks can be trained to find nonlinear relation-
ships in data, and are used in applications such as robot-
ics, speech recognition, signal processing or medical
diagnosis.
Backpropagation Algorithm: Learning algorithm of
ANNs, based on minimising the error obtained from the
comparison between the outputs that the network gives
after the application of a set of network inputs and the
outputs it should give (the desired outputs).
Data Mining: The application of analytical methods
and tools to data for the purpose of identifying patterns,
relationships or obtaining systems that perform useful
tasks such as classification, prediction, estimation, or
affinity grouping.
Evolutionary Computation: Solution approach guided
by biological evolution, which begins with potential so-
lution models, then iteratively applies algorithms to find
the fittest models from the set to serve as inputs to the next
iteration, ultimately leading to a model that best repre-
sents the data.
Knowledge Extraction: Explicitation of the internal
knowledge of a system or set of data in a way that is easily
interpretable by the user.
Rule Induction: Process of learning, from cases or
instances, if-then rule relationships that consist of an
antecedent (if-part, defining the preconditions or cover-
age of the rule) and a consequent (then-part, stating a
classification, prediction, or other expression of a prop-
erty that holds for cases defined in the antecedent).
Search Space: Set of all possible situations of the
problem that we want to solve could ever be in.
673
TEAM LinG
674
Learning Bayesian Networks
Marco F. Ramoni
Harvard Medical School, USA
Paola Sebastiani
Boston University School of Public Health, USA
INTRODUCTION
Born at the intersection of artificial intelligence, statistics,
and probability, Bayesian networks (Pearl, 1988) are a
representation formalism at the cutting edge of knowl-
edge discovery and data mining (Heckerman, 1997). Baye-
sian networks belong to a more general class of models
called probabilistic graphical models (Whittaker, 1990;
Lauritzen, 1996) that arise from the combination of graph
theory and probability theory, and their success rests on
their ability to handle complex probabilistic models by
decomposing them into smaller, amenable components. A
probabilistic graphical model is defined by a graph, where
nodes represent stochastic variables and arcs represent
dependencies among such variables. These arcs are an-
notated by probability distribution shaping the interac-
tion between the linked variables. A probabilistic graphi-
cal model is called a Bayesian network, when the graph
connecting its variables is a directed acyclic graph (DAG).
This graph represents conditional independence assump-
tions that are used to factorize the joint probability distri-
bution of the network variables, thus making the process
of learning from a large database amenable to computa-
tions. A Bayesian network induced from data can be used
to investigate distant relationships between variables, as
well as making prediction and explanation, by computing
the conditional probability distribution of one variable,
given the values of some others.
BACKGROUND
The origins of Bayesian networks can be traced back as
far as the early decades of the 20th century, when Sewell
Wright developed path analysis to aid the study of
genetic inheritance (Wright, 1923, 1934). In their current
form, Bayesian networks were introduced in the early
1980s as a knowledge representation formalism to encode
and use the information acquired from human experts in
automated reasoning systems in order to perform diag-
nostic, predictive, and explanatory tasks (Charniak, 1991;
Pearl, 1986, 1988). Their intuitive graphical nature and
their principled probabilistic foundations were very at-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
tractive features to acquire and represent information
burdened by uncertainty. The development of amenable
algorithms to propagate probabilistic information through
the graph (Lauritzen, 1988; Pearl, 1988) put Bayesian
networks at the forefront of artificial intelligence research.
Around the same time, the machine-learning community
came to the realization that the sound probabilistic nature
of Bayesian networks provided straightforward ways to
learn them from data. As Bayesian networks encode
assumptions of conditional independence, the first ma-
chine-learning approaches to Bayesian networks con-
sisted of searching for conditional independence struc-
tures in the data and encoding them as a Bayesian network
(Glymour, 1987; Pearl, 1988). Shortly thereafter, Cooper
and Herskovitz (1992) introduced a Bayesian method that
was further refined by Heckerman, et al. (1995) to learn
Bayesian networks from data.
These results spurred the interest of the data-mining
and knowledge-discovery community in the unique fea-
tures of Bayesian networks (Heckerman, 1997); that is, a
highly symbolic formalism, originally developed to be
used and understood by humans, well-grounded on the
sound foundations of statistics and probability theory,
able to capture complex interaction mechanisms and to
perform prediction and classification.
MAIN THRUST
A Bayesian network is a graph, where nodes represent
stochastic variables and (arrowhead) arcs represent de-
pendencies among these variables. In the simplest case,
variables are discrete, and each variable can take a finite
set of values.
Representation
Suppose we want to represent the variable gender. The
variable gender may take two possible values: male and
female. The assignment of a value to a variable is called the
state of the variable. So, the variable gender has two
states: Gender = Male and Gender = Female. The graphical
structure of a Bayesian network looks like this:
TEAM LinG
Learning Bayesian Networks
Figure 1.
The network represents the notion that obesity and
gender affect the heart condition of a patient. The variable
obesity can take three values: yes, borderline and no. The
variable heart condition has two states: true and false. In
this representation, the node heart condition is said to be
a child of the nodes gender and obesity, which, in turn,
are the parents of heart condition.
The variables used in a Bayesian networks are sto-
chastic, meaning that the assignment of a value to a
variable is represented by a probability distribution. For
instance, if we do not know for sure the gender of a patient,
we may want to encode the information so that we have
better chances of having a female patient rather than a
male one. This guess, for instance, could be based on
statistical considerations of a particular population, but
this may not be our unique source of information. So, for
the sake of this example, lets say that there is an 80%
chance of being female and a 20% chance of being male.
Similarly, we can encode that the incidence of obesity is
10%, and 20% are borderline cases. The following set of
distributions tries to encode the fact that obesity in-
creases the cardiac risk of a patient, but this effect is more
significant in men than women:
The dependency is modeled by a set of probability
distributions, one for each combination of states of the
variables gender and obesity, called the parent variables
of heart condition.
Figure 2.
Learning
L
Learning a Bayesian network from data consists of the
induction of its two different components: (1) the graphi-
cal structure of conditional dependencies (model selec-
tion) and (2) the conditional distributions quantifying the
dependency structure (parameter estimation).
There are two main approaches to learning Bayesian
networks from data. The first approach, known as con-
straint-based approach, is based on conditional indepen-
dence tests. As the network encodes assumptions of
conditional independence, along this approach we need
to identify conditional independence constraints in the
data by testing and then encoding them into a Bayesian
network (Glymour, 1987; Pearl, 1988; Whittaker, 1990).
The second approach is Bayesian (Cooper &
Herskovitz, 1992; Heckerman et al., 1995) and regards
model selection as an hypothesis testing problem. In this
approach, we suppose to have a set M= {M0,M1, ...,Mg} of
Bayesian networks for the random variables Y1, ..., Yv,, and
each Bayesian network represents an hypothesis on the
dependency structure relating these variables. Then, we
choose one Bayesian network after observing a sample of
data D = {y 1k, ..., yvk}, for k = 1, . . . , n. If p(M h) is the prior
probability of model M h, a Bayesian solution to the model
selection problem consists of choosing the network with
maximum posterior probability:
p(Mh|D) p(Mh)p(D|Mh).
The quantity p(Mh|D) is the marginal likelihood, and its
computation requires the specification of a parameteriza-
tion of each model Mh and the elicitation of a prior distri-
bution for model parameters. When all variables are dis-
crete or all variables are continuous, follow Gaussian
distributions, and the dependencies are linear and the
marginal likelihood factorizes into the product of marginal
likelihoods of each node and its parents. An important
property of this likelihood modularity is that in the com-
parison of models that differ only for the parent structure
of a variable Yi, only the local marginal likelihood matters.
Thus, the comparison of two local network structures that
specify different parents for Yi can be done simply by
evaluating the product of the local Bayes factor BFh,k =
p(D|Mhi) / p(D|Mki), and the ratio p(Mhi)/ p(Mki), to compute
the posterior odds of one model vs. the other as p(M hi|D)
/ p(Mki|D).
In this way, we can learn a model locally by maximizing
the marginal likelihood node by node. Still, the space of the
possible sets of parents for each variable grows exponen-
tially with the number of parents involved, but successful
heuristic search procedures (both deterministic and sto-
chastic) exist to render the task more amenable (Cooper &
Herskovitz, 1992; Singh & Larranaga, 1996; Valtorta, 1995).
675
TEAM LinG

Once the structure has been learned from a dataset, we
still need to estimate the conditional probability distribu-
tions associated to each dependency in order to turn the
graphical model into a Bayesian network. This process,
called parameter estimation, takes a graphical structure
and estimates the conditional probability distributions of
each parent-child combination. When all the parent vari-
ables are discrete, we need to compute the conditional
probability distribution of the child variable, given each
combination of states of its parent variables. These con-
ditional distributions can be estimated either as relative
frequencies of cases or, in a Bayesian fashion, by using
these relative frequencies to update some, possibly uni-
form, prior distribution. A more detailed description of
these estimation procedures for both discrete and continu-
ous cases is available in Ramoni and Sebastiani (2003).
Prediction and Classification
Once a Bayesian network has been defined, either by hand
or by an automated discovery process from data, it can be
used to reason about new problems for prediction, diagno-
sis, and classification. Bayes theorem is at the heart of the
propagation process.
One of the most useful properties of a Bayesian net-
work is the ability to propagate evidence irrespective of
the position of a node in the network, contrary to standard
classification methods. In a typical classification system,
for instance, the variable to predict (i.e., the class) must be
chosen in advance before learning the classifier. Informa-
tion about single individuals then will be entered, and the
classifier will predict the class (and only the class) of these
individuals. In a Bayesian network, on the other hand, the
information about a single individual will be propagated in
any direction in the network so that the variable(s) to
predict must not be chosen in advance.
Although the problem of propagating probabilistic in-
formation in Bayesian networks is known to be, in the
general case, NP-complete (Cooper, 1990), several scalable
algorithms exist to perform this task in networks with hun-
dreds of nodes (Castillo, et al., 1996; Cowell et al., 1999; Pearl,
1988). Some of these propagation algorithms have been
extended, with some restriction or approximations, to net-
works containing continuous variables (Cowell et al., 1999).
FUTURE TRENDS
The technical challenges of current research in Bayesian
networks are focused mostly on overcoming their current
limitations. Established methods to learn Bayesian net-
works from data work under the assumption that each
variable is either discrete or normally distributed around a
mean that linearly depends on its parent variables. The
676
Learning Bayesian Networks
latter networks are termed linear Gaussian networks,
which still enjoy the decomposability properties of the
marginal likelihood. Imposing the assumption that con-
tinuous variables follow linear Gaussian distributions
and that discrete variables only can be parent nodes in
the network but cannot be children of any continuous
node, leads to a closed-form solution for the computation
of the marginal likelihood (Lauritzen, 1992). The second
technical challenge is the identification of sound meth-
ods to handle incomplete information, either in the form
of missing data (Sebastiani & Ramoni, 2001) or com-
pletely unobserved variables (Binder et al., 1997). A third
important area of development is the extension of Baye-
sian networks to represent dynamic processes
(Ghahramani, 1998) and to decode control mechanisms.
The most fundamental challenge of Bayesian net-
works today, however, is the full deployment of their
potential in groundbreaking applications and their es-
tablishment as a routine analytical technique in science
and engineering. Bayesian networks are becoming in-
creasingly popular in various fields of genomic and
computational biologyfrom gene expression analysis
(Friedman, 2004) to proteimics (Jansen et al., 2003) and
genetic analysis (Lauritzen & Sheehan, 2004)but they
are still far from being a received approach in these areas.
Still, these areas of application hold the promise of
turning Bayesian networks into a common tool of statis-
tical data analysis.
CONCLUSION
Bayesian networks are a representation formalism born
at the intersection of statistics and artificial intelligence.
Thanks to their solid statistical foundations, they have
been turned successfully into a powerful data-mining
and knowledge-discovery tool that is able to uncover
complex models of interactions from large databases.
Their high symbolic nature makes them easily under-
standable to human operators. Contrary to standard
classification methods, Bayesian networks do not re-
quire the preliminary identification of an outcome vari-
able of interest, but they are able to draw probabilistic
inferences on any variable in the database. Notwith-
standing these attractive properties and the continuous
interest of the data-mining and knowledge-discovery
community, Bayesian networks still are not playing a
routine role in the practice of science and engineering.
REFERENCES
Binder, J. et al. (1997). Adaptive probabilistic networks
with hidden variables. Mach Learn, 29(2-3), 213-244.
TEAM LinG
Learning Bayesian Networks
Castillo, E. et al. (1996). Expert systems and probabiistic
network models. New York: Springer.
Charniak, E. (1991). Bayesian networks without tears. AI
Magazine, 12(8), 50-63.
Cooper, G.F. (1990). The computational complexity of
probabilistic inference using Bayesian belief networks.
Artif Intell, 42(2-3), 393-405.
Cooper, G.F., & Herskovitz, G.F. (1992). A Bayesian method
for the induction of probabilistic networks from data.
Mach Learn, 9, 309-347.
Cowell, R.G., et al. (1999). Probabilistic networks and
expert systems. New York: Springer.
Friedman, N. (2004). Inferring cellular networks using
probabilistic graphical models. Science, 303, 799-805.
Ghahramani, Z. (1998). Learning dynamic Bayesian net-
works. In C.L. Giles, & M. Gori (Eds.), Adaptive process-
ing of sequences and data structures (pp. 168-197). New
York: Springer.
Glymour, C., Scheines, R., Spirtes, P., & Kelly, K. (1987).
Discovering causal structure: Artificial intelligence,
philosophy of science, and statistical modeling. San
Diego, CA: Academic Press.
Heckerman, D. (1997). Bayesian networks for data mining.
Data Mining and Knowledge Discovery, 1(1), 79-119.
Heckerman, D. et al. (1995). Learning Bayesian networks:
The combinations of knowledge and statistical data. Mach
Learn, 20, 197-243.
Jansen, R. et al. (2003). A Bayesian networks approach for
predicting protein-protein interactions from genomic data.
Science, 302, 449-453.
Larranaga, P., Kuijpers, C., Murga, R., & Yurramendi, Y.
(1996). Learning Bayesian network structures by search-
ing for the best ordering with genetic algorithms. IEEE T
Syst Man Cyb, 26, 487-493.
Lauritzen, S.L. (1992). Propagation of probabilities, means
and variances in mixed graphical association models. J
Amer Statist Assoc, 87, 1098-108.
Lauritzen, S.L. (1996). Graphical models. Oxford:
Clarendon Press.
Lauritzen, S.L. (1988). Local computations with probabilities
on graphical structures and their application to expert sys-
tems (with discussion). J Roy Stat Soc B Met, 50, 157-224.
Lauritzen, S.L., & Sheehan, N.A. (2004). Graphical models
for genetic analysis. Statist Sci, 18(4), 489-514.
Pearl, J. (1986). Fusion, propagation, and structuring in
belief networks. Artif. Intell., 29(3), 241-288. L
Pearl, J. (1988). Probabilistic reasoning in intelligent
systems: Networks of plausible inference. San Francisco:
Morgan Kaufmann.
Ramoni, M., & Sebastiani, P. (2003). Bayesian methods. In
M.B. Hand (Ed.), Intelligent data analysis: An introduc-
tion (pp. 128-166). New York: Springer.
Sebastiani, P., & Ramoni, M. (2001). Bayesian selection of
decomposable models with incomplete data. J Am Stat
Assoc, 96(456), 1375-1386.
Singh, M., & Valtorta, M. (1995). Construction of Baye-
sian network structures from data: A brief survey and an
efficient algorithm. Int J Approx Reason, 12, 111-131.
Whittaker, J. (1990). Graphical models in applied multi-
variate statistics. New York: John Wiley & Sons.
Wright, S. (1923). The theory of path coefficients: A reply
to Niles criticisms. Genetics, 8, 239-255.
Wright, S. (1934). The method of path coefficients. Ann
Math Statist, 5, 161-215.
KEY TERMS
Bayes Factor: Ratio between the probability of the
observed data under one hypothesis divided by its prob-
ability under an alternative hypothesis.
Conditional Independence: Let X, Y, and Z be three
sets of random variables; then X and Y are said to be
conditionally independent given Z, if and only if
p(x|z,y)=p(x|z) for all possible values x, y, and z of X, Y,
and Z.
Directed Acyclic Graph (DAG): A graph with di-
rected arcs containing no cycles; in this type of graph,
for any node, there is no directed path returning to it.
Probabilistic Graphical Model: A graph with nodes
representing stochastic variables annotated by probabil-
ity distributions and representing assumptions of condi-
tional independence among its variables.
Statistical Independence: Let X and Y be two dis-
joint sets of random variables; then X is said to be
independent of Y, if and only if p(x)=p(x|y) for all
possible values x and y of X and Y.
677
TEAM LinG
678
Learning Information Extraction Rules for Web
Data Mining
Chia-Hui Chang
National Central University, Taiwan
Chun-Nan Hsu
Institute of Information Science, Academia Sinica, Taiwan
INTRODUCTION
The explosive growth and popularity of the World Wide
Web has resulted in a huge number of information sources
on the Internet. However, due to the heterogeneity and
the lack of structure of Web information sources, access
to this huge collection of information has been limited to
browsing and keyword searching. Sophisticated Web-
mining applications, such as comparison shopping, re-
quire expensive maintenance costs to deal with different
data formats. The problem in translating the contents of
input documents into structured data is called informa-
tion extraction (IE). Unlike information retrieval (IR), which
concerns how to identify relevant documents from a
document collection, IE produces structured data ready
for post-processing, which is crucial to many applications
of Web mining and search tools.
Formally, an information extraction task is defined by
its input and its extraction target. The input can be
unstructured documents like free text that are written in
natural language or semi-structured documents that are
pervasive on the Web, such as tables, itemized and
enumerated lists, and so forth. The extraction target of an
IE task can be a relation of k-tuple (where k is the number
of attributes in a record), or it can be a complex object with
hierarchically organized data. For some IE tasks, an at-
tribute may have zero (missing) or multiple instantiations
in a record. The difficulty of an IE task can be complicated
further when various permutations of attributes or typo-
graphical errors occur in the input documents.
Programs that perform the task of information extrac-
tion are referred to as extractors or wrappers. A wrapper
is originally defined as a component in an information
integration system that aims at providing a single uniform
query interface to access multiple information sources. In
an information integration system, a wrapper is generally
a program that wraps an information source (e.g., a data-
base server or a Web server) such that the information
integration system can access that information source
without changing its core query answering mechanism. In
the case where the information source is a Web server, a
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
wrapper must perform information extraction in order to
extract the contents in HTML documents.
Wrapper induction (WI) systems are software tools
that are designed to generate wrappers. A wrapper usu-
ally performs a pattern-matching procedure (e.g., a form of
finite-state machines), which relies on a set of extraction
rules. Tailoring a WI system to a new requirement is a task
that varies in scale, depending on the text type, domain,
and scenario. To maximize reusability and minimize main-
tenance cost, designing a trainable WI system has been
an important topic in research fields, including message
understanding, machine learning, pattern mining, and so
forth. The task of Web IE differs largely from traditional
IE tasks in that traditional IE aims at extracting data from
totally unstructured free texts that are written in natural
language. In contrast, Web IE processes online docu-
ments that are semi-structured and usually generated
automatically by a server-side application program. As a
result, traditional IE usually take advantage of natural
language processing techniques such as lexicons and
grammars, while Web IE usually applies machine learning
and pattern-mining techniques to exploit the syntactical
patterns or to lay out structures of the template-based
documents.
BACKGROUND
In the past few years, many approaches of WI systems
and how to apply machine learning and pattern mining
techniques to train WI systems have been proposed with
various degrees of automation. Kushmerick and Thomas
(2003) conducted a survey that categorizes WI systems
based on the wrapper programs underlying formalism
(whether they are finite-state approaches or Prolog-like
logic programming systems). Sarawagi (2002) further dis-
tinguished deterministic finite-state approaches from
probabilistic hidden Markov models in her 2002 VLDB
tutorial. Another survey can be found in Laender, et al.
(2002), which categorizes Web extraction tools into six
classes based on their underlying techniquesdeclara-
TEAM LinG
Learning Information Extraction Rules for Web Data Mining
tive languages, HTML structure analysis, natural lan-
guage processing, machine learning, data modeling, and
ontology.
MAIN THRUST
We classify previous work in Web IE into three catego-
ries. The first category contains the systems that require
users to possess programming expertise. This category of
wrapper generation systems provides specialized lan-
guages or toolkits for wrapper construction, such as W4F
(Sahuguet & Azavant, 2001) and XWrap (Liu et al., 2000).
Such languages or toolkits were proposed as alternatives
to general-purpose languages in order to allow program-
mers to concentrate on formulating the extraction rules
without being concerned about the detailed process of
input strings. To apply these systems, users must learn
the language in order to write their extraction rules. There-
fore, such systems also feature user-friendly interfaces
for easy use of the toolkits. However, writing correct
extraction rules requires significant programming exper-
tise. In addition, since the structures of Web pages are not
always obvious and change frequently, writing special-
ized extraction rules can be time-consuming, error-prone,
and not scalable to a large number of Web sites. There-
fore, there is a need for automatic wrapper induction that
can generalize extraction rules for each distinct IE task.
The second category contains the WI systems that
require users to label some extraction targets as training
examples for WI systems to apply a machine-learning
algorithm to learn extraction rules from the training ex-
amples. No programming is needed to configure these WI
systems. Many IE tasks for Web mining belong to this
category; for example, IE for semi-structured text such as
RAPIER (Califf & Mooney, 1999), SRV (Freitag, 2000),
WHISK (Soderland, 1999), and for IE for template-based
pages such as WIEN (Kushmerick et al., 2000), SoftMealy
(Hsu and Dung, 1998), STALKER (Muslea, et al., 2001),
and so forth. Compared to the first category, these WI
systems are preferable, since general users, instead of
only programmers, can be trained to use these WI systems
for wrapper construction.
However, since the learned rules only apply to Web
pages from a particular Web site, labeling training ex-
amples can be laborious, especially when we need to
extract contents from thousands of data sources. There-
fore, researchers have focused on developing tools that
can reduce labeling effort. For instance, Muslea, et al.
(2002) proposed selective sampling, a form of active
learning that reduces the number of training examples.
Chidlovskii, et al. (2000) designed a wrapper generation
system that requires a small amount (one training record)
of labeling by the user. Earlier annotation-based WI
systems place emphasize on the learning techniques in
their paper. Recently, several works have been proposed L
to simplify the annotation process. For example, Lixto
(Baumgartner et al., 2001), DEByE (Laender et al., 2002)
and OLERA (Chang & Kuo, 2004) are three such systems
that stress the importance of how annotation or examples
are received from users. Note that OLERA also features
the so-called semi-supervised approach, which receives
rough rather than exact and perfect examples from users
to reduce labeling effort.
The third category contains the WI systems that do
not require any preprocessing of the input documents by
the users. We call them annotation-free WI systems.
Example systems include IEPAD (Chang & Lui, 2001),
RoadRunner (Crescenzi et al., 2001), DeLa (Wang &
Lochovsky, 2003), and EXALG (Arasu & Garcia-Molina,
2003). Since no extraction targets are specified, such WI
systems make heuristic assumptions about the data to be
extracted. For example, the first three systems assume the
existence of multiple tuples to be extracted in one page;
therefore, the approach is to discover repeated patterns
in the input page. With such an assumption, IEPAD and
DeLa only apply to Web pages that contain multiple data
tuples. RoadRunner and EXALG, on the other hand, try to
extract structured data by deducing the template and the
schema of the whole page from multiple Web pages. Their
assumption is that strings that are stationary across
pages are presumably template, and strings that are vari-
ant are presumably schema and need to be extracted.
However, as commercial Web pages often contain mul-
tiple topics where a lot of information is embedded in a
page for navigation, decoration, and interaction pur-
poses, their systems may extract both useful and useless
information from a page. However, the criterion of what is
useful is quite subjective and depends on the application.
In summary, these approaches are, in fact, not fully
automatic. Rather, post-processing is required for users
to select useful data and to assign the data to a proper
attribute.
Task Difficulties
A critical issue of WI systems is what types of documents
and structuring variations can be handled. Documents
can be classified into structured, semi-structured, and
unstructured sets (Hsu & Dung, 1998). Early IE systems
like RAPIER, SRV, and WHISK are designed to handle
documents that contain semi-structured texts, while re-
cent IE systems are designed mostly to handle documents
that contain semi-structured data (Laender et al., 2002). In
this survey, we focus on semi-structured data extraction
and possible structure variation. These include missing
data attributes, multi-valued attributes, attribute permu-
tations, nested data structures, and so forth. Table 1 lists
679
TEAM LinG
 Learning Information Extraction Rules for Web Data Mining

these variations and whether a WI system can correctly
extract the contents from a document with a layout variation.
Most WI systems can handle missing attributes except
for WIEN. Multi-valued attributes can be considered as a
special case of nested objects. However, it is neither right
nor wrong to say annotation-free WI systems can handle
multi-valued attributes, because it depends on how the
values are delimited (by HTML tags or by delimiters like
comma, spaces, etc.). Similarly, though RoadRunner and
EXALG support the extraction of nested objects in gen-
eral, whether they can handle a particular set of documents
with nested objects depends, in fact, on the quality of
template-based input.
Permutations of attributes refer to multiple attribute
orders in different data tuples in the target documents (see
the PMI example in Chang & Kuo, 2004). Note that both
missing attributes and permutations of attributes can lead
to multiple attribute orders. Some approaches (e.g.,
STALKER and multi-pass SoftMealy) utilize multiple scans
to deal with attribute permutation. Some (e.g., IEPAD,
DeLa) employ string alignment technique for this issue.
However, the way they handle the extract data results in
different power. EXALG combines two equivalence classes
to produce disjunction rules for handling permutated at-
tributes. However, the procedure may fail when all tokens
fall in one equivalence class or when no equivalence class
is formed.
Another difficult issue is what we called common
delimiters (CD) for attributes and record boundaries. An
example can be found in an Internet address finder docu-
ment set from the repository of information sources for
information extraction (RISE, http://www.isi.edu/info-
agents/RISE/repository.html). In that document set, HTML
tags like <TR> and <TD> are used as delimiters for both
record boundaries and attributes. Such document sets are
especially difficult for annotation-free systems. Even for
annotation-based systems, the problem cannot be com-
pletely handled with their default extraction mechanism.
Nonexistent delimiters (ND) also cause problems for some
WI systems that rely on delimiter-based extraction rules.
For example, suppose we want to extract the department
code and course number from the following string:
COMP4016. WI systems that depend on recognizing de-
limiters will fail in this case, because no delimiter exists
between the department codes COMP and the course
number 4016. Note that delimiter-related issues some-
times are caused by the tokenization/encoding method
(see next paragraph) of the WI systems. Therefore, they
do not necessarily cause problems for all WI systems.
Finally, most WI systems assume that the attributes of a
data object occur in a contiguous string, which does not
interleave with other data objects except for MDR (Liu et
al., 2003), which is able to handle non-contiguous (NC)
data records.
Encoding Scheme, Scanning Passes,
and Other Features
Table 2 compares important features of WI systems. In
order to learn the set of extraction rules, WI systems
need to know how to segment a string of characters (the
input) into tokens. For example, SoftMealy segments a
page into tokens including HTML tags as well as words
separated by spaces and uses a token taxonomy tree for
extraction rule generation. On the other hand, IEPAD
and RoadRunner regard every text string between two
HTML tags as one token, which leads to coarser extrac-
tion granularity. Most WI systems have a predefined
feature set for rule generalization. Still, some systems
such as OLERA (Chang & Kuo, 2004) explicitly allow
user-defined encoding schemes for tokenization. The
extraction mechanisms of various WI systems also
play an important role for extraction efficiency. Some
WI systems scan the input document once, referred to
as single-pass extractor. Others scan the input docu-
ment several times to complete the extraction. Gener-
ally speaking, single-pass wrappers are more efficient
than multi-pass wrappers. However, multi-pass wrap-
pers are more effective at handling data objects with
unrestricted attribute permutations or complex object
extraction.
Some of the WI systems have special characteristics
or requirements that deserve discussion. For example,
EXALG limits extraction failure to only part of the records
(a few attributes) instead of the entire page. STALKER

Table 1. The expressive power of various WI systems

WI systems Nested Missing Multi- Permute ND CD NC

Objects valued
Annotation- WIEN No No No No No No No
based SoftMealy No Yes Yes Yes Yes Part No
STALKER Yes Yes Yes Yes Yes Part No
OLERA Yes Yes Yes Limited Yes No No
Annotation- IEPAD No Yes -- Limited Part No No
free DeLa Yes Yes -- Yes Part No No
RoadRunner Yes* Yes -- No No No No
EXALG Yes* Yes -- Limited No No No

680

TEAM LinG
Learning Information Extraction Rules for Web Data Mining

Table 2. Features of WI systems (C*=consecutive, E*=episodic)

WI Systems Encoding Passes EF ER Requirement

L
Annotation- WIEN Word Single Entire 1 C*
based SoftMealy Word Single / Entire / Disj C*
Multiple Partial
Stalker Word Multiple Partial Disj E*
OLERA Multiple Both Partial Disj C*
Annotation- IEPAD HTML Single Entire Disj C* Only multi-
free DeLa HTML + Single Entire Disj C* record pages
Word
Roadrunner HTML Single Entire 1 C* At least two
EXALG Word Single Partial Disj C* input pages
required

and multi-pass SoftMealy also feature this characteristic CONCLUSION

(see the extraction failure [EF] column). In such cases,
lower-level encoding schemes may be necessary for fine-
grain extraction tasks. Therefore, multi-level encodings
are even more important. As for requirements, IEPAD
cannot handle single-record pages, for it assumes that
there are multiple records in one page. RoadRunner and
EXALG require at least two pages as the input in order to
generate extraction templates. In contrast, most other WI
systems can potentially generate extraction rules from
just one example page.
The supports for disjunctive rules or extraction rules
that allow intermittent delimiters (i.e., episodic rules) in
addition to consecutive delimiters also are an indication
of the expressive power of WI systems (see the extraction
rule [ER] column). These supports are helpful when no
single consecutive extraction rule can be generalized;
thus, disjunctive rules and episodic rules serve as alter-
natives. Finally, accuracy and automation are two of the
major concerns for all WI systems. For annotation-free
systems, they may reach a status where several patterns
or grammars are induced. Some systems (e.g., IEPAD)
leave the decision to users to choose the correct pattern
or grammar; therefore, they are not fully automatic. Fully
automatic systems may require further analysis on the
derived schema to ensure good quality data extraction
(Yang et al., 2003).
FUTURE TRENDS
The most critical issue for these state-of-the-art WI sys-
tems is that the generated extraction rules only apply to
a small collection of documents that share the same layout
structure. Usually, those rules only apply to documents
from one Web site. Generalization across different layout
structures is still an extremely difficult and challenging
research problem. Before this problem can be resolved,
information integration and data mining to the scale of
billions of Web sites is still too expensive and impractical.
In this overview, we present our taxonomy of WI systems
from the users viewpoint and compare important features
of WI systems that affect their effectiveness. We focus on
semi-structured documents that usually are produced
with templates by a server-side Web application. The
extension to non-HTML can be achieved for some WI
systems, if some proper tokenization or feature set is used.
Some of the WI systems surveyed here have been applied
successfully in large-scale real-world applications, in-
cluding Web intelligence, comparison shopping, knowl-
edge management, and bioinformatics (Chang et al., 2003;
Hsu et al., 2005).
REFERENCES
Arasu, A., & Garcia-Molina, H. (2003). Extracting struc-
tured data from Web pages. Proceedings of ACM SIGMOD
International Conference on Management of Data (pp. 337-
348), San Diego, CA, USA.
Baumgartner, R., Flesca, S., & Gottlob, G. (2001). Super-
vised wrapper generation with Lixto. Proceedings of the
27th International Conference on Very Large Data Bases
(VLDB) (pp. 715-716), Roma, Italy.
Califf, M., & Mooney, R. (1999). Relational learning of
pattern-match rules for information extraction. Proceed-
ings of the 16th National Conference on Artificial Intel-
ligence (AAAI) (pp. 328-334), Orlando, FL, USA.
Chang, C.-H., & Kuo, S.-C. (2004). OLERA: A semi-super-
vised approach for Web data extraction with visual sup-
port. IEEE Intelligent Systems, 19(6), 56-64.
Chang, C.-H., & Lui, S.-C. (2001). IEPAD: Information
extraction based on pattern discovery. Proceedings of
the 10th International World Wide Web (WWW) Confer-
ence (pp. 681-688), Hong Kong, China.

681

TEAM LinG
 Learning Information Extraction Rules for Web Data Mining
Chang, C.-H., Siek, H., Lu, J.-J., Chiou, J.-J., & Hsu, C.-N.
(2003). Reconfigurable Web wrapper agents. IEEE Intel-
ligent Systems, 18(5), 34-40.
Chidlovskii, B., Ragetli, J., & Rijke, M. (2000). Automatic
wrapper generation for Web search engines. Proceedings
of WAIM (pp. 399-410), Shanghai, China.
Crescenzi, V., Mecca, G., & Merialdo, P. (2001). Roadrun-
ner: Towards automatic data extraction from large Web
sites. Proceedings of the 27th International Conference
on Very Large Data Bases (VLDB) (pp. 109-118), Roma,
Italy.
Freitag, D. (2000). Machine learning for information ex-
traction in informal domains. Machine Learning, 39(2/3),
169-202.
Hsu, C.-N., Chang, C.-H., Hsieh, C.-H., Lu, J.-J., & Chang,
C.-C. (2004). Reconfigurable Web wrapper agents for
biological information integration. Journal of the Ameri-
can Society for Information Science and Technology
(JASIST), 56(5), 505-517.
Hsu, C.-N., & Dung, M.-T. (1998). Generating finite-state
transducers for semi-structured data extraction from the
Web. Information Systems, 23(8), 521-538.
Kushmerick, N. (2000). Wrapper induction: Efficiency and
expressiveness. Artificial Intelligence Journal, 118(1-
2), 15-68.
Kushmerick, N., & Thomas, B. (2002). Adaptive informa-
tion extraction: Core technologies for information agents.
Intelligent Information Agents R&D In Europe: An
AgentLink Perspective. Lecture Notes in Computer Sci-
ence, 2586, 2003, 79-103. Springer.
Laender, A.H.F., Ribeiro-Neto, B.A., & Da Silva, A.S.
(2002). DEByEData extraction by example. Data and
Knowledge Engineering, 40(2), 121-154.
Laender, A.H.F., Ribeiro-Neto, B.A., Da Silva, A.S., &
Teixeira, J.S. (2002). A brief survey of Web data extraction
tools. SIGMOD Record, 31(2), 84-93.
Liu, B., Grossman, R., & Zhai, Y. (2003). Mining data
records in Web pages. Proceedings of the Ninth ACM
SIGKDD International Conference on Knowledge Dis-
covery and Data Mining (pp. 601-606), Washington, D.C.,
USA.
Liu, L., Pu, C., & Han, W. (2000). Xwrap: An Xml-enabled
wrapper construction system for Web information sources.
Proceedings of the 16th International Conference on
Data Engineering (ICDE) (pp. 611-621), San Diego, Cali-
fornia, USA.
682
Muslea, I. (1999). Extraction patterns for information ex-
traction tasks: A survey. The AAAI-99 Workshop on
Machine Learning for Information Extraction (pp. 435-
442), Sydney, Australia.
Muslea, I., Minton, S., & Knoblock, C.A. (2001). Hierarchi-
cal wrapper induction for semi-structured information
sources. Journal of Autonomous Agents and Multi-Agent
Systems, 4, 93-114.
Muslea, I., Minton, S., & Knoblock, C. (2002). Active +
semi-supervised learning = Robust multi-view learning.
Proceedings of the 19th International Conference on
Machine Learning (ICML), Sydney, Australia.
Sahuguet, A., & Azavant, F. (2001). Building intelligent
Web applications using lightweight wrappers. Data and
Knowledge Engineering, 36(3), 283-316.
Sarawagi, S. (2002). Automation in information extraction
and integration. Proceedings of the 28th International
Conference on Very Large Data Bases (VLDB), Tutorial,
Hong Kong, China.
Soderland, S. (1999). Learning information extraction rules
for semi-structured and free text. Journal of Machine
Learning, 34(1-3), 233-272.
Wang, J., & Lochovsky, F.H. (2003). Data extraction and
label assignment. Proceedings of the 10th International
World Wide Web Conference (pp. 187-196), Budapest,
Hungary.
Yang, G., Ramakrishnan, I.V., & Kifer, M. (2003). On the
complexity of schema inference from Web pages in the
presence of nullable data attributes. Proceedings of the
12th International Conference on Information and
Knowledge Management (pp. 224-231), New Orleans,
Louisiana, USA.
KEY TERMS
Hidden Markov Model: A variant of a finite state
machine having a set of states, an output alphabet,
transition probabilities, output probabilities, and initial
state probabilities. It is only the outcome, not the state
visible to an external observer, and, therefore, states are
hidden to the outside; hence, the name Hidden Markov Model.
Information Extraction: An information extraction
task is to extract or pull out of user-defined and pertinent
information from input documents.
Logic Programming: A declarative, relational style of
programming based on first-order logic. The original logic
TEAM LinG
Learning Information Extraction Rules for Web Data Mining
programming language was Prolog. The concept is based
on Horn clauses.
Semi-Structured Documents: Semi-structured (or
template-based) documents refer to documents that are
formatted between structured and unstructured documents.
Software Agents: An artificial agent that operates in
a software environment such as operating systems, com-
puter applications, databases, networks, and virtual domains.
Transducer: A finite state machine specifically with
a read-only input and a write-only output. The input and L
output cannot be reread or changed.
Wrapper: A program that extracts data from the input
documents and wraps them in user desired and structured
form.
683
TEAM LinG
684
Locally Adaptive Techniques for Pattern
Classification
Carlotta Domeniconi
George Mason University, USA
Dimitrios Gunopulos
University of California, USA
INTRODUCTION
Pattern classification is a very general concept with nu-
merous applications ranging from science, engineering,
target marketing, medical diagnosis, and electronic com-
merce to weather forecast based on satellite imagery. A
typical application of pattern classification is mass mail-
ing for marketing. For example, credit card companies
often mail solicitations to consumers. Naturally, they
would like to target those consumers who are most likely
to respond. Often, demographic information is available
for those who have responded previously to such solici-
tations, and this information may be used in order to target
the most likely respondents. Another application is elec-
tronic commerce of the new economy. E-commerce pro-
vides a rich environment to advance the state of the art in
classification, because it demands effective means for text
classification in order to make rapid product and market
recommendations.
Recent developments in data mining have posed new
challenges to pattern classification. Data mining is a
knowledge-discovery process whose aim is to discover
unknown relationships and/or patterns from a large set of
data, from which it is possible to predict future outcomes.
As such, pattern classification becomes one of the key
steps in an attempt to uncover the hidden knowledge
within the data. The primary goal is usually predictive
accuracy, with secondary goals being speed, ease of use,
and interpretability of the resulting predictive model.
While pattern classification has shown promise in
many areas of practical significance, it faces difficult
challenges posed by real-world problems, of which the
most pronounced is Bellmans curse of dimensionality,
which states that the sample size required to perform
accurate prediction on problems with high dimensionality
is beyond feasibility. This is because, in high dimensional
spaces, data become extremely sparse and are apart from
each other. As a result, severe bias that affects any
estimation process can be introduced in a high-dimen-
sional feature space with finite samples.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Learning tasks with data represented as a collection of
a very large number of features abound. For example,
microarrays contain an overwhelming number of genes
relative to the number of samples. The Internet is a vast
repository of disparate information growing at an expo-
nential rate. Efficient and effective document retrieval and
classification systems are required to turn the ocean of
bits around us into useful information and, eventually,
into knowledge. This is a challenging task, since a word
level representation of documents easily leads 30,000 or
more dimensions.
This paper discusses classification techniques to
mitigate the curse of dimensionality and to reduce bias by
estimating feature relevance and selecting features ac-
cordingly. This paper has both theoretical and practical
relevance, since many applications can benefit from im-
provement in prediction performance.
BACKGROUND
In a classification problem, an observation is character-
ized by q feature measurements x = (x1 , L, x q ) q and
is presumed to be a member of one of J classes, L j ,
j = 1, L, J . The particular group is unknown, and the
goal is to assign the given object to the correct group,
using its measured features x .
Feature relevance has a local nature. Therefore, any
chosen fixed metric violates the assumption of locally
constant class posterior probabilities, and fails to make
correct predictions in different regions of the input space.
In order to achieve accurate predictions, it becomes
crucial to be able to estimate the different degrees of
relevance that input features may have in various loca-
tions of the feature space.
Consider, for example, the rule that classifies a new
data point with the label of its closest training point in the
measurement space (1-Nearest Neighbor rule). Suppose
TEAM LinG
Locally Adaptive Techniques for Pattern Classification
that each instance is described by 20 features, but only
three of them are relevant to classifying a given instance.
In this case, two points that have identical values for the
three relevant features may, nevertheless, be distant from
one another in the 20-dimensional input space. As a
result, the similarity metric that uses all 20 features will be
misleading, since the distance between neighbors will be
dominated by the large number of irrelevant features. This
shows the effect of the curse of dimensionality phenom-
enon; that is, in high dimensional spaces, distances
between points within the same class or between different
classes may be similar. This fact leads to highly-biased
estimates. Nearest neighbor approaches (Ho, 1998; Lowe,
1995) are especially sensitive to this problem.
In many practical applications, things often are further
complicated. In the previous example, the three relevant
features for the classification task at hand may be depen-
dent on the location of the query point (i.e., the point to
be classified) in the feature space. Some features may be
relevant within a specific region, while other features may
be more relevant in a different region. Figure 1 illustrates
a case in point, where class boundaries are parallel to the
coordinate axes. For query a, dimension X is more rel-
evant, because a slight move along the X axis may change
the class label, while for query b, dimension Y is more
relevant. For query c, however, both dimensions are
equally relevant.
These observations have two important implications.
Distance computation does not vary with equal strength
or in the same proportion in all directions in the feature
space emanating from the input query. Moreover, the
value of such strength for a specific feature may vary from
location to location in the feature space. Capturing such
information, therefore, is of great importance to any clas-
sification procedure in high-dimensional settings.
MAIN THRUST
Severe bias can be introduced in pattern classification in
a high dimensional input feature space with finite samples.
In the following, we introduce adaptive metric techniques
Figure 1. Feature relevance varies with query locations
for distance computation capable of reducing the bias of
the estimation. L
Friedman (1994) describes an adaptive approach (the
Machete and Scythe algorithms) for classification that
combines some of the best features of kNN learning and
recursive partitioning. The resulting hybrid method in-
herits the flexibility of recursive partitioning to adapt the
shape of the neighborhood N (x 0 ) of query x0 , as well as
the ability of nearest neighbor techniques to keep the
points within N (x 0 ) close to the point being predicted.
The method is capable of producing nearly continuous
probability estimates with the region N (x 0 ) centered at x0
and the shape of the region separately customized for
each individual prediction point.
The major limitation concerning the Machete/Scythe
method is that, like recursive partitioning methods, it
applies a greedy strategy. Since each split is conditioned
on its ancestor split, minor changes in an early split, due
to any variability in parameter estimates, can have a
significant impact on later splits, thereby producing dif-
ferent terminal regions. This makes the predictions highly
sensitive to the sampling fluctuations associated with the
random nature of the process that produces the training
data and, therefore, may lead to high variance predictions.
In Hastie and Tibshirani (1996), the authors propose
a discriminant adaptive nearest neighbor classification
method (DANN), based on linear discriminant analysis.
Earlier related proposals appear in Myles and Hand (1990)
and Short and Fukunaga (1981). The method in Hastie and
Tibshirani (1996) computes a local distance metric as a
product of weighted within and between the sum of
squares matrices. The authors also describe a method of
performing global dimensionality reduction by pooling
the local dimension information over all points in the
training set (Hastie & Tibshirani, 1996a, 1996b).
While sound in theory, DANN may be limited in
practice. The main concern is that in high dimensions, one
may never have sufficient data to fill in q q (within and
between sum of squares) matrices (where q is the dimen-
sionality of the problem). Also, the fact that the distance
metric computed by DANN approximates the weighted
Chi-squared distance only when class densities are
Gaussian and have the same covariance matrix may cause
a performance degradation in situations where data do not
follow Gaussian distributions or are corrupted by noise,
which is often the case in practice.
A different adaptive nearest neighbor classification
method (ADAMENN) has been introduced to try to mini-
mize bias in high dimensions (Domeniconi, Peng &
Gunopulos, 2002) and to overcome the previously men-
tioned limitations. ADAMENN performs a Chi-squared
distance analysis to compute a flexible metric for produc-
685
TEAM LinG

ing neighborhoods that are highly adaptive to query loca-
tions. Let x be the nearest neighbor of a query x0 com-
puted according to a distance metric D(x, x0 ) . The goal is
to find a metric D(x, x0 ) that minimizes E[ r (x 0 , x)] , where
r (x 0 , x) = j =1 Pr( j | x 0 )(1 Pr( j | x)) . Here Pr( j | x) is the
J
class conditional probability at x . That is, r (x 0 , x) is the
finite sample error risk given that the nearest neighbor to
x0 by the chosen metric is x . It can be shown (Domeniconi,
Peng & Gunopulos, 2002) that the weighted Chi-squared
distance
J
[Pr( j | x) Pr( j | x0 )]2
D(x, x 0 ) = . (1)
j =1 Pr( j | x0 )
approximates the desired metric, thus providing the founda-
tion upon which the ADAMENN algorithm computes a mea-
sure of local feature relevance, as shown in the following.
The first observation is that Pr( j | x) is a function of
x . Therefore, one can compute the conditional expecta-
tion of Pr( j | x) , denoted by Pr( j | xi = z ) , given that xi
assumes value z , where xi represents the i th compo-
nent of x . That is,
Pr( j | xi = z ) = E[Pr( j | x) | xi = z ] = Pr( j | x) p( x | xi = z ) dx .
Here, p (x | xi = z ) is the conditional density of the other
input variables defined as
p (x | xi = z ) = p (x) ( xi z )/ p (x) ( xi z )dx ,
where ( x z ) is the Dirac delta function having the
properties (x z ) = 0 if x z and
( x z )dx = 1 . Let
J
[Pr( j | z ) Pr( j | xi = zi )]2
ri (z ) = . (2)
j =1 Pr( j | xi = zi )
ri (z ) represent the ability of feature i to predict the
Pr( j | z) s at xi = zi . The closer Pr( j | xi = zi ) is to Pr( j | z ) ,
the more information feature i carries for predicting the
class posterior probabilities locally at z .
We now can define a measure of feature relevance for
x0 as
686
Locally Adaptive Techniques for Pattern Classification
1
r i (x 0 ) =
K
zN ( x0 )
ri (z ), (3)
where N (x 0 ) denotes the neighborhood of x0 containing
the K nearest training points, according to a given
metric. r i measures how well on average the class pos-
terior probabilities can be approximated along input
feature i within a local neighborhood of x0 . Small r i
implies that the class posterior probabilities will be well
approximated along dimension i in the vicinity of x0 .
Note that r i (x0 ) is a function of both the test point x0 and
the dimension i , thereby making r i (x0 ) a local relevance
measure in dimension i .
The relative relevance, as a weighting scheme, then
R ( x )t
can be given by wi (x0 ) = q Rl ( x0 )t , where t = 1, 2 , giving
i 0
l =1
rise to linear and quadratic weightings, respectively, and
Ri (x 0 ) = max qj =1{r j (x 0 )} r i (x 0 ) (i.e., the larger the Ri , the
more relevant dimension i ). We propose the following
exponential weighting scheme
q
wi (x0 ) = exp(cRi ( x0 )) / exp(cRl ( x0 )) (4)
l =1
where c is a parameter that can be chosen to maximize
(minimize) the influence of r i on wi . When c = 0 , we have
wi = 1/q , which has the effect of ignoring any differ-
ence among the r i s. On the other hand, when c is large,
a change in r i will be exponentially reflected in wi . The
exponential weighting is more sensitive to changes in
local feature relevance and, in general, gives rise to better
performance improvement. In fact, it is more stable,
because it prevents neighborhoods from extending infi-
nitely in any direction (i.e., zero weight). However, this
can occur when either linear or quadratic weighting is
used. Thus, equation (4) can be used to compute the
weight associated with each feature, resulting in the
weighted distance computation:
q
D(x, y ) = w (x y )
i =1
i i i
2
. (5)
The weights w i enable the neighborhood to elon-
gate less important feature dimensions and, at the same
time, to constrict the most influential ones. Note that
TEAM LinG
Locally Adaptive Techniques for Pattern Classification
the technique is query-based, because the weights de-
pend on the query (Aha, 1997; Atkeson, Moore & Shaal,
1997).
An intuitive explanation for (2) and, hence, (3) goes as
follows. Suppose that the value of ri (z) is small, which
implies a large weight along dimension i . Consequently,
the neighborhood is shrunk along that direction. This, in
turn, penalizes points along dimension i that are moving
away from z i . Now, ri (z ) can be small, only if the subspace
spanned by the other input dimensions at xi = z i likely
contains samples similar to z in terms of the class condi-
tional probabilities. Then, a large weight assigned to
dimension i based on (4) says that moving away from the
subspace and, hence, from the data similar to z is not a
good thing to do. Similarly, a large value of ri (z) and,
hence, a small weight indicates that in the vicinity of z i
along dimension i , one is unlikely to find samples similar
to z . This corresponds to an elongation of the neighbor-
hood along dimension i . Therefore, in this situation, in
order to better predict the query, one must look farther
away from z i .
One of the key differences between the relevance
measure (3) and Friedmans is the first term in the squared
difference. While the class conditional probability is used
in (3), its expectation is used in Friedmans. This differ-
ence is driven by two different objectives: in the case of
Friedmans, the goal is to seek a dimension along which
the expected variation of Pr( j | x) is maximized, whereas in
(3) a dimension is found that minimizes the difference
between the class probability distribution for a given
query and its conditional expectation along that dimen-
sion (2). Another fundamental difference is that the ma-
chete/scythe methods, like recursive partitioning, employ
a greedy peeling strategy that removes a subset of data
points permanently from further consideration. As a re-
sult, changes in an early split, due to any variability in
parameter estimates, can have a significant impact on later
splits, thereby producing different terminal regions. This
makes predictions highly sensitive to the sampling fluc-
tuations associated with the random nature of the process
that produces the training data, thus leading to high
variance predictions. In contrast, ADAMENN employs a
patient averaging strategy that takes into account not
only the test point x0 , but also its K 0 nearest neighbors.
As such, the resulting relevance estimates (3) are, in
general, more robust and have the potential to reduce the
variance of the estimates.
In Hastie and Tibshirani (1996), the authors show
that the resulting metric approximates the weighted
Chi-squared distance (1) by a Taylor series expansion,
given that class densities are Gaussian and have the same L
covariance matrix. In contrast, ADAMENN does not
make such assumptions, which are unlikely in real-
world applications. Instead, it attempts to approximate
the weighted Chi-Squared distance (1) directly. The
main concern with DANN is that, in high dimensions, we
may never have sufficient data to fill in q q matrices.
It is interesting to note that the ADAMENN algorithm
potentially can serve as a general framework upon which
to develop a unified adaptive metric theory that encom-
passes both Friedmans work and that of Hastie and
Tibshirani.
FUTURE TRENDS
Almost all problems of practical interest are high dimen-
sional. With the recent technological trends, we can
expect an intensification of research efforts in the area of
feature relevance estimation and selection. In
bioinformatics, the analysis of micro-array data poses
challenging problems. Here, one has to face the problem
of dealing with more dimensions (genes) than data points
(samples). Biologists want to find marker genes that are
differentially expressed in a particular set of conditions.
Thus, methods that simultaneously cluster genes and
samples are required to find distinctive checkerboard
patterns in matrices of gene expression data. In cancer
data, these checkerboards correspond to genes that are
up- or down-regulated in patients with particular types of
tumors. Increased research efforts in this area are needed
and expected.
Clustering is not exempt from the curse of dimension-
ality. Several clusters may exist in different subspaces,
comprised of different combinations of features. Since
each dimension could be relevant to at least one of the
clusters, global dimensionality reduction techniques are
not effective. We envision further investigation on this
problem with the objective of developing robust tech-
niques in the presence of noise.
Recent developments on kernel-based methods sug-
gest a framework to make the locally adaptive techniques
discussed previously more general. One can perform
feature relevance estimation in an induced feature space
and then use the resulting kernel metrics to compute
distances in the input space. The key observation is that
kernel metrics may be non-linear in the input space but are
still linear in the induced feature space. Hence, the use of
suitable non-linear features allows the computation of
locally adaptive neighborhoods with arbitrary orienta-
tions and shapes in input space. Thus, more powerful
classification techniques can be generated.
687
TEAM LinG
 Locally Adaptive Techniques for Pattern Classification
CONCLUSION
Pattern classification faces a difficult challenge in fi-
nite settings and high dimensional spaces, due to the
curse of dimensionality. In this paper, we have presented
and compared techniques to address data exploration
tasks such as classification and clustering. All methods
design adaptive metrics or parameter estimates that are
local in input space in order to dodge the curse of
dimensionality phenomenon. Such techniques have been
demonstrated to be effective for the achievement of
accurate predictions.
REFERENCES
Aha, D. (1997). Lazy learning. Artificial Intelligence Re-
view, 11, 1-5.
Atkeson, C., Moore, A.W., & Schaal, S. (1997). Locally
weighted learning. Artificial Intelligence Review, 11, 11-73.
Domeniconi, C., Peng, J., & Gunopulos, D. (2002). Locally
adaptive metric nearest neighbor classification. IEEE
Transactions on Pattern Analysis and Machine Intelli-
gence, 24(9), 1281-1285.
Friedman, J.H. (1994). Flexible metric nearest neighbor
classification. Technical Report. Stanford University.
Hastie, T., & Tibshirani, R (1996a). Discriminant adaptive
nearest neighbor classification. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 18(6), 607-615.
Hastie, T., & Tibshirani, R. (1996b). Discriminant analysis
by Gaussian mixtures. Journal of the Royal Statistical
Society, 58, 155-176.
Ho, T.K. (1998). Nearest neighbors in random subspaces.
Proceedings of the Joint IAPR International Workshops
on Advances in Pattern Recognition.
Lowe, D.G. (1995). Similarity metric learning for a variable-
kernel classifier. Neural Computation, 7(1), 72-85.
Myles, J.P., & Hand, D.J. (1990). The multi-class metric
problem in nearest neighbor discrimination rules. Pattern
Recognition, 23(11), 1291-1297.
Short, R.D., & Fukunaga, K. (1981). Optimal distance
measure for nearest neighbor classification. IEEE Trans-
actions on Information Theory, 27(5), 622-627.
688
KEY TERMS
Classification: The task of inferring concepts from
observations. It is a mapping from a measurement space
into the space of possible meanings, viewed as finite and
discrete target points (class labels). It makes use of
training data.
Clustering: The process of grouping objects into
subsets, such that those within each cluster are more
closely related to one another than objects assigned to
different clusters, according to a given similarity measure.
Curse of Dimensionality: Phenomenon that refers to
the fact that, in high-dimensional spaces, data become
extremely sparse and are far apart from each other. As a
result, the sample size required to perform an accurate
prediction in problems with high dimensionality is usually
beyond feasibility.
Kernel Methods: Pattern analysis techniques that
work by embedding the data into a high-dimensional
vector space and by detecting linear relations in that
space. A kernel function takes care of the embedding.
Local Feature Relevance: Amount of information that
a feature carries to predict the class posterior probabilities
at a given query.
Nearest Neighbor Methods: Simple approach to the
classification problem. It finds the K nearest neighbors
of the query in the training set and then predicts the class
label of the query as the most frequent one occurring in
the K neighbors.
Pattern: A structure that exhibits some form of regu-
larity, able to serve as a model representing a concept of
what was observed.
Recursive Partitioning: Learning paradigm that em-
ploys local averaging to estimate the class posterior
probabilities for a classification problem.
Subspace Clustering: Simultaneous clustering of both
row and column sets in a data matrix.
Training Data: Collection of observations (character-
ized by feature measurements), each paired with the cor-
responding class label.
TEAM LinG
 689

Logical Analysis of Data L

Endre Boros
RUTCOR, Rutgers University, USA

Peter L. Hammer
RUTCOR, Rutgers University, USA

Toshihide Ibaraki
Kwansei Gakuin University, Japan

INTRODUCTION theme of LAD. In principle, any Boolean function that

agrees with the given data is a potential extension and is
The logical analysis of data (LAD) is a methodology considered in LAD. However, as in general learning theory,
aimed at extracting or discovering knowledge from data in simplicity and generalization power of the chosen exten-
logical form. The first paper in this area was published as sions are the main objectives. To achieve these goals,
Crama, Hammer, & Ibaraki (1988) and precedes most of the LAD breaks up the problem of finding a most promising
data mining papers appearing in the 1990s. Its primary extension into a series of optimization problems, each with
target is a set of binary data belonging to two classes for their own objectives, first finding a smallest subset of the
which a Boolean function that classifies the data into two variables needed to distinguish the vectors in T from
classes is built. In other words, the extracted knowledge those in F (finding a so called support set), next finding all
is embodied as a Boolean function, which then will be the monomials which have the highest agreement with the
used to classify unknown data. As Boolean functions that given data (finding the strongest patterns), finally find-
classify the given data into two classes are not unique, ing a best combination of the generated patterns (finding
there are various methodologies investigated in LAD to a theory). In what follows, we shall explain briefly each of
obtain compact and meaningful functions. As will be these steps. More details about the theory of partially
mentioned later, numerical and categorical data also can defined Boolean functions can be found in Boros, et al.
be handled, and more than two classes can be represented (1998, 1999).
by combining more than one Boolean function. An example of a binary dataset (or pdBf) is shown in
Many of the concepts used in LAD have much in Table 1; Table 2 gives three extensions of it, among many
common with those studied in other areas, such as data others. Extension f1 may be considered as a most compact
mining, learning theory, pattern recognition, possibility one, as it contains only two variables, while extension f 3
theory, switching theory, and so forth, where different
terminologies are used in each area. A more detailed
description of LAD can be found in some of the references Table 1. An example for pdBf (T, F)
listed in the following and, in particular, in the forthcom-
x1 x2 x3 x4 x5 x6 x7 x8
ing book (Crama & Hammer, 2004).
0 1 0 1 0 1 1 0
T 1 1 0 1 1 0 0 1
0 1 1 0 1 0 0 1
BACKGROUND 1 0 1 0 1 0 1 0
F 0 0 0 1 1 1 0 0
Let us consider a binary dataset (T, F, where T {0,1} n 1 1 0 1 0 1 0 1
(resp., F {0,1} n) is the set of positive (resp., negative) 0 0 1 0 1 0 1 0
data (i.e., those belonging to positive (resp., negative)
class). The pair (T, F) is called a partially defined Boolean
function (or, in short, a pdBf), which is the principal Table 2. Some extensions of the pdBf (T, F) given in Table 1
mathematical notion behind the theory of LAD. A Bool-
ean function f: {0,1}n {0,1} is called an extension of the
f1 = x5 x8 x5 x8
pdBf (T, F) if f(x)=1 for all vectors x in T, and f(y)=0 for all f 2 = x1 x5 x3 x7 x1 x5 x7
vectors y in F. The construction of extensions that carry
the essential information of the given dataset is the main
f 3 = x5 x8 x6 x7

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG

also may be interesting, as it reveals a monotone depen-
dence on the involved variables.
MAIN THRUST
In many applications, the available data is not binary,
which necessitates the generation of relevant binary
features, for which the three-staged LAD analysis then
can be applied. Following are some details about all the
main stages of LAD, including the generation of binary
features, finding support sets, generating patterns, and
constructing theories.
Binarization of Numerical and
Categorical Data
Many of the existing datasets contain numerical and/or
categorical variables. Such data also can be handled by
LAD after converting such data into binary data (i.e.,
binarization). A typical method for this is to introduce a
cut point i for a numerical variable xi; if xii holds; then,
it is converted to 1, and 0 otherwise. It is possible to use
more than one cut point for a variable to create one or more
number of regions of 1. Similar ideas also can be used for
a categorical variable, defining a region converted into 1
by a subset of values of the variable. Binarization aims at
finding the cut points for numerical attributes/finding the
subsets of values for categorical attributes, such that the
given positive and negative observations can be distin-
guished with the resulting binary variables, and their
number is as small as possible. Mathematical properties
of and algorithms for binarization are studied in Boros, et
al. (1997). Efficient methods for the binarization of cat-
egorical data are proposed in Boros and Mekov (2004).
Support Sets
It is desirable from the viewpoint of simplicity to build an
extension by using as small a set of variables as possible.
A subset S of the n original variables is called a support
set, if the projections of T and F on S still has an extension.
The pdBf in Table 1 has the following minimal support
sets: S1={5,8}, S2={6,7}, S3={1,2,5}, S4={1,2,6}, S5={2,5,7},
S6={2,6,8}, S7={1,3,5,7}, S8={1,4,5,7}. For example, f1 in
Table 2 is constructed from S1. Several methods to find
small support sets are discussed and compared in Boros,
et al. (2003).
Pattern Generation
As a basic tool to construct an extension f, a conjunction
of a set of literals is called a pattern of (T, F), if there is at
690
Logical Analysis of Data
least one vector in T satisfying this conjunction, but no
vector in F satisfies it, where a literal is either a variable or
its complement. In the example of Tables 1 and 2,
x5 x8 , x5 x8 , x1 x5 , K are patterns. The notion of a pat-
tern is closely related to the association rule, which is
commonly used in data mining. Each pattern captures a
certain characteristic of (T, F) and forms a part of knowl-
edge about the data set. Several types of patterns (prime,
spanned, strong) have been analyzed in the literature
(Alexe et al., 2002), and efficient algorithms for enumerat-
ing large sets of patterns have been described (Alexe &
Hammer, 2004; Alexe & Hammer, 2004; Eckstein et al., 2004).
Theories of a pdBf
A set of patterns that together cover T (i.e., for any xT,
there is at least one pattern that x satisfies) is called a
theory of (T, F). Note that a theory defines an extension
of (T, F), since disjunction of the patterns yields such a
Boolean function. The Boolean functions fi in Table 2
correspond to theories constructed for the dataset of
Table 1. Exchanging the roles of T and F, we can define co-
patterns and co-theories, which play a similar role in LAD.
Efforts in LAD have been directed to the construction of
compact patterns and theories from (T, F), as such theo-
ries are expected to have better performance to classify
unknown data. An implementation in this direction and its
results can be found in Boros, et al. (2000). The tradeoff
between comprehensive and comprehensible theories is
studied in Alexe, et al. (2002).
FUTURE TRENDS
Extensions by Special Types of
Boolean Functions
In many cases, it is known in advance that the given
dataset has certain properties, such as monotone depen-
dence on variables. To utilize such information, exten-
sions by Boolean functions with the corresponding prop-
erties are important. For special classes of Boolean func-
tions, such as monotone (or positive), Horn, k-DNF,
decomposable, and threshold, the algorithms and com-
plexity of finding extensions were investigated (Boros et
al., 1995; Boros et al., 1998). If there is no extension in the
specified class of Boolean functions, we may still want to
find an extension in the class with the minimum number of
errors. Such an extension is called the best-fit extension
and studied in Boros, et al. (1998).
TEAM LinG
Logical Analysis of Data
Imperfect Datasets
In practice, the available data may contain missing or
erroneous bits, due to various reasons (e.g., obtaining
their values is dangerous or expensive). More typically,
the binary attributes obtained by binarization can result in
some uncertainties; for example, for a numerical variable xi
the binary attribute xii may not be evaluated with high
confidence, if the actual value of xi is too close to the value
of the cut point i (e.g., if it is within the error rate of the
measurement procedure used to obtain the value of xi). For
such binary datasets involving such missing/uncertain
bits, three types of extensionsrobust, consistent, and
most robuswere defined. For example, a robust exten-
sion is one that remains as an extension of (T, F) for any
assignment of 0, 1 values to the missing bits. Theoretical
and algorithmic studies in this direction are made in Boros,
et al. (1999, 2003).
CONCLUSION
The LAD approach has been applied successfully to a
number of real-world datasets. Among the published re-
sults, Boros, et al. (2000) deals with the datasets from the
repository of the University of California at Irvine (http:/
/www.ics.uci.edu/~mlearn/MLRepository.html), as well as
some other pilot studies. LAD has been applied success-
fully for several medical problems, including the detection
of ovarian cancer (Alexe et al., 2004), stratification of risk
among cardiac patients (Alexe et al., 2003), and polymer
design for artificial bones (Abramson et al., 2002). Among
other applications of LAD, we mention those concerning
economic problems, including an analysis of labor produc-
tivity in China (Hammer et al., 1999) and a recent analysis
of country risk ratings (Hammer et al., 2003).
REFERENCES
Abramson, S., Alexe, G., Hammer, P.L., Knight, D., & Kohn,
J. (2002). A computational approach to predicting cell
growth on polymeric biomaterials. To appear in J. Biomed.
Mater. Res.
Alexe, G. et al. (2004). Ovarian cancer detection by logical
analysis of proteomic data. Proteomics, 4(3), 766-783.
Alexe, G., Alexe, S., Hammer, P.L., & Kogan, A. (2002).
Comprehensive vs. comprehensible classifiers in LAD. To
appear in Discrete Applied Mathematics.
Alexe, G., & Hammer, P.L. (2004). Spanned patterns in the
logical analysis of data. Discrete Applied Mathematics.
Alexe, S. et al. (2003). Coronary risk prediction by logical
analysis of data. Annals of Operations Research, 119, L
15-42.
Alexe, S., & Hammer, P.L. (2004). Accelerated algorithm
for pattern detection in logical analysis of data. Discrete
Applied Mathematics.
Boros, E. et al. (2000). An implementation of logical
analysis of data. IEEE Trans. on Knowledge and Data
Engineering, 12, 292-306.
Boros, E., Gurvich, V., Hammer, P.L., Ibaraki, T., & Kogan,
A. (1995). Decompositions of partially defined Boolean
functions. Discrete Applied Mathematics, 62, 51-75.
Boros, E., Hammer, P.L., Ibaraki, T., & Kogan, A. (1997).
Logical analysis of numerical data. Mathematical Pro-
gramming, 79, 163-190.
Boros, E., Horiyama, T., Ibaraki, T., Makino, K., & Yagiura,
M. (2003). Finding essential attributes from binary data.
Annals of Mathematics and Artificial Intelligence, 39,
223-257.
Boros, E., Ibaraki, T., & Makino, K. (1998). Error-free and
best-fit extensions of a partially defined Boolean func-
tion. Information and Computation, 140, 254-283.
Boros, E., Ibaraki, T., & Makino, K. (1999). Logical analy-
sis of data with missing bits. Artificial Intelligence, 107,
219-264.
Boros, E., Ibaraki, T., & Makino K. (2003). Variations on
extending partially defined Boolean functions with miss-
ing bits. Information and Computation, 180(1), 53-70.
Boros, E., & Mekov, V. (2004). Exact and approximate
discrete optimization algorithms for finding useful dis-
junctions of categorical predicates in data analysis. Dis-
crete Applied Mathematics, 144(1-2), 43-58.
Crama, Y., & Hammer, P.L. (2004). Boolean Functions,
forthcoming.
Crama, Y., Hammer, P.L., & Ibaraki, T. (1988). Cause-
effect relationship and partially defined Boolean func-
tions. Annals of Operations Research, 16, 299-326.
Eckstein, J., Hammer, P.L., Liu, Y., Nediak, M., & Simeone,
B. (2004). The maximum box problem and its application
to data analysis. Computational Optimization and Ap-
plications, 23(3), 285-298.
Hammer, A., Hammer, P.L., & Muchnik, I. (1999). Logical
analysis of Chinese labor productivity patterns. Annals
of Operations Research, 87, 165-176.
691
TEAM LinG

Hammer, P.L., Kogan, A., & Lejeune, M.A. (2003). A non-
recursive regression model for country risk rating.
RUTCOR Research Report RRR 9-2003.
KEY TERMS
Binarization: The process of deriving a binary repre-
sentation for numerical and/or categorical attributes.
Boolean Function: A function from {0,1}n to {0,1}. A
function from a subset of {0,1}n to {0,1} is called a partially
defined Boolean function (pdBf). A pdBf is defined by a
pair of datasets (T, F), where T (resp., F) denotes a set of
data vectors belonging to positive (resp., negative) class.
Extension: A Boolean function f that satisfies f(x)=1
for xT and f(x)=0 for xF for a given pdBf (T, F).
692
Logical Analysis of Data
LAD (Logical Analysis of Data): A methodology that
tries to extract and/or discover knowledge from datasets
by utilizing the concept of Boolean functions.
Pattern: A conjunction of literals that is true for some
data vectors in T but is false for all data vectors in F, where
(T, F) is a given pdBf. A co-pattern is similarly defined by
exchanging the roles of T and F.
Support Set: A set of variables S for a data set (T, F)
such that projections of T and F on S still have an
extension.
Theory: A set of patterns of a pdBf (T, F), such that
each data vector in T has a pattern satisfying it.
TEAM LinG

The Lsquare System for Mining Logic Data
Giovanni Felici
Istituto di Analisi dei Sistemi ed Informatica (IASI-CNR), Italy
Klaus Truemper
University of Texas at Dallas, USA
INTRODUCTION
The method described in this chapter is designed for
data mining and learning on logic data. This type of data
is composed of records that can be described by the
presence or absence of a finite number of properties.
Formally, such records can be described by variables
that may assume only the values true or false, usually
referred to as logic (or Boolean) variables. In real
applications, it may also happen that the presence or
absence of some property cannot be verified for some
record; in such a case we consider that variable to be
unknown (the capability to treat formally data with
missing values is a feature of logic-based methods). For
example, to describe patient records in medical diagno-
sis applications, one may use the logic variables healthy,
old, has_high_temperature, among many others. A
very common data mining task is to find, based on
training data, the rules that separate two subsets of the
available records, or explains the belonging of the data
to one subset or the other. For example, one may desire
to find a rule that, based one the many variables observed
in patient records, is able to distinguish healthy patients
from sick ones. Such a rule, if sufficiently precise, may
then be used to classify new data and/or to gain informa-
tion from the available data. This task is often referred
to as machine learning or pattern recognition and ac-
counts for a significant portion of the research con-
ducted in the data mining community. When the data
considered is in logic form or can be transformed into
it by some reasonable process, it is of great interest to
determine explanatory rules in the form of the combina-
tion of logic variables, or logic formulas. In the ex-
ample above, a rule derived from data could be:
if (has_high_temperature is true) and (running_nose
is true) then (the patient is not healthy).
Clearly such rules convey a lot of information and
can be easily understood and interpreted by domain
experts. Despite the apparent simplicity of this setting,
the problem of determining, if possible, a logic formula
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
693
L
that holds true for all the records in one set while it is false
for all records in another set, can become extremely
difficult when the dimension involved is not trivial, and
many different techniques and approaches have been
proposed in the literature. In this article we describe one
of them, the Lsquare System, developed in collaboration
between IASI-CNR and UTD and described in detail in
Felici & Truemper (2002), Felici, Sun, & Truemper (2004),
and Truemper (2004). The system is freely distributed for
research and study purposes at www.leibnizsystem.com.
Data mining in logic domains is becoming a very
interesting topic for both research and applications. The
motivations for the study of such models are frequently
found in real life situations where one wants to extract
usable information from data expressed in logic form.
Besides medical applications, these types of problems
often arise in marketing, production, banking, finance,
and credit rating. A quick scan of the updated Irvine
Repository (see Murphy & Aha, 1994) is sufficient to
show the relevance of logic-based models in data min-
ing and learning application. The literature describes
several methods that address learning in logic domains,
for example the very popular decision trees (Breiman et
al., 1984), the highly combinatorial approach of Boros
et al. (1996), the interior point method of Kamath et al.
(1992), or the partial enumeration scheme proposed by
Triantaphyllou & Soyster (1996). While the problem
formulation adopted by Lsquare is somewhat related the
work in Kamath et al. (1992) and Triantaphyllou et al.
(1994), substantial differences are found in the solution
method adopted. Most of the methods considered in this
area are of intrinsic deterministic nature, being based on
the formal description of a problem in mathematical form
and in its solution by a specific algorithm. Nevertheless,
some real life situations present uncertainty and errors in
the data that are often successfully dealt with by the use
of fuzzy set and fuzzy membership theory. In such cases
the proposed system may embed the uncertainty and the
fuzziness of the data in a pre-processing step, providing
fuzzy functions that determine the value of the Boolean
variables.
TEAM LinG

BACKGROUND
This section provides the needed definitions and concepts.
Propositional Logic, SAT and MINSAT
A Boolean variable may take on the values true or
false. One or more Boolean variables can be assembled
in a Boolean formula using the operators , , and . A
Boolean formula where Boolean variables, possibly
negated, are joined by the operator (resp. ) is called
a conjunction (resp. disjunction). A conjunctive nor-
mal form system (CNF) is a Boolean formula where
some disjunctions are joined with the operator . A
disjunctive normal form system (DNF) is a Boolean
formula where some conjunctions are joined with the
operator . Each disjunction (resp. conjunction) of a
CNF (resp. DNF) system is called a clause. A Boolean
formula is satisfiable if there exists an assignment of
true/false values to its Boolean variables so that the
value of the formula is true. The problem of deciding
whether a CNF formula is satisfiable is known as the
satisfiability problem (SAT). In the affirmative case,
one also must find an assignment of true/false values
for the Boolean variables of the CNF that make the
formula true. A variation of SAT is the minimum cost
satisfiability problem (MINSAT), where rational costs
are associated with the true value of each logic variable
and the solution to be determined has minimum sum of
costs for the variables with value true.
Logic Data and Logic Separation
We consider {0, +/-1} vectors of given length n, each of
which has an associated outcome with value true or
false. We call these vectors records of logic data and
view them as an encoding of logic information. A 1 in a
record means that a certain Boolean variable has value
true, and a 1 that the variable has value false. The value
0 is used for unknown. The outcome is considered to be
the value of a Boolean variable t that we want to explain
or predict. We collect the records for which the prop-
erty t is present in a set A, and those for which t is not
present in set B. For ease of recognition, we usually
denote a member of A by a, and of B by b. The Lsquare
system deduces {0, +/-1} separating vectors that effec-
tively represent logic formulas and may be used to
compute for each record the associated outcome, that
is, to separate the records in A from the records in B. A
separating set is a collection of separating vectors. The
separation of A and B makes sense only when both A and
B are non-empty, and when each record of A or B
contains at least one {+/-1} entry. Consider two records
694
The Lsquare System for Mining Logic Data
of logic data; say g and f. We say that f is nested in g if for
any entry fi of f equal to +1 or 1, the corresponding entry
gi of g satisfies gi = f i. It is easy to show that if A and B are
sets of {0, +/-1} records of the same length, a separating
set S exists if and only if no record a A is nested in any
record b B. A clear characterization of separating
vectors can thus be given: a {0, +/-1} vector s separates
a record a A from B if:
s is not nested in any b B (1)
s is nested in a (2)
Accordingly, we say that a separating set S separates
B from A if, for each a A, there exists a separating
vector s S that separates a from B.
MAIN THRUST
The problem of finding a separating set for A and B is
decomposed into a sequence of subproblems, each of
which identifies a vector s that separates a nonempty
subset of A from B solving two minimization problems.
To formulate these problems we introduce Boolean
variables linked with the elements s i of the vector s to be
found. More precisely, we introduce Boolean variables
pi and qi and state that s i = 1 if pi = True and qi = False,
s i = 1 if pi = False and qi = True, and si = 0 if pi = qi =
False. The case pi = qi = True is ruled out by enforcing
the following conditions:
pi qi , i = 1, 2, ..., n (3)
We can express the separation conditions (1) and (2)
with the Boolean variables pi and qi. For (1) we have that
s must not be nested in any b B. Defining b+ as the set
of indices i for which bi of b is equal to 1, that is, b+ =
{ibi = 1} and b- = {ibi = -1}, b0 = {ibi = 0}, we
summarize condition (1) writing that
( i(b+ b0) qi) (i(b- b0) pi); bB (4)
For condition (2) we have to enforce that, if s sepa-
rates a from B, then s is nested in a. In order to do so we
introduce a new Boolean variable da that determines
whether s must separate a from B. That is, da = true
means that s need not separate a from B, while da = false
requires that separation. For a A, the separation condi-
tion is:
qi da; i (a+ a0)
pi da; i (a+ a0) (5)
TEAM LinG
The Lsquare System for Mining Logic Data
We now formulate the problem of determining a vector
s that separates as many a A from B as possible (this
amounts to a satisfying solution for (3)-(5) that assigns
value true to as few variables da as possible). For each a
A, define a rational cost c a that is equal to 1 if da is true,
and equal to 0 otherwise. Using these costs and (3)-(5), the
desired s may be found by solving the following MINSAT
problem, with variables da, a A, and pi, qi, i = 1, 2, , n.
min c
s. t. p q i = 1, 2,..., n
((b+ b0) q ) (i(b- b0) pi) bB (6)
q d a , (a+a0)
pi da a , (a+a0)
The solution of problem (6) identifies an s and a
largest subset A = {a A da = False} that is separated
from B by s. Restricting the problem to A and using
costs c(p i) and c(qi) associated with the variables pi and qi,
we define a second objective function and solve again the
problem, obtaining a separating vector whose properties
depend on the c(pi) and c(qi). A simple example of the role
of cost values c(p i) and c(q i) is the following. Assume that
c(pi) and c(qi) assign cost of 1 when pi and qi are true and
cost 0 when they are false. The separating vector will use
the minimum number of logic variables, that is, the
minimum amount of information contained in the data to
separate the sets. On the opposite, if we assign cost 0 for
true and 1 for false, it will use the maximum amount of
information to define the separating sets. If one separating
vector is not sufficient to separate all set A, we set A A
and iterate. The disjunction of all separating vectors consti-
tutes the final logic formula that separates A from B.
The Leibniz System
The MINSAT problems considered belong to the so
called NP class, that means, shortly, that the time needed
for their solution cannot be bounded by a polynomial
function in the size of the input, as explained by the
modern theory of computational complexity (Garey &
Johnson, 1979). We solve the MINSAT instances with
the Leibniz System, a logic programming solver devel-
oped at the University of Texas at Dallas. Such solver is
based on decomposition and combinatorial optimization
results described in Truemper (1998).
Error Control and Voting System
In learning problems one views the sets A and B as
training sets, and considers them to be subsets of sets A
and B where A consists of all {0, +/-1} records of length
n with property t, and B consists of all such records without
property t. One then determines a set S that separates B
from A, and uses that set to guess whether a new vector L
r is in A or B. That is, we guess r to be in A if at least one
s S is nested in r, and to be in B otherwise. Of course,
the classification of r based on S is correct if r is in A or
B, but otherwise need not be correct. Specifically, we may
guess a record of A -A to be in B, and a record of B -B to
be in A. Usually such errors are referred to as type error
and type error respectively. The utility of S depends on
which type of error is made how many times. In some
settings, an error of one of the two types is bad, but an
error of the other type is worse. For example, a non-
invasive diagnostic system for cancer that claims a case
to be benign when a malignancy is present has failed
badly. On the other hand, prediction of a malignancy for
an actually benign case triggers additional tests, and
thus is annoying but not nearly as objectionable as an
error of the first type. We can influence the extent of type
error and type errors by an appropriate choice of the
objective function c(p i) and c(q i). As anticipated, when
c(pi) = c(qi) = 1 for all i = 1, , n the formulas determined
by the solution of the sequence of MINSAT problems
will have minimal support, that is, will try to use the
minimum number of variables to separate A from B. On
the other hand, setting c(pi) = c(q i) = -1 for all i = 1, ,
n, we will obtain the opposite effect, that is, a formula with
maximum support. If we use a single vector s to classify
a vector r, then we guess r to be in A if s is nested in r.
The latter condition tends to become less stringent when
the number of nonzero entries in s is reduced. Hence, we
heuristically guess that a solution vector s with mini-
mum support tends to avoid type errors. Conversely,
an s with maximum support tends to avoid type errors.
We apply this heuristic argument to the separating set S
produced under one of the two choices of objective
functions, and thus expect that a set S with minimum
(resp. maximum) support tends to avoid type (resp. )
errors. The above considerations are combined with a
sophisticated voting scheme embedded in Lsquare, partly
inspired to the notion of stacked generalization origi-
nally described in Wolpert (1992) (see also Breiman,
1996). Such scheme refines the separating procedure
described in the previous section by a resampling tech-
nique of the logic records available for training. For
each subsample we determine 4 separating formulas,
by switching the roles of A and B in the MINSAT
formulation and by switching the signs of the cost
coefficients. Then, each formula so determined is used
to produce a vote for each elements of A and B, the vote
being 1 (resp. 1) if the element is recognised to
belong to A (resp. B). The sum of all the votes so
obtained is the vote total V. If the vote V is positive (resp.
negative), then the record belongs to A (resp. to B). If A
and B are representative subsets of A and B, then the
695
TEAM LinG

votes V for records of A (resp. of B) tend to be positive
(resp. negative). To assess the reliability of the classifi-
cation of records of A and B, Lsquare computes two
estimated probability distributions. For any odd integer k,
the first (resp. second) distribution supplies an estimate
(resp. ) of the probability that the vote V for a record
of A (resp. B) is less (resp. greater) than k. For example, if
one declares records with votes V greater than a given k to
be in B, then is an estimate of the probability that a record
of A will be misclassified. Such threshold k can be choosen
in such a way that the total estimated error (+) is small
but also balanced in the desired direction, according to what
error is more serious for the specific application. An exten-
sive treatment of this feature and of its statistical and
mathematical background may be found in Felici, Sun, &
Truemper (1998).
FUTURE TRENDS
Data mining in logic setting presents several interesting
features, amongst which is the interpretability of its
results, that are making it a more and more appreciated
tool. The mathematical problems generated by this type
of data mining are typically very complex, require very
powerful computing machines and sophisticated solu-
tion approaches. All efforts made to design and test new
efficient algorithmic frameworks to solve mathemati-
cal problems related with data mining in logic will thus
be a rewarding topic for future research. Moreover, the
size of the available databases is increasing at a high
rate, and these methods suffer from both the computa-
tional complexity of the algorithm and from the explo-
sion of the dimensions often related with the conversion
of rational variables into Boolean ones. The solution of
data mining in logic setting in an effective and scalable
way is thus an exciting challenge. Only recently, re-
searchers have started to consider distributed and grid
computing as a potential key feature to win it.
CONCLUSION
In this chapter we have considered a particular type of
data mining and learning problem and have described
one method to solve such problems. Despite their in-
trinsic difficulty, these problems are of high interest
for their applicability and for their theoretical proper-
ties. For brevity, we have omitted many details and also
results and comparisons with other methods on real life
and test instances. Extensive information on the behav-
ior of Lsquare on many data sets coming from the Irvine
Repository (Murphy & Aha, 1994) is available in Felici &
Truemper (2002) and Felici, Sun, and Truemper (2004).
696
The Lsquare System for Mining Logic Data
REFERENCES
Boros, E. et al. (1996). An implementation of logical
analysis of data. RUTCOR Research Report 29-96. NJ:
Rutgers University.
Breiman, L. (1996). Stacked regressions. Machine Learn-
ing, 24, 49-64.
Breiman, L., Friedman J.H., Olshen R.A., & Stone, C.J.
(1984). Classification and regression trees. Wadsworth
International.
Felici, G., Sun, F-S., & Truemper, K. (2004). A method for
controlling errors in two-class classifications. IASI
Technical Report n. 980, November, 1998.
Felici, G., & Truemper, K. (2002). A Minsat approach for
learning in logic domains. INFORMS Journal on Comput-
ing, 14 (1), 20-36.
Garey, M.R., & Johnson, D.S. (1979). Computers and
intractability: A guide to the theory of NP-completeness.
San Francisco: Freeman.
Kamath, A.P., Karmarkar, N.K., Ramakrishnan, K.J., &
Resende M.G.C. (1992). A continuous approach to in-
ductive inference. Mathematical Programming, 57,
215-238.
Murphy, P.M., & Aha, D.W. (1994). UCI repository of
machine learning databases: Machine readable data
repository. Department of Computer Science, Univer-
sity of California, Irvine.
Triantaphyllou, E., Allen, L., Soyster, L., & Kumara,
S.R.T. (1994). Generating logical expressions from
positive and negative examples via a branch-and-bound
approach. Computers and Operations Research, 21,
185-197.
Triantaphyllou, E., & Soyster, L. (1996). On the mini-
mum number of logical clauses inferred from examples.
Computers and Operations Research, 21, 783-799.
Truemper, K. (1998). Effective logic computation. New
York: Wiley-Interscience.
Truemper, K. (2004). Design of logic-based intelligent
systems. New York: Wiley.
Truemper, K. (2004). The Leibniz system for logic pro-
gramming. Version 7.1 Leibniz. Plano, Texas.
Wolpert, D.H. (1992). Stacked generalization. Neural
Networks, 5, 241-259.
TEAM LinG
The Lsquare System for Mining Logic Data
KEY TERMS
Classification Error: Number of elements that are
classified in the wrong class by a classification rule. In two
class problems, the classification error is divided into the
so-called false positive and false negative.
Combinatorial Optimization: Branch of optimization in
applied mathematics and computer science related to
algorithm theory and computational complexity theory.
Error Control: Selection of a classification rule in
such a way to obtain a desired proportion of false positive
and false negative.
Logic Data Mining: The application of data mining
techniques where both data and extracted information are L
expressed by logic variables.
MINSAT: Given a Boolean expression E and a cost
function on the variables, determine, if exists, an assign-
ment to the variables in E such that E is true and the sum
of the cost is minumum.
Propositional Logic: A system of symbolic logic based
on the symbols and, or, not to stand for whole
propositions and logical connectives. It only considers
whether a proposition is true or false.
Voting: Classification method based on the combina-
tion of several different classifiers obtained by different
methods and/or different data.
697
TEAM LinG
698
Marketing Data Mining
Victor S.Y. Lo
Fidelity Personal Investments, USA
INTRODUCTION
Data mining has been widely applied over the past two
decades. In particular, marketing is an important appli-
cation area. Many companies collect large amounts of
customer data to understand their customers needs and
predict their future behavior. This article discusses
selected data mining problems in marketing and pro-
vides solutions and research opportunities.
BACKGROUND
Analytics are heavily used in two marketing areas: mar-
ket research and database marketing. The former ad-
dresses strategic marketing decisions through the analy-
sis of survey data, and the latter handles campaign deci-
sions through the analysis of behavioral and demo-
graphic data. Due to the limited sample size of a survey,
market research normally is not considered data mining.
This article focuses on database marketing, where data
mining is used extensively to maximize marketing re-
turn on investment by finding the optimal targets. A
typical application is developing response models to
identify likely campaign responders. As summarized in
Figure 1, a previous campaign provides data on the
dependent variable (responded or not), which is merged
with individual characteristics, including behavioral and
demographic variables, to form an analyzable data set. A
response model is then developed to predict the re-
sponse rate given the individual characteristics. The
model is then used to score the population and predict
response rates for all individuals. Finally, the best list of
individuals will be targeted in the next campaign in order
to maximize effectiveness and minimize expense.
Response modeling can be applied in the following
activities (Peppers & Rogers, 1997, 1999):
Figure 1. Response modeling process
Collect Develop
data model
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
(1) Acquisition: Which prospects are most likely to
become customers?
(2) Development: Which customers are most likely
to purchase additional products (cross-selling) or
to add monetary value (up-selling)?
(3) Retention: Which customers are most retainable?
This can be relationship or value retention.
MAIN THRUST
Standard database marketing problems have been de-
scribed in literature (e.g., Hughes, 1994; Jackson & Wang,
1996; and Roberts & Berger, 1999). In this article, I
describe the problems that are infrequently mentioned
in the data mining literature as well as their solutions
and research opportunities. These problems are embed-
ded in various components of the campaign process,
from campaign design to response modeling to cam-
paign optimization; see Figure 2. Each problem is de-
scribed in the Problem-Solution-Opportunity format.
CAMPAIGN DESIGN
The design of a marketing campaign is the starting point
of a campaign process. It often does not receive enough
attention in data mining. If a campaign is not designed
properly, postcampaign learning can be infeasible (e.g.,
insufficient sample size). On the other hand, if a cam-
paign is scientifically designed, learning opportunities
can be maximized.
The design process includes activities such as deter-
mining the sample size for treatment and control groups
(both have to be sufficiently large such that measure-
ment and modeling, when required, are feasible), decid-
ing on sampling methods (pure or stratified random
Score
Select best
population
targets
TEAM LinG
Marketing Data Mining
Figure 2. Database marketing campaign process
Design
Execute
sampling), and creating a cell design structure (testing
various offers and also by age, income, or other vari-
ables). I focus on the latter here.
Problem 1
Classical designs often test one variable at a time. For
example, in a cell phone direct mail campaign, you may
test a few price levels of the phone. After launching the
campaign and uncovering the price level that led to the
highest revenue, another campaign is launched to test a
monthly fee, a third campaign tests the direct mail
message, and so forth. A more efficient way is to struc-
ture the cell design such that all these variables are
testable in one campaign. Consider an example: A credit
card company would like to determine the best combi-
nation of treatments for each prospect; the treatment
attributes and attribute levels are summarized in Table 1.
The number of all possible combinations = 44 x 22 =
1024 cells, which is not practical to test.
Solution 1
To reduce the number of cells, a fractional factorial
design can be applied (full factorial refers to the design
that includes all possible combinations); see Montgom-
ery (1991) and Almquist and Wyner (2001). Two types
of fractional factorials are a) an orthogonal design
where all attributes are made orthogonal (uncorrelated)
with each other and b) an optimal design where a certain
criterion related to the variance-covariance matrix of
parameter estimates is optimized; see Kuhfeld (1997,
2004) for the applications of SAS PROC FACTEX and
PROC OPTEX in market research. (Kuhfelds market
research applications are also applicable to database
marketing).
For the preceding credit card problem, an orthogo-
nal fractional factorial design using PROC FACTEX in
Table 1. Example of design attributes and attribute levels
Attribute Attribute level
APR 5.9% (0), 7.9% (1), 10.9% (2), 12.9% (3)
Credit limit $2,000 (0), $5,000 (1), $7,000 (2), $10,000 (3)
Color Platinum (0), Titanium (1), Ruby (2), Diamond (3)
Rebate None (0), 0.5% (1), 1% (2), 1.5% (3)
Brand SuperCard (0), AdvantagePlus (1)
Creative A (0), B (1)
M
Measure Model Optimize
SAS with estimable main effects, all two-way interac-
tion effects, and quadratic effects on quantitative vari-
ables generates a design of 256 cells, which may still be
considered large. An optimal design using PROC OPTEX
with the same estimable effects generates a design of
only 37 cells (see Table 2; refer to Table 1 for attribute
level definitions).
Fractional factorial design has been used in credit
card acquisition but is not widely used in other indus-
tries for marketing. Two reasons are (a) a lack of experi-
mental design knowledge and experience and (b) busi-
ness process requirement tight coordination with list
selection and creative design professionals is required.
Opportunity 1
Mayer and Sarkissien (2003) proposed using individual
characteristics as attributes in the optimal design, where
individuals are chosen optimally. Using both indi-
vidual characteristics and treatment attributes as design
attributes is theoretically interesting. In practice, we
should compare this optimal selection of individuals with
stratified random sampling. Simulation and theoretical
and empirical studies are required to evaluate this idea.
Additionally, if many individual variables (say, hundreds)
are used in the design, then constructing an optimal
design may be very computationally intensive due to the
large design matrix and thus the design may require a
unique optimization technique to solve.
RESPONSE MODELING
Problem 2
As stated in the introduction, response modeling uses
data from a previous marketing campaign to identify
699
TEAM LinG
 Marketing Data Mining

Table 2. An optimal design example

Cell Apr Limit Rebate Color Brand Creative Cell Apr Limit Rebate Color Brand Creative
1 0 0 0 3 0 1 21 2 1 3 1 1 1
2 0 0 3 3 1 1 22 2 3 0 1 0 1
3 0 0 3 3 0 0 23 3 0 0 3 1 0
4 0 0 0 2 1 1 24 3 0 3 3 0 1
5 0 0 0 2 0 0 25 3 0 3 2 1 1
6 0 0 0 1 0 1 26 3 0 3 2 0 0
7 0 0 0 0 1 0 27 3 0 0 1 1 1
8 0 0 3 0 0 1 28 3 0 0 0 0 0
9 0 1 3 2 1 0 29 3 0 3 0 1 0
10 0 3 0 3 1 0 30 3 1 0 2 0 1
11 0 3 3 3 0 1 31 3 1 2 1 0 0
12 0 3 3 2 0 1 32 3 3 0 3 1 1
13 0 3 2 1 1 1 33 3 3 0 3 0 0
14 0 3 3 1 0 0 34 3 3 3 3 1 0
15 0 3 0 0 0 1 35 3 3 0 2 1 0
16 0 3 3 0 1 1 36 3 3 0 0 1 0
17 1 0 3 1 1 0 37 3 3 3 0 0 1
18 1 2 0 1 1 0
19 1 3 2 0 0 0
20 2 0 1 0 1 1

likely responders given individual characteristics (Berry & Solution 2

Linoff, 1997, 2000; Jackson & Wang, 1996; Rud, 2001).
Treatment and control groups were set up in the campaign
for measurement. The methodology typically uses treat-
ment data to identify characteristics of customers who are
likely to respond. In other words, for individual iW,
P(Y i = 1| Xi; treatment) = f(Xi),
where W represents all the individuals in the campaign,
Yi is the dependent variable (defined by the campaigns
call to action, indicating whether or not those people
responded), Xi is a vector of independent variables that
represent individual characteristics, and f(.) is the func-
tional form. For examples, in logistic regression, f(.) is a
logistic function of Xi (Hosmer & Lemeshow, 1989); in
decision trees, f(.) is a step function of Xi (Zhang & Singer,
1999); and in neural network, f(.) is a nonlinear function of
Xi (Haykin, 1999).
A problem may arise when we apply Model (1) to the
next campaign. Depending on the industry and product,
the model may select individuals who are likely to respond
regardless of the marketing campaign. The results of the
new campaign may show equal or similar (treatment minus
control) performance in model and random cells; see Table
3 for an example.
Table 3. Sample campaign measurement of model
effectiveness (response rates)
Treatment Control Incremental (treatment
(e.g., mail) (e.g., no mail) minus control)
Model 1% 0.8% 0.2%
Random 0.5% 0.3% 0.2%
The key issue is that fitting Model 1 and applying it to
find the best individuals is inappropriate. The correct
way is to find those who are most positively influenced
by the campaign. Doing so requires you to predict lift:
P(Y i = 1| X i; treatment) P(Y i = 1| Xi; control) for each
(1) i and then select those with the highest values of esti-
mated lift. Alternative solutions to this problem are:
(1) Fitting two separate treatment and control
models: This solution is relatively straightfor-
ward except that estimated lift can be sensitive to
statistically insignificant differences in param-
eters of the treatment and control models.
(2) Using dummy variable and interaction effects:
Lo (2002) proposed using the independent vari-
ables X i, Ti, and Xi*Ti, where Ti equals 1 if i is in
the treatment group but otherwise equals 0, and
modeling the response rate by using a logistic
regression:
exp( + 'X i + Ti + 'X i Ti )
Pi = (2)
1 + exp( + 'X i + Ti + 'X i Ti )
In Equation 2, denotes the intercept, is a vector of
parameters measuring the main effects of the indepen-
dent variables, denotes the main treatment effect, and
measures additional effects of the independent vari-
ables due to treatment. (See Russek-Cohen and Simon
(1997) for similar problems in biomedicine.) To pre-
dict the lift, use the following equation:

700

TEAM LinG
Marketing Data Mining
Pi | treatment Pi | control
exp( + + 'X i + 'X i ) exp( + 'X i )
= (
1+ exp( + + 'X i + 'X i ) 1+ exp( + 'X i )
That is, we can use the parameter estimates from
Model (2) in Equation (3) to predict the lift for each i in a
new data set. Individuals with high positive predicted lift
values will be selected for the next campaign. We can
apply a similar technique of using a dummy variable and
interactions in other supervised learning algorithms, such
as neural network. See Lo (2002) for a detailed description
of the methodology.
(3) Decision tree approach: To maximize lift, the
only known commercial software is Quadstone, based
on decision trees where each split at every parent
node is processed to maximize the difference be-
tween lift (treatment minus control response rates)
in the left and right children nodes; see Radcliffe
and Surry (1999) for their Uplift Analysis.
Opportunity 2
(1) Alternative modeling methods for this lift-based
problem should exist, and readers are encouraged to
uncover them. (2) As in typical data mining, the number
of potential predictors (independent variables) is large.
Variable reduction techniques are available for standard
problems (i.e., finding the best subset associated with
the dependent variable), but a unique variable reduction
technique for the lift-based problem is needed (i.e.,
finding the best subset associated with the lift).
Problem 3
An extension of Problem 2 is the presence of multiple
treatments. Data can be obtained from an experimen-
tally designed campaign where multiple treatment at-
tributes were tested. The question is how to incorporate
treatment attributes and individual characteristics in the
same response model.
Solution 3
An extension of Equation (2) is to incorporate both treat-
ment attributes and individual characteristics in the
same lift-based response model.
M
exp( + 'X i + Ti + 'X iTi + 'Z + ' Z X )
Pi = i i i
1 + exp( + 'X i + Ti + 'X iTi + 'Z + 'Z X )
i i i
(3) where Z i is the vector of treatment attributes, (4)
and and are additional parameters ( Z i = 0 if Ti = 0).
In practice, Equation (4) can have many variables
including many interaction effects, and thus
multicollinearity may be a concern. Developing two sepa-
rate treatment and control models may be more practical.
Also, unlike the single-treatment problem, no commercial
software is known to solve this problem directly.
Opportunity 3
Similar to Opportunity 2, alternative methods to handle
multiple treatment response modeling should exist.
OPTIMIZATION
Marketing optimization refers to delivering the best
treatment combination to the right people so as to
optimize certain criteria subject to constraints. The step
naturally follows campaign design and response model-
ing, especially when multiple treatment combinations
were tested in the previous campaign(s), and the re-
sponse model score for each treatment combination and
each individual is available (see Problem 3).
Problem 4
The main issue is that the number of decision variables
equals the number of individuals in the targets (n) times
number of treatment combinations (m). For example, in
an acquisition campaign where the target population has
30 million individuals and the number of treatment
combinations is 100. Then the number of decision vari-
ables is n times m, which equals 3 billion. Consider the
following binary integer programming problem
(Bertsimas & Tsitsiklis, 1997):
Maximize = ij xij
i j
subject to :
x ij max. # of individuals receiving treatment combination j,
i
c x
i j
ij ij expense budget, plus other relevant constraints,
xij = 0 or 1,
where ij = inc. value received by sending treatment comb. j to individual i,
cij = cost of sending treatment comb. j to individual i. (5)
701
TEAM LinG

In Model (5), ij is linked to the predicted incremental
response rate of individual i and treatment combination j.
For example, if ij is the predicted incremental response
rate (i.e., lift), the objective function is to maximize the total
number of incremental responders; if ij is incremental
revenue, it may be a constant times the incremental re-
sponse rate. Similar forms of Model 5 appear in Storey and
Cohen (2002) and Ansari and Mela (2003); here, however,
I am maximizing incremental value or responders. In the
literature, similar formulations for aggregate-level mar-
keting optimization, such as Stapleton, Hanna, and
Markussen (2003) and Larochelle and Sanso (2000), are
common, but not for individual-level optimization.
Solution 4
One way to address such a problem is through heuristics,
which do not guarantee global optimum. For example, to
reduce the size of the problem, you may group the large
number of individuals in the target population into clus-
ters and then apply linear programming to solve at the
cluster level. This is exactly the first stage outlined in
Storey and Cohen (2002). The second stage of their
approach is to optimize at the individual level for each
of the optimal clusters obtained from the first stage.
SAS has implemented this methodology in their new
software known as Marketing Optimization. For exact
solutions, consider Marketswitch at
www.marketswitch.com, where the problem is solved
with an exact solution through a mathematical transfor-
mation. The software has been applied at various compa-
nies (Leach, 2001).
Opportunity 4
(1) Simulation and empirical studies can be used to
compare existing heuristics to exact global optimiza-
tion. (2) General heuristics such as simulated annealing,
genetic algorithms, and tabu search can be attempted
(Goldberg, 1989; Michalewicz & Fogel, 2002).
Problem 5
Predictive models never produce exact results. Random
variation of predicted response rates can be estimated in
various ways. For example, in the holdout sample, the
modeler can compute the standard deviation of lift by
decile. More advanced methods, such as bootstrapping,
can also be applied to assess variability (Efron &
Tibshirani, 1998). Then how should the variability be
accounted for in the optimization stage?
702
Marketing Data Mining
Solution 5
The simplest way is to perform sensitivity analysis of
response rates on optimization. The downside is that if the
variability is high and/or the number of treatment combi-
nations is large, the large number of possibilities will make
it difficult. An alternative is to solve the stochastic ver-
sion of the optimization problem by using stochastic
programming (see Birge & Louveaux, 1997). Solving such
a large stochastic programming problem typically relies
on the Monte Carlo simulation method (Ahmed & Shapiro,
2002) and can easily become a large project.
Opportunity 5
The two alternatives in Solution 5 may not be practical
for large optimization problems. Thus, researchers have
an opportunity to develop a practical solution. You may
consider using robust optimization, which incorporates
the trade-off between optimal solution and conserva-
tism in linear programming; see Ben-Tal, Margalit, and
Nemirovski (2000) and Bertsimas and Sim (2004).
FUTURE TRENDS
(1) Experimental design is expected to be more uti-
lized to maximize learning.
(2) More marketers are now aware of the lift problem
in response modeling (see Solution 2), and alter-
native methods will be developed to address the
problem.
(3) Optimization can be performed across campaigns,
channels, customer segments, and business initia-
tives so that a more global optimal solution is
achieved. This requires not only data and response
models but also coordination and cultural shift.
CONCLUSION
Several problems with suggested solutions and opportu-
nities for research have been described in this article.
They are not frequently mentioned in standard literature
but are very valuable in marketing. The solutions involve
multiple areas, including data mining, statistics, opera-
tions research, database marketing, and marketing sci-
ence. Researchers are encouraged to evaluate current
solutions and develop new methodologies.
TEAM LinG
Marketing Data Mining
REFERENCES
Ahmed, S., & Shapiro, A. (2002). The sample average
approximation method for stochastic programs with inte-
ger recourse (Tech. Rep.). Georgia Institute of Technol-
ogy, School of Industrial & Systems Engineering.
Almquist, E., & Wyner, G. (2001, October). Boost your
marketing ROI with experimental design. Harvard Busi-
ness Review, 135-141.
Ansari, A., & Mela, C. (2003). E-customization. Journal of
Marketing Research, XL, 131-145.
Ben-Tal, A., Margalit, T., & Nemirovski, A. (2000). Robust
modeling of multi-stage portfolio problems. In H. Frenk, K.
Roos, T. Terlaky, & S. Zhang (Eds.), High performance
optimization. Kluwer.
Berry, M. J. A., & Linoff, G. S. (1997). Data mining
techniques: For marketing, sales, and customer sup-
port. Wiley.
Berry, M. J. A., & Linoff, G. S. (2000). Mastering data
mining. Wiley.
Bertsimas, D., & Sim, M. (2004). The price of robust-
ness. Operations Research, 52(1), 35-53.
Bertsimas, D., & Tsitsiklis, J. N. (1997). Introduction to
linear optimization. Athena Scientific.
Birge, J. R., & Louveaux, F. (1997). Introduction to sto-
chastic programming. Springer.
Efron, B., & Tibshirani, R. J. (1998). An introduction to the
bootstrap. CRC.
Goldberg, D. E. (1989). Genetic algorithms in search,
optimization & machine learning. Addison-Wesley.
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The
elements of statistical learning. Springer.
Haykin, S. (1999). Neural networks: A comprehensive
foundation. Prentice Hall.
Hosmer, D. W., & Lemeshow, S. (1989). Applied logis-
tic regression. Wiley.
Hughes, A. M. (1994). Strategic database marketing.
McGraw-Hill.
Jackson, R., & Wang, P. (1996). Strategic database
marketing. NTC.
Kuhfeld, W. (1997). Efficient experimental designs
using computerized searches. Sawtooth Software Re-
search Paper Series.
Kuhfeld, W. (2004). Experimental design, efficiency, cod-
ing, and choice designs. SAS Technical Note, TS-694C. M
Larochelle, J., & Sanso, B. (2000). An optimization model
for the marketing-mix problem in the banking industry.
INFOR, 38(4), 390-406.
Leach, P. (2001, August). Capital One optimizes customer
relationships. 1to1 Magazine.
Lo, V. S. Y. (2002). The true-lift model a novel data
mining approach to response modeling in database mar-
keting. SIGKDD Explorations, 4(2), 78-86.
Mayer, U. F., & Sarkissien, A. (2003). Experimental design
for solicitation campaigns. Proceedings of the Ninth ACM
SIGKDD International Conference on Knowledge Dis-
covery and Data Mining (pp. 717-722).
Michalewicz, Z., & Fogel, D. B. (2002). How to solve it:
Modern heuristics. Springer.
Montgomery, D. C. (1991). Design and analysis of
experiments (3rd ed.). Wiley.
Peppers, D., & Rogers, M. (1997). Enterprise one-to-
one. Doubleday.
Peppers, D., & Rogers, M. (1999). The one-to-one
fieldbook. Doubleday.
Radcliffe, N. J., & Surry, P. (1999). Differential re-
sponse analysis: Modeling true response by isolating
the effect of a single action. Proceedings of Credit
Scoring and Credit Control VI, Credit Research Cen-
tre, University of Edinburgh Management School.
Roberts, M. L., & Berger, P. D. (1999). Direct market-
ing management. Prentice Hall.
Rud, O. P. (2001). Data mining cookbook. Wiley.
Russek-Cohen, E., & Simon, R. M. (1997). Evaluating
treatments when a gender by treatment interaction may
exist. Statistics in Medicine, 16(4), 455-464.
Stapleton, D. M., Hanna, J. B., & Markussen, D. (2003).
Marketing strategy optimization: Using linear program-
ming to establish an optimal marketing mixture. Ameri-
can Business Review, 21(2), 54-62.
Storey, A., & Cohen, M. (2002). Exploiting response
models: Optimizing cross-sell and up-sell opportuni-
ties in banking. Proceedings of the Eighth ACM
SIGKDD International Conference on Knowledge Dis-
covery and Data Mining (pp. 325-331).
Zhang, H., & Singer, B. (1999). Recursive partitioning in
the health sciences. Springer.
703
TEAM LinG

KEY TERMS
Campaign Design: The art and science of designing
a marketing campaign, including cell structure, sam-
pling, and sizing.
Control Group: Individuals who look like those in the
treatment group but are not contacted.
Database Marketing: A branch of marketing that
applies database technology and analytics to understand
customer behavior and improve effectiveness in mar-
keting campaigns.
704
Marketing Data Mining
Fractional Factorial: A subset of the full factorial
design; that is, a subset of all possible combinations.
Marketing Optimization: Delivering the best treat-
ments to the right individuals.
Response Modeling: Predicting a response given in-
dividual characteristics by using data from a previous
campaign.
Treatment Group: Individuals who are contacted
(e.g., mailed) in a campaign.
TEAM LinG

Material Acquisitions Using Discovery
Informatics Approach
Chien-Hsing Wu
National University of Kaohsiung, Taiwan, ROC
Tzai-Zang Lee
National Cheng Kung University, Taiwan, ROC
INTRODUCTION
Material acquisition is a time-consuming but important
task for a library, because the quality of a library is not
in the number of materials that are available but in the
number of materials that are actually utilized. Its goal is
to predict the users needs for information with respect
to the materials that will most likely be used (Bloss,
1995). Discovery informatics using the technology of
knowledge discovery in databases can be used to inves-
tigate in-depth how the acquired materials are being
used and, in consequence, can be a predictive sign of
information needs.
BACKGROUND
Material searchers regularly spend large amounts of
time to acquire resources for enormous numbers of
library users. Therefore, something significant should
be relied on to produce the acquisition recommendation
list for the limited budget (Whitmire, 2002). Major
resources for material acquisitions are, in general, the
personal collections of the librarians and recommenda-
tions by users, departments, and vendors (Stevens, 1999).
The collections provided by these collectors are usually
determined by their individual preferences, rather than
by a global view, and thus may not be adequate for the
material acquisitions to rely on. Information in the
usage data may show something different from the
collectors recommendations (Hamaker, 1995). For
example, knowing which materials were most utilized
by the patrons would be highly useful for material
acquisitions.
First, circulation statistics is one of the most sig-
nificant references for library material acquisition de-
cisions (Budd & Adams, 1989; Tuten & Lones, 1995;
Pu, Lin, Chien, & Juan, 1999). It is a reliable factor by
which to evaluate the success of material utilization
(Wise & Perushek, 2000). Second, the data-mining
technique with a capability of description and predic-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
705
M
tion can explore patterns in databases that are meaningful,
interpretable, and decision supportable (Wang, 2003).
The discovery informatics in circulation databases using
induction mechanism are used in the decision of library
acquisition budget allocation (Kao, Chang, & Lin, 2003;
Wu, 2003). The circulation database is one of the impor-
tant knowledge assets for library managerial decisions.
For example, information such as 75.4% of patrons who
made use of Organizations also made use of Financial
Economics via association relationship discovery is
supportive for the material acquisition operation. Conse-
quently, the data-mining technique with an association
mechanism can be utilized to explore informatics that are
useful.
MAIN THRUST
Utilization discovery as a base of material acquisitions,
comprising a combination of association utilization and
statistics utilization, is discussed in this article. The
association utilization is derived by a data-mining tech-
nique. Systemically, when data mining is applied in the
field of material acquisitions, it follows five stages:
collecting datasets, preprocessing collected datasets,
mining preprocessed datasets, gathering discovery
informatics, interpreting and implementing discovered
informatics, and evaluating discovered informatics. The
statistics utilization is simply the sum of numeric values
of strength for all different types of categories in pre-
processed circulation data tables (Kao et al., 2003).
These need both domain experts and data miners to
accomplish the tasks successfully.
Collecting Datasets
Most libraries have employed computer information
systems to collect circulation data that mainly includes
users identifier, name, address, and department for a
user; identifier, material category code, name, author,
publisher, and publication date for a material; and users
TEAM LinG
 Material Acquisitions Using Discovery Informatics Approach
identifier, material identifier, material category code, date
borrowed, and date returned for a transaction. In order to
consider the importance of a material category, a data
table must be created to define the degree of importance
that a material presents to a department (or a group of
users). For example, five scales of degree can be abso-
lutely matching, highly matching, matching, likely transaction, materials in Organization & Management
matching, and absolutely not matching, and their were also borrowed in the same transaction with a support
importance strength can be defined as 0.4, 0.3, 0.2, 0.1, and
0.0, respectively (Kao et al., 2003).
Preprocessing Data
Preprocessing data may have operations of refinement
and reconstruction of data tables, consistency of
multityped data tables, elimination of redundant (or
unnecessary) attributes, combination of highly corre-
lated attributes, and discretization of continuous at-
tributes. Two operations in this stage for material acqui-
sitions are the elimination of unnecessary attributes and
the reconstruction of data tables. For the elimination of
unnecessary attributes, four data tables are preprocessed
to derive the material utilization. They are users tables
(two attributes: department identifier and user identi-
fier), category tables (two attributes: material identifi-
ers and material category code), circulation table (three
attributes: user identifier, material category code, and
date borrowed), and importance table (three attributes:
department identifier, material category identifier, and
importance). For the reconstruction of data tables, a
new table can be generated that contains attributes of
department identifier, user identifier, material category
code, strength, and date borrowed.
Mining Data
Mining mechanisms can perform knowledge discovery
with the form of association, classification, regression,
clustering, and summarization/generalization (Hirota &
Pedrycz, 1999). The association with a form of If
Condition Then Conclusion captures relationships be-
tween variables. The classification is to categorize a set
of data based on their values of the defined attributes.
The regression is to derive a prediction model by alter-
ing the independent variables for dependent one(s) in a
defined database. Clustering is to put together the physi-
cal or abstract objects into a class based on similar
characteristics. The summarization/generalization is
to abridge the general characteristics over a set of
defined attributes in a database.
The association informatics can be employed in
material acquisitions. Like a rule, it takes the form of
P=>Q (, ), where P and Q are material categories, and
706
and are support and confidence, respectively (Meo,
Pseila, & Ceri, 1998). The P is regarded as the condition,
and Q as the conclusion, meaning that P can produce Q
implicitly. For example, an association rule Systems
=>Organizations & Management (0.25, 0.33) means, If
materials in the category of Systems were borrowed in a
of 0.25 and a confidence of 0.33. Support is defined as the
ratio of the number of transactions observed to the total
number of transactions, whereas confidence is the ratio of
the number of transactions to the number of conditions.
Although association rules having the form of P=>Q (,
) can be generated in a transaction, the inverse associa-
tion rules and single material category in a transaction
also need to be considered.
When two categories (C1 and C2) are utilized in a
transaction, it is difficult to determine the association
among C1=>C2, C2=>C1, and both. A suggestion from
librarians is to take the third one (both) as the decision
of this problem (Wu, Lee, & Kao, 2004). This is also
supported by the study of Meo et al. (1998), which deals
with association rule generation in customer purchasing
transactions. The number of support and confidence of
C1=>C2 may be different from those of C2=>C1. As a
result, the inverse rules are considered as an extension
for the transactions that contain more than two catego-
ries to determine the number of association rules. The
number of rules can be determined via 2*[n*(n-1)/2],
where n is the number of categories in a transaction. For
example, {C1, C2, C3} are the categories of a transac-
tion, and 6 association rules are then produced to be
{C1=>C2, C1=>C3, C2=>C3, C2=>C1, C3=>C1,
C3=>C2}. Unreliable association rules may occur be-
cause their supports and confidences are too small.
Normally, there is a predefined threshold that defines
the value of support and confidence to filter the unreli-
able association rules. Only when the support and con-
fidence of a rule satisfy the defined threshold is the rule
regarded as a reliable rule. However, no evidence exists
so far is reliable determining the threshold. It mostly
depends on how reliable the management would like the
discovered rules to be. For a single category in a trans-
action, only the condition part without support and
confidence is considered, because of the computation
of support and confidence for other transactions.
Another problem is the redundant rules in a transac-
tion. It is realized that an association rule is to reveal the
company of a certain kind of material category, inde-
pendent of the number of its occurrences. Therefore, all
redundant rules are eliminated. In other words, there is
only one rule for a particular condition and only one conclu-
sion in a transaction. Also, the importance of a material to a
TEAM LinG
Material Acquisitions Using Discovery Informatics Approach
department is omitted. However, the final material utilization
will take into account this concern when the combination with
statistics utilization is performed.
Gathering Discovery Informatics
The final material utilization as the discovery informatics
contains two parts. One is statistics utilization, and the
other is association utilization (Wu et al., 2004). It is
expressed as Formula 1 for a material category C.
k made and implemented. Simplicity looks at the degree of
MatU (C ) = nC + nk * ( * support + * confidence ) (1)
i
Where
MatU(C): material utilization for category C
nC: statistics utilization
nk: statistics utilization of the kth category that can
produce C
: intensity of support
support: number of support
: intensity of confidence
confidence: number of confidence
Interpreting and Implementing
Discovery Informatics
Interpretation of discovery informatics can be performed
by any visualization techniques, such as table, figure,
graph, animation, diagram, and so forth. The main discov-
ery informatics for material acquisition have three tables
indicating statistics utilization, association rules, and
material utilization. The statistics utilization table lists
each material category and its utilization. The associa-
tion rule table has four attributes, including condition,
conclusion, supports, and confidence. Each tuple in this
table represents an association rule. The association
utilization is computed according to this table. The ma-
terial utilization table has five attributes, including ma-
terial category code, statistics utilization, association
utilization, material utilization, and percentage. In this
table, the value of the material utilization is the sum of
statistics utilization and association utilization. For each
material category, the percentage is the ratio of its
utilization to the total utilization. Implementation deals
with how to utilize the discovered informatics. The ma-
terial utilization can be used as a base of material acqui-
sitions by which informed decisions about allocating
budget is made.
Evaluating Discovered Informatics
M
Performance of the discovered informatics needs to be
tested. Criteria used can be validity, significance/unique-
ness, effectiveness, simplicity, and generality (Hirota &
Pedrycz, 1999). The validity looks at whether the discov-
ered informatics is practically applicable. Uniqueness/
significance deals with how different the discovered
informatics are to the knowledge that library manage-
ment already has. Effectiveness is to see the impact the
discovered informatics has on the decision that has been
understandability, while generality looks at the degree
of scalability. The criteria used to evaluate the discov-
ered material utilization for material acquisitions can be
in particular the uniqueness/significance and effective-
ness. The uniqueness/significance can show that mate-
rial utilization is based not only on statistics utilization,
but also on association utilization. The solution of effec-
tiveness evaluation can be found by answering the
questions Do the discovered informatics significantly
help reflect the information categories and subject areas
of materials requested by users? and Do the discov-
ered informatics significantly help enhance material uti-
lizations for next year?
FUTURE TRENDS
Digital libraries using innovative Internet technology
promise a new information service model, where li-
brary materials are digitized for users to access any-
time from anywhere. In fact, it is almost impossible for
a library to provide patrons with all the materials avail-
able because of budget limitations. Having the collec-
tions that closely match the patrons needs is a primary
goal for material acquisitions. Libraries must be cen-
tered on users and based on contents while building a
global digital library (Kranich, 1999). This results in
the increased necessity of discovery informatics tech-
nology. Advanced research tends to the integrated stud-
ies that may have requests for information for different
subjects (material categories). The material associa-
tions discovered in circulation databases may reflect
these requests.
Library management has paid increased attention to
easing access, filtering and retrieving knowledge
sources, and bringing new services onto the Web, and
users are industriously looking for their needs and
figuring out what is really good for them. Personalized
information service becomes urgent. The availability
707
TEAM LinG
 Material Acquisitions Using Discovery Informatics Approach
of accessing materials via the Internet is rapidly changing
the strategy from print to digital forms for libraries. For
example, what can be relied on while making the decision
on which electronic journals or e-books are required for a
library, how do libraries deal with the number of login
names and the number of users entering when analysis of
arrival is concerned, and how do libraries create person-
alized virtual shelves for patrons by analyzing their trans-
action profiles? Furthermore, data collection via daily
circulation operation may be greatly impacted by the way
a user makes use of the online materials and, as a conse-
quence, makes the material acquisitions operation even
more difficult. Discovery informatics technology can help
find the solutions for these issues.
CONCLUSION
Material acquisition is an important operation for library
that needs both technology and management involved.
Circulation data are more than data that keep material
usage records. Discovery informatics technology is an
active domain that is connected to data processing, ma-
chine learning, information representation, and manage-
ment, particularly when it has shown a substantial aid in
decision making. Data mining is an application-depen-
dent issue, and applications in domain will need adequate
techniques to deal with. Although discovery informatics
depends highly on the technologies used, its use with
respect to applications in domain still needs more efforts
to concurrently benefit management capability.
REFERENCES
Bloss, A. (1995). The value-added acquisitions librar-
ian: Defining our role in a time of change. Library
Acquisitions: Practice & Theory, 19(3), 321-330.
Budd, J. M., & Adams, K. (1989). Allocation formulas in
practice. Library Acquisitions: Practice & Theory, 13,
381-390.
Hamaker, C. (1995). Time series circulation data for collec-
tion development; Or, you cant intuit that. Library Ac-
quisition: Practice & Theory, 19(2), 191-195.
Hirota, K., & Pedrycz, W. (1999). Fuzzy computing for data
mining. Proceedings of the IEEE, 87(9), 1575-1600.
Kao, S. C., Chang, H. C., & Lin, C. H. (2003). Decision
support for the academic library acquisition budget allo-
cation via circulation data base mining. Information Pro-
cessing & Management, 39(1), 133-147.
708
Kranich, N. (1999). Building a global digital library. In C.-
C. Chen (Ed.), IT and global digital library development
(pp. 251-256). West Newton, MA: MicroUse Information.
Lu, H., Feng, L., & Han, J. (2000). Beyond intratransaction
association analysis: Mining multidimensional
intertransaction association rules. ACM Transaction on
Information Systems, 18(4), 423-454.
Meo, R., Psaila, G., & Ceri, S. (1998). An extension to SQL
for mining association rules. Data Mining & Knowledge
Discovery, 2, 195-224.
Pu, H. T., Lin, S. C., Chien, L. F., & Juan, Y. F. (1999).
Exploration of practical approaches to personalized
library and networked information services. In C.-C.
Chen (Ed.), IT and global digital library development
(pp. 333-343). West Newton, MA: MicroUse Information.
Stevens, P. H. (1999). Whos number one? Evaluating
acquisitions departments. Library Collections, Acqui-
sitions, & Technical Services, 23, 79-85.
Tuten, J. H., & Lones, B. (1995). Allocation formulas in
academic libraries. Chicago, IL: Association of Col-
lege and Research Libraries.
Wang, J. (2003). Data mining: Opportunities and chal-
lenges. Hershey, PA: Idea Group.
Whitmire, E. (2002). Academic library performance mea-
sures and undergraduates library use and educational
outcomes. Library & Information Science Research, 24,
107-128.
Wise, K., & Perushek, D. E. (2000). Goal programming as
a solution technique for the acquisition allocation prob-
lem. Library & Information Science Research, 22(2), 165-
183.
Wu, C. H. (2003). Data mining applied to material acquisi-
tion budget allocation for libraries: Design and develop-
ment. Expert Systems with Applications, 25(3), 401-411.
Wu, C. H., Lee, T. Z., & Kao, S. C. (2004). Knowledge
discovery applied to material acquisitions for libraries.
Information Processing & Management, 40(4), 709-725.
KEY TERMS
Association Rule: The implication of connections for
variables that are explored in databases, having a form of
AB, where A and B are disjoint subsets of a dataset of
binary attributes.
TEAM LinG
Material Acquisitions Using Discovery Informatics Approach
Circulation Database: The information of material
usages that are stored in a database, including user
identifier, material identifier, date the material is borrowed
and returned, and so forth.
Digital Library: A library that provides the resources
to select, structure, offer, access, distribute, preserve, and
maintain the integrity of the collections of digital works.
Discovery Informatics: Knowledge explored in data-
bases with the form of association, classification, regres-
sion, summarization/generalization, and clustering.
Material Acquisition: A process of material informa-
tion collection by recommendations of users, vendors, M
colleges, and so forth. Information explored in databases
can be also used. The collected information is, in general,
used in purchasing materials.
Material Category: A set of library materials with
similar subjects.
709
TEAM LinG
710
Materialized Hypertext View Maintenance
Giuseppe Sindoni
ISTAT - National Institute of Statistics, Italy
INTRODUCTION
A hypertext view is a hypertext containing data from an
underlying database. The materialization of such
hypertexts, that is, the actual storage of their pages in the
site server, is often a valid option1. Suitable auxiliary data
structures and algorithms must be designed to guarantee
consistency between the structures and contents of each
heterogeneous component where base data is stored and
those of the derived hypertext view.
This topic covers the maintenance features required
by the derived hypertext to enforce consistency between
page content and database status (Sindoni, 1998). Spe-
cifically, the general problem of maintaining hypertexts
after changes in the base data and how to incrementally
and automatically maintain the hypertext view are dis-
cussed and a solution using a Definition Language for
Web page generation and an algorithm and auxilary data
structure for automatic and incremental hypertext view
maintenance is presented.
BACKGROUND
Some additional maintenance features are required by a
materialized hypertext to enforce consistency between
page contents and the current database status. In fact,
every time a transaction is issued on the database, its
updates must be efficiently and effectively extended to
the derived hypertext. In particular, (i) updates must be
incremental, that is, only the hypertext pages dependent
on database changes must be updated and (ii) all data-
base updates must propagate to the hypertext.
The principle of incremental maintenance has been
previously explored by several authors in the context of
materialized database views (Blakeley et al., 1986; Gupta
et al., 2001; Paraboschi et al., 2003; Vista, 1998; Zhuge et
al., 1995). Paraboschi et al. (2003) give a useful overview
of the materialized view maintenance problem in the con-
text of multidimensional databases. Blakeley et al. (1986)
propose a method in which all database updates are first
filtered to remove those that cannot possibly affect the
view. For the remaining updates, they apply a differential
algorithm to re-evaluate the view expression. This ex-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
ploits the knowledge provided by both the view definition
expression and the database update operations. Gupta et
al. (2001) consider a variant of the view maintenance
problem: to keep a materialized view up-to-date when the
view definition itself changes. They try to adapt the
view in response to changes in the view definition. Vista
(1998) reports on the integration of view maintenance
policies into a database query optimizer. She presents the
design, implementation and use of a query optimizer
responsible for the generation of both maintenance ex-
pressions to be used for view maintenance and execution
plans. Zhuge et al. (1995) show that decoupling of the
base data (at the sources) from the view definition and
view maintenance machinery (at the warehouse) can lead
the warehouse to compute incorrect views. They intro-
duce an algorithm that eliminates the anomalies.
Fernandez et al. (2000), Sindoni (1998) and Labrinidis
& Roussopoulos (2000) have brought these principles to
the Web hypertext field. Fernandez et al. (2000) provide a
declarative query language for hypertext view specifica-
tion and a template language for specification of its HTML
representation. Sindoni (1998) deals with the maintenance
issues required by a derived hypertext to enforce consis-
tency between page content and database state. Hypertext
views are defined as nested oid-based views over the set
of base relations. A specific logical model is used to
describe the structure of the hypertext and a nested
relational algebra extended with an oid invention operator
is proposed, which allows views and view updates to be
defined. Labrinidis & Roussopoulos (2000) analytically
and quantitatively compare three materialization policies
(inside the DBMS, at the web server and virtual). Their
results indicate that materialization at the Web server is
a more scalable solution and can facilitate an order of
magnitude more users than the other two policies, even
under high update workloads.
The orthogonal problem of deferring maintenance
operations, thus allowing the definition of different poli-
cies, has been studied by Bunker et al. (2001), who provide
an overview of the view maintenance subsystem of a
commercial data warehouse system. They describe opti-
mizations and discuss how the systems focus on star
schemas and data warehousing influences the mainte-
nance subsystem.
TEAM LinG
Materialized Hypertext View Maintenance
MAIN THRUST
With respect to incremental view maintenance, the het-
erogeneity caused by the semistructured nature of views,
different data models and formats between base data and
derived views makes the materialized hypertext view dif-
ferent to the database view context and introduces some
new issues.
Hypertexts are normally modeled by object-like mod-
els, because of their nested and network-like structure.
Thus with respect to relational materialized views, where
base tables and views are modeled using the same logical
model, maintaining a hypertext view derived from rela-
tional base tables involves the additional challenge of
taking into account for the materialized views a different
data model to that of base tables.
In addition each page is physically stored as a marked-
up text file, possibly on a remote server. Direct access to
single values on the page is thus not permitted. Whenever
a page needs to be updated, it must therefore be com-
pletely regenerated from the new database status. Fur-
thermore, consistency between pages must be preserved,
which is an operation analogous to the one of preserving
consistency between nested objects.
The problem of dynamically maintaining consistency
between base data and derived hypertext is the hypertext
view maintenance problem. It has been addressed in the
framework of the STRUDEL project (Fernandez et al.,
2000) as the problem of incremental view updates for
semistructured data, by Sindoni (1998) and by Labrinidis
& Roussopoulos (2000).
There are a number of related issues:
different maintenance policies should be allowed
(immediate or deferred);
this implies the design of auxiliary data structures
to keep track of database updates, but their man-
agement overloads the system and they must there-
fore be as light as possible;
finally, due to the particular network structure of
a hypertext, consistency must be maintained not
only between the single pages and the database, but
also between page links.
To deal with such issues, a manipulation language for
derived hypertexts; an auxiliary data structure for (i)
representing the dependencies between the database and
hypertext and (ii) logging database updates; and an
algorithm for automatic hypertext incremental mainte-
nance can be introduced. For example, hypertext views
and view updates can be defined using a logical model and
an algebra (Sindoni, 1998).
An auxiliary data structure allows information on the
dependencies between database tables and hypertext
pages to be maintained. It may be based on the concept
of view dependency graph, for the maintenance of the M
hypertext class described by the logical model. A view
dependency graph stores information about the base
tables, which are used in the hypertext view definitions.
Finally, incremental page maintenance can be per-
formed by a maintenance algorithm that takes as its input
a set of changes on the database and produces a minimal
set of update instructions for hypertext pages. The algo-
rithm can be used whenever hypertext maintenance is
required.
A Manipulation Language for
Materialized Hypertexts
Once a derived hypertext has been designed with a logical
model and an algebra has been used to define its materi-
alization as a view on base tables, a manipulation lan-
guage is of course needed to populate the site with page
scheme instances2 and maintain them when database
tables are updated. The language is based on invocations
of algebra expressions.
The languages used for page creation and mainte-
nance may be very simple, such as that composed of only
two instructions: GENERATE and REMOVE. They allow ma-
nipulation of hypertext pages and can refer to the whole
hypertext, to all instances of a page scheme, or to pages
that satisfy a condition.
The GENERATE statement has the following general
syntax:
G ENERATE ALL | <P AGESCHEME >
[W HERE <CONDITION>]
Its semantics essentially create the proper set of
pages, taking data from the base tables as specified by the
page scheme definitions. The ALL keyword allows gen-
eration of all instances of each page scheme. The REMOVE
statement has a similar syntax and allows the specified
sets of pages to be removed from the hypertext.
Incremental Maintenance of
Materialized Hypertexts
Whenever new data are inserted in the database, the
status of all affected tables changes and the hypertexts
whose content derives from those tables no longer reflect
the databases current status. These pages must there-
fore be updated in line with the changes. An extension to
the system is thus needed to incrementally enforce con-
sistency between database and hypertext.
The simple brute force approach to the problem
would simply regenerate the whole hypertext from the new
711
TEAM LinG

database status, thus performing a huge number of unnec-
essary page creation operations. A more sophisticated
approach would regenerate only the instances of the
involved page schemes, again however unnecessarily
regenerating all page instances not actually containing
the newly inserted data. The optimal solution is hence to
regenerate only the page instances containing the newly
inserted data. To perform such an incremental mainte-
nance process, the database transaction must be extended
by a GENERATE statement; with a suitable WHERE condition
restricting the generation of pages to only those contain-
ing new data. This is done by using a list of URLs of pages
affected by the database change, which is produced by the
maintenance algorithm.
Let us now consider a deletion of data from the data-
base. In principle, this corresponds to the deletion and/or
modification of derived hypertext instances. Pages will be
deleted whenever all the information they contain has
been deleted from the database, and re-generated when-
ever only part of their information has been deleted. Dele-
tion transactions from the database are therefore extended
by either REMOVE or GENERATE statements on the derived
hypertext.
The main maintenance problems in this framework are:
(i) to produce the proper conditions for the GENERATE and
REMOVE statements automatically and (ii) to distinguish
database updates corresponding to page removals from
those corresponding to page replacements. These prob-
lems are normally solved by allowing the system to log
deletions from each base table and maintain information on
which base tables are involved in the generation of each
page scheme instance.
When the underlying database is relational, this can be
done by systems that effectively manage the sets of
inserted and deleted tuples of each base relation since the
last hypertext maintenance operation. These sets will be
addressed as + and -. were first introduced in the defining a framework for its incremental updates. In fact,
database programming language field, while Sindoni (1998)
and Labrinidis & Roussopoulos (2001) used them in the
context of hypertext views. In this framework, for a data-
base relation, a is essentially a relation with the same
scheme storing the set of inserted (+)or deleted (-)
tuples since the last maintenance operation.
A view dependency graph is also implemented, allowing
information on dependencies between base relations and
derived views to be maintained. It also maintains informa-
tion on oid attributes, that is, the attributes of each base
relation whose values are used to generate hypertext URLs.
The maintenance algorithm takes +, - and the view
dependency graph as inputs and produces as output the
sequence of GENERATE and REMOVE statements necessary
for hypertext maintenance. It allows maintenance to be
postponed and performed in batch mode, by using the table
update logs and implementing proper triggering mecha-
nisms. For more details of the algorithm see Sindoni (1998).
712
Materialized Hypertext View Maintenance
FUTURE TRENDS
There are a number of connected problems that may merit
further investigation.
Defining the most suitable maintenance policy for
each page class involves a site analysis process to
extract data on page access and, more generally, on
site quality and impact.
If a deferred policy is chosen for a given class of
pages, this causes part of the hypertext to become
temporarily inconsistent with its definition. Con-
sequently, transactions reading multiple page
schemes may not perform efficiently. The
concurrency control problem needs to be extended
to the context of hypertext views and suitable
algorithms must be developed. There is a particular
need to address the problem of avoiding dangling
links, that is, links pointing nowhere because the
former target page has been removed.
Performance analysis is needed to show how trans-
action overhead and view refresh time are affected
in the above approaches. Its results should be used
to define a set of system-tuning parameters, to be
used by site administrators for optimization pur-
poses.
CONCLUSION
We have shown that one of the most important problems
in managing information repositories containing struc-
tured and semistructured data is to provide applications
able to manage their materialization.
This approach to the definition and generation of a
database-derived Web site is particularly suitable for
the ability to formally define mappings between base
data and derived hypertext allows easy definition of
hypertext updates in the same formalism.
To enable the system to propagate database updates
to the relevant pages, a Data Manipulation Language
must be defined, which allows automatic generation or
removal of a derived site or specific subset of its pages.
The presence of a DML for pages allows an algorithm for
their automatic maintenance to be defined. This can use
a specific data structure that keeps track of view-table
dependencies and logs table updates. Its output is a set
of the Data Manipulation Language statements to be
executed in order to update the hypertext.
The algorithm may be used to implement different
view maintenance policies. It allows maintenance to be
deferred and performed in batch mode, by using the
table update logs and implementing proper triggering
mechanisms.
TEAM LinG
Materialized Hypertext View Maintenance
These techniques, like many others defined previ-
ously, are now being applied to XML data by various
researchers (Braganholo et al., 2003; Chen et al., 2002;
Alon et al., 2003; Zhang et al., 2003; Shanmugasundaram
et al., 2001). The most challenging issue for long-term
research is probably that of extending hypertext incre-
mental maintenance to the case where data come from
many heterogeneous, autonomous and distributed data-
bases.
REFERENCES
Alon, N. et al. (2003). Typechecking XML views of rela-
tional databases. ACM Transactions on Computational
Logic, 4 (3), 315-354.
Blakeley, J. et al. (1986). Efficiently updating materialized
views. In ACM SIGMOD International Conf. on Manage-
ment of Data (SIGMOD86) (pp. 61-71).
Braganholo, V. P. et al. (2003). On the updatability of XML
views over relational databases. In WebDB (pp. 31-36).
Bunker, C.J. et al. (2001). Aggregate maintenance for data
warehousing in Informix Red Brick Vista. In VLDB 2001
(pp. 659-662).
Chen, Y.B. et al. (2000). Designing valid XML views. In
Entity Relationship Conference (pp. 463-478).
Fernandez, M.F. et al. (2000). Declarative specification of
Web sites with Strudel. The VLDB Journal, 9(1), 38-55.
Gupta, A. et al. (2001). Adapting materialized views after
redefinitions: Techniques and a performance study. In-
formation Systems, 26 (5), 323-362.
Labrinidis, A., & Roussopoulos, N. (2000). WebView
materialization. In SIGMOD00 (pp. 367-378).
Labrinidis, A., & Roussopoulos, N. (2001). Update propa-
gation strategies for improving the quality of data on the
Web. In VLDB (pp. 391-400).
Paraboschi, S. et al. (2003). Materialized views in multidi-
mensional databases. In Multidimensional databases
(pp. 222-251). Hershey, PA: Idea Group Publishing.
Shanmugasundaram, J. et al. (2001). Querying XML views
of relational data. In VLDB (pp. 261-270).
Sindoni, G. (1998). Incremental maintenance of hypertext
views. In Proceedings of the Workshop on the Web and
Databases (WebDB98) (in conjunction with EDBT98).
LNCS 1590 (pp. 98-117). Berlin: Springer-Verlag.
Vista, D. (1998). Integration of incremental view mainte-
nance into query optimizers. In EDBT (pp. 374-388). M
Zhang, X. et al. (2003). Rainbow: Multi-XQuery optimiza-
tion using materialized XML views. SIGMOD Conference
(pp. 671).
Zhuge, Y. et al. (1995). View maintenance in a warehousing
environment. In SIGMOD Conference (pp. 316-327).
KEY TERMS
Database Status: The structure and content of a
database at a given time stamp. It comprises the database
object classes, their relationships and their object in-
stances.
Deferred Maintenance: The policy of not performing
database maintenance operations when their need be-
comes evident, but postponing them to a later moment.
Dynamic Web Pages: Virtual pages dynamically
constructed after a client request. The request is usually
managed by a specific program or described using a
specific query language whose statements are embed-
ded into pages.
Immediate Maintenance: The policy of perform-
ing database maintenance operations as soon as their
need becomes evident.
Link Consistency: The ability of a hypertext net-
work links to always point to an existing and semanti-
cally coherent target.
Materialized Hypertext: A hypertext dynamically
generated from an underlying database and physically
stored as a marked-up text file.
Semistructured Data: Data with a structure not as
rigid, regular, or complete as that required by traditional
database management systems.
ENDNOTES
1
For more details, see the paper Materialized
Hypertext Views.
2
A page scheme is essentially the abstract repre-
sentation of pages with the same structure and a
page scheme instance is a page with the structure
described by the page scheme. For the definition
of page scheme and instance, see paper Material-
ized Hypertext Views.
713
TEAM LinG
714
Materialized Hypertext Views
Giuseppe Sindoni
ISTAT - National Institute of Statistics, Italy
INTRODUCTION
A materialized hypertext view can be defined as a hypertext
containing data coming from a database and whose pages
are stored in files (Sindoni, 1999). A Web site presenting
data from a data warehouse is an example of such a view.
Even if the most popular approach to the generation of
such sites is based on dynamic Web pages, a rationale for
the materialized approach has produced many research
efforts. The topic will cover logical models to describe the
structure of the hypertext.
BACKGROUND
Hypertext documents in the Web are in essence collec-
tions of HTML (HyperText Markup Language) or XML
(the eXtensible Markup Language) files and are delivered
to users by an HTTP (HyperText Transfer Protocol) server.
Hypertexts are very often used to publish very large
amounts of data on the Web, in what are known as data
intensive Web sites. These sites are characterized by a
large number of pages sharing the same structure, such as
in a University Web site, where there are numerous pages
containing staff information, that is, Name, Position,
Department, and so on. Each staff page is different, but
they all share the types of information and their logical
organization. A group of pages sharing the same structure
is called page class. Similarly to databases, where it is
possible to distinguish between the intensional (data-
base structure) and extensional (the database records)
levels, in data intensive Web sites it is possible to distin-
guish between the site structure (the structure of the
different page classes and page links) and site pages
(instances of page classes).
Pages of a data intensive site may be generated dy-
namically, that is, on demand, or be materialized, as will
be clarified in the following. In both approaches, each
page corresponds to an HTML file and the published data
normally come from a database, where they can be up-
dated more efficiently than in the hypertext files them-
selves. The database is queried to extract records relevant
to the hypertext being generated and page instances are
filled with values according to a suitable hypertext model
describing page classes (Agosti et al., 1995; Aguilera et
al., 2002; Beeri et al., 1998; Crestani & Melucci, 2003;
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Baresi et al., 2000; Balasubramanian et al., 2001; Merialdo
et al., 2003; Rossi & Schwabe, 2002; Simeon & Cluet, 1998).
The hypertext can then be accessed by any Internet-
enabled machine running a Web browser.
In such a framework, hypertext can be regarded as
database views, but in contrast with classic databases,
such as relational databases, the model describing the
view cannot be the same as the one describing the data-
base storing the published data.
The most relevant hypertext logical models proposed
so far can be classified into three major groups, according
to the purpose of the hypertext being modeled. Some
approaches are aimed at building hypertext as integration
views of distributed data sources (Aguilera et al., 2002;
Beeri et al., 1998; Simeon & Cluet, 1998), other as views of
an underlying local database (Fernandez et al., 2000;
Merialdo et al., 2003). There are also proposals for models
and methods to build hypertext independently from the
source of the data that they publish (Agosti et al., 1995;
Baresi et al., 2000; Balasubramanian et al., 2001; Rossi &
Schwabe, 2002; Simeon & Cluet, 1998).
Models can be based on graphs (Fernandez et al.,
2000; Simeon & Cluet, 1998), on XML Data Type Defini-
tions (Aguilera et al., 2002), extension of the Entity Rela-
tionship model (Balasubramanian et al., 2001), logic rules
(Beeri et al., 1998) or object-like paradigms (Merialdo et al.,
2003; Rossi & Schwabe, 2002).
MAIN THRUST
The most common way to automatically generate a de-
rived hypertext is based principally on the dynamic con-
struction of virtual pages following a client request.
Usually, the request is managed by a specific program (for
example a Common Gateway Interface CGI called as a
link in HTML files) or described using a specific query
language, whose statements are embedded into pages.
These pages are often called pull pages, because it is up
to the client browser to pull out the interesting informa-
tion. Unfortunately, this approach has some major draw-
backs:
it involves a degree of Data Base Management
System overloading, because every time a page is
TEAM LinG
Materialized Hypertext Views
requested by a client browser, a query is issued to
the database in order to extract the relevant data;
it introduces some platform-dependence, because
the embedded queries are usually written in a pro-
prietary language and the CGIs must be compiled on
the specific platform;
it hampers site mirroring, because if the site needs
to be moved to another server, either the database
needs to be replicated, or some network overload is
introduced due to remote queries;
it doesnt allow the publication of some site metadata,
more specifically information about the structure of
the site, which may be very useful to querying
applications.
An alternative approach is based on the concept of
materialized hypertext view: a derived hypertext whose
pages are actually stored by the system on a server or
directly on the client machine, using a mark-up language
like HTML. This approach overcomes the above disad-
vantages because: (i) pages are static, so the HTTP
server can work on its own; (ii) there is no need to embed
queries or script calls in the pages, as standard sites are
generated; (iii) due to their standardization, sites can be
mirrored more easily, as they are not tied to a specific
technology; and finally, (iv) metadata can be published
by either embedding them into HTML comments or di-
rectly generating XML files.
A data model, preferably object-oriented, is used to
describe a Web hypertext. This allows the system to
manage nested objects by decomposition. This means
that each hypertext page is seen as an object with at-
tributes, which can be atomic or complex, such as a list of
values. Complex attributes are also modeled as nested
objects into the page object. These objects can also have
both atomic and complex attributes (objects) and the
nesting mechanism is virtually unlimited.
Below we will describe the Araneus Data Model (ADM)
(Merialdo et al., 2003), as an example of a hypertext data
model. Different models can be found in (Fernandez et al.,
2000; Fraternali & Paolini, 1998).
ADM is a page-oriented model, as page is the main
concept. Each hypertext page is seen as an object having
an identifier (its Uniform Resource Locator - URL) and a
number of attributes. Its structure is abstracted by its
page scheme and each page is an instance of a page
scheme. The notion of page scheme may be compared to
that of relation scheme, in the relational data model, or
object class, in object oriented databases. The following
example describes the page of an author in a bibliographic
site, described by the AUTHOR PAGE page scheme.
PAGE SCHEME AuthorPage
Name : TEXT;
WorkList: LIST OF
M
(Authors: TEXT;
Title: TEXT;
Reference: TEXT;
Year : TEXT;
ToRefPage: LINK TO ConferencePage
UNION JournalPage;
AuthorList:LIST OF
(Name: TEXT;
ToAuthorPage: LINK TO AuthorPage OPTIONAL;););
END PAGE SCHEME
Each AUTHOR PAGE instance has a simple attribute
(NAME). Pages can also have complex attributes: lists,
possibly nested at an arbitrary level, and links to other
pages. The example shows the page scheme with a list
attribute (WORKLIST). Its elements are tuples, formed by
three simple attributes (AUTHORS, TITLE and YEAR), a link
to an instance of either a C ONFERENCE P AGE or a
J OURNALPAGE and the corresponding anchor (REFERENCE),
and a nested list (AUTHORLIST) of other authors of the same
work.
Once a description of the hypertext is available, its
materialization is made possible using a mapping lan-
guage, such as those described in Merialdo et al., 2003).
FUTURE TRENDS
One of the topics currently attracting the interest of many
researchers and practitioners of the Web and databases
fields is XML. Most efforts are aimed at modeling XML
repositories and defining query languages for querying
and transforming XML sources (World Wide Web Con-
sortium, 2004).
One of the current research directions is to explore
XML as both a syntax for metadata publishing and a
document model to be queried and restructured. Mecca,
Merialdo, & Atzeni (1999) show that XML modeling
primitives may be considered as a subset of the Object
Data Management Group standard enriched with union
types and XML repositories may in principle be queried
using a language like the Object Query Language.
CONCLUSION
Data-intensive hypertext can be published by assuming
that the pages contain data coming from an underlying
database and that their logical structure is described
715
TEAM LinG

according to a specific model. Pages may be mapped on the
database and automatically generated using a program-
ming language.
To allow external applications to access these metadata,
a materialized approach to page generation can be adopted.
The massive diffusion of XML as a preferred means for
describing a Web pages structure and publishing it on the
Internet is facilitating integrated access to heterogeneous,
distributed data sources: the Web is rapidly becoming a
repository of global knowledge. The research challenge
for the 21st century will probably be to provide global users
with applications to efficiently and effectively find the
required information. This could be achieved by utilizing
models, methods and tools which have been already devel-
oped for knowledge discovery and data warehousing in
more controlled and local environments.
REFERENCES
Agosti, M. et al. (1995). Automatic authoring and con-
struction of hypertext for information retrieval. Multime-
dia Systems, 3, 15-24.
Aguilera, V. et al. (2002). Views in a large-scale XML
repository. Very Large DataBase Journal, 11(3), 238-255.
Balasubramanian, V. et al. (2001). A case study in system-
atic hypertext design. Information Systems, 26(4), 295-320.
Baresi, L. et al. (2000). From Web sites to Web applications:
New issues for conceptual modeling. In Entity Relation-
ship (Workshops) (pp. 89-100).
Beeri, C. et al. (1998). WebSuite: A tools suite for harness-
ing Web data. In Proceedings of the Workshop on the Web
and Databases (Web and DataBases 98) (in conjunction
with Extending DataBase Technology 98). Lecture Notes
in Computer Science (Vol. 1590) (pp. 152-171).
Crestani, F., & Melucci, M. (2003). Automatic construction
of hypertexts for self-referencing: The hyper-text book
project. Information Systems, 28(7), 769-790.
Fernandez, M. F. et al. (2000). Declarative specification of
Web sites with Strudel. Very Large DataBase Journal,
9(1), 38-55.
Fraternali, P., & Paolini, P. (1998). A conceptual model and
a tool environment for developing more scalable, dynamic,
and customizable Web applications. In VI Intl. Conference
on Extending Database Technology (EDBT 98) (pp. 421-
435).
716
Materialized Hypertext Views
Mecca, G. et al. (1999). Araneus in the Era of XML. IEEE
Data Engineering Bulletin, 22(3), 19-26.
Merialdo, P. et al. (2003). Design and development of
data-intensive Web sites: The araneus approach. ACM
Transactions on Internet Technology, 3(1), 49-92.
Rossi, G., & Schwabe, D. (2002). Object-oriented design
structures in Web application models. Annals of Soft-
ware Engineering, 13(1-4), 97-110.
Simeon, G., & Cluet, S. (1998). Using YAT to build a Web
server. In Proceedings of the Workshop on the Web and
Databases (Web and DataBases 98) (in conjunction
with Extending DataBase Technology 98). Lecture Notes
in Computer Science (Vol. 1590) (pp. 118-135).
Sindoni, G. (1999). Maintenance of data and metadata in
Web-based information systems. PhD Thesis. Universit
degli studi di Roma La Sapienza.
World Wide Web Consortium. (2004). XML Query
(XQuery). Retrieved August 23, 2004, from http://
www.w3.org/XML/Query
KEY TERMS
Dynamic Web Pages: Virtual pages dynamically con-
structed after a client request. Usually, the request is
managed by a specific program or is described using a
specific query language whose statements are embed-
ded into pages.
HTML: The Hypertext Markup Language. A lan-
guage based on labels to describe the structure and
layout of a hypertext.
HTTP: The HyperText Transaction Protocol. An
Internet protocol, used to implement communication
between a Web client, which requests a file, and a Web
server, which delivers it.
Knowledge Management: The practice of transform-
ing the intellectual assets of an organization into busi-
ness value.
Materialized Hypertext View: A hypertext contain-
ing data coming from a database and whose pages are
stored in files.
Metadata: Data about data. Structured information
describing the nature and meaning of a set of data.
XML: The eXtensible Markup Language. An evolu-
tion of HTML, aimed at separating the description of the
hypertext structure from that of its layout.
TEAM LinG

Materialized View Selection for Data Warehouse M
Design
Dimitri Theodoratos
New Jersey Institute of Technology, USA
Alkis Simitsis
National Technical University of Athens, Greece
INTRODUCTION
A data warehouse (DW) is a repository of information
retrieved from multiple, possibly heterogeneous, autono-
mous, distributed databases and other information sources
for the purpose of complex querying, analysis, and deci-
sion support. Data in the DW are selectively collected
from the sources, processed in order to resolve inconsis-
tencies, and integrated in advance (at design time) before
data loading. DW data are usually organized multi-dimen-
sionally to support online analytical processing (OLAP).
A DW can be seen abstractly as a set of materialized views
defined over the source relations. During the initial design
of a DW, the DW designer faces the problem of deciding
which views to materialize in the DW. This problem has
been addressed in the literature for different classes of
queries and views, and with different design goals.
BACKGROUND
Figure 1 shows a simplified DW architecture. The DW
contains a set of materialized views. The users address
their queries to the DW. The materialized views are used
partially or completely for the evaluation of the user
queries. This is achieved through partial or complete
rewritings of the queries using the materialized views.
Figure 1. A simplified DW architecture
queries answers
Data
Warehouse
Maintenance
Expressions
Data
Sources
...
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
717
When the source relations change, the materialized
views need to be updated. The materialized views are
usually maintained using an incremental strategy. In such
a strategy, the changes to the source relations are propa-
gated to the DW. The changes to the materialized views
are computed using the changes of the source relations
and are eventually applied to the materialized views. The
expressions used to compute the view changes involve
the changes of the source relations and are called main-
tenance expressions. Maintenance expressions are is-
sued by the DW against the data sources, and the answers
are sent back to the DW. When the source relation
changes affect more than one materialized view, multiple
maintenance expressions need to be evaluated. The tech-
niques of multi-query optimization can be used to detect
common subexpressions among maintenance expressions
in order to derive an efficient global evaluation plan for all
the maintenance expressions.
MAIN THRUST
When selecting views to materialize in a DW, one attempts
to satisfy one or more design goals. A design goal is either
the minimization of a cost function or a constraint. A
constraint can be classified as user-oriented or system-
oriented. Attempting to satisfy the constraints can result
in no feasible solution to the view selection problem. The
design goals determine the design of the algorithms that
select views to materialize from the space of alternative
view sets.
Minimization of Cost Functions
Most approaches comprise in their design goals the
minimization of a cost function.
Query Evaluation Cost: Often, the queries that the
DW has to satisfy are given as input to the view
selection problem. The overall query evaluation
cost is the sum of the cost of evaluating each input
TEAM LinG
 Materialized View Selection for Data Warehouse Design
query rewritten (partially or completely) over the
materialized views. This sum also can be weighted,
each weight indicating the frequency or importance
of the corresponding query. Several approaches
aim at minimizing the query evaluation cost (Gupta
& Mumick, 1999; Harinarayan et al., 1996; Shukla et
al., 1998).
View Maintenance Cost: The view maintenance
cost is the sum of the cost of propagating each
source relation change to the materialized views.
This sum can be weighted, each weight indicating
the frequency of propagation of the changes of the
corresponding source relation. The maintenance
expressions can be evaluated more efficiently if
they can be partially rewritten over views already
materialized at the DW; the evaluation of parts of the
maintenance expression is avoided since their ma-
terializations are present at the DW. Moreover,
access of the remote data sources and expensive
data transmissions are reduced. Materialized views
that are added to the DW for reducing the view
maintenance cost are called auxiliary views (Ross
et al., 1996; Theodoratos & Sellis, 1999). Obviously,
maintaining the auxiliary views incurs additional
maintenance cost. However, if this cost is less than
the reduction to the maintenance cost of the initially
materialized views, it is worth keeping the auxiliary
views in the DW. Ross, et al. (1996) derive auxiliary
views to materialize in order to minimize the view
maintenance cost.
Operational Cost: Minimizing the query evaluation
cost and the view maintenance cost are conflicting
requirements. Low view maintenance cost can be
obtained by replicating source relations at the DW.
In this case, though, the query evaluation cost is
high, since queries need to be computed from the
replicas of the source relations. Low query evalua-
tion cost can be obtained by materializing at the DW
all the input queries. In this case, all the input
queries can be answered by a simple lookup, but the
view maintenance cost is high, since complex main-
tenance expressions over the source relations need
to be computed. The input queries may overlap; that
is, they may share many common subexpressions.
By materializing common subexpressions and other
views over the source relations, it is possible, in
general, to reduce the view maintenance cost. These
savings must be balanced against higher query
evaluation cost. For this reason, one can choose to
minimize a linear combination of the query evalua-
tion and view maintenance cost, which is called
operational cost. Most approaches endeavor to
minimize the operational cost (Baralis et al., 1997; Gupta,
1997; Theodoratos & Sellis, 1999; Yang et al., 1997).
718
System-Oriented Constraints
System-oriented constraints are dictated by the restric-
tions of the system and are transparent to the users.
Space Constraint: Although the degradation of the
cost of disk space allows for massive storage of
data, one cannot consider that the disk space is
unlimited. The space constraint restricts the space
occupied by the selected materialized views not to
exceed the space allocated to the DW for this end.
Space constraints are adopted in many works (Gupta,
1997; Golfarelli & Rizzi, 2000; Harinarayan et al.,
1996, Theodoratos & Sellis, 1999).
View Maintenance Cost Constraint: In many prac-
tical cases, the refraining factor in materializing all
the views in the DW is not the space constraint but
the view maintenance cost. Usually, DWs are up-
dated periodically (e.g., at nighttime) in a large batch
update transaction. Therefore, the update window
must be sufficiently short so that the DW is avail-
able for querying and analysis during the daytime.
The view maintenance cost constraint states that
the total view maintenance cost should be less than
a given amount of view maintenance time. Gupta and
Mumick (1999), Golfareli and Rizzi (2000), and Lee
and Hammer (2001) consider a view maintenance
cost constraint in selecting materialized views.
Self Maintainability: A materialized view is self-
maintainable if it can be maintained for any instance
of the source relations over which it is defined and
for all source relation changes, using only these
changes, the view definition, and the view material-
ization. The notion is extended to a set of views in
a straightforward manner. By adding auxiliary views
to a set of materialized views, one can make the
whole view set self-maintainable. There are differ-
ent reasons for making a view set self-maintainable:
(a) the remote source relations need not be con-
tacted in turn for evaluating maintenance expres-
sions during view updating; (b) anomalies due to
concurrent changes are eliminated, and the view
maintenance process is simplified; (c) the material-
ized views can be maintained efficiently even if the
sources are not able to answer queries (e.g., legacy
systems), or if they are temporarily unavailable
(e.g., in mobile systems). By adding auxiliary views
to a set of materialized views, the whole view set can
be made self-maintainable. Self-maintainability can
be trivially achieved by replicating at the DW all the
source relations used in the view definitions. Self-
maintainability viewed as a constraint requires that
the set of materialized views taken together is self-
maintainable. Quass, et al., (1996), Akinde, et al.,
TEAM LinG
Materialized View Selection for Data Warehouse Design
(1998), Liang, et al., (1999), and Theodoratos (2000)
aim at making the DW self-maintainable.
Answering the Input Queries Using Exclusively the
Materialized Views: This constraint requires the
existence of a complete rewriting of the input que-
ries, initially defined over the source relations, over
the materialized views. Clearly, if this constraint is
satisfied, the remote data sources need not be con-
tacted for evaluating queries. This way, expensive
data transmissions from the DW to the sources, and
conversely, are avoided. Some approaches assume
a centralized DW environment, where the source
relations are present at the DW site. In this case, the
answerability of the queries from the materialized
views is trivially guaranteed by the presence of the
source relations. The answerability of the queries
also can be trivially guaranteed by appropriately
defining select-project views on the source relations
and replicating them at the DW. This approach as-
sures also the self-maintainability of the materialized
views. Theodoratos and Sellis (1999) do not assume
a centralized DW environment or replication of part
of the source relations at the DW and explicitly
impose this constraint in selecting views for materi-
alization.
User-Oriented Constraints
User-oriented constraints express requirements of the
users.
Answer Data Currency Constraints: An answer
data currency constraint sets an upper bound on the
time elapsed between the point in time the answer to
a query is returned to the user and the point in time
the most recent changes of a source relation that are
taken into account in the computation of this answer
are read (this time reflects the currency of answer
data). Currency constraints are associated with ev-
ery source relation in the definition of every input
query. The upper bound in an answer data currency
constraint (minimal currency required) is set by the
users according to their needs. This formalization of
data currency constraints allows stating currency
constraints at the query level and not at the materi-
alized view level, as is the case in some approaches.
Therefore, currency constraints can be exploited by
DW view selection algorithms, where the queries are
the input, while the materialized views are the output
(and, therefore, are not available). Furthermore, it
allows stating different currency constraints for dif-
ferent relations in the same query.
Query Response Time Constraints: A query re-
sponse time constraint states that the time needed to
evaluate an input query using the views material-
ized at the DW should not exceed a given bound. M
The bound for each query is given by the users and
reflects their needs for fast answers. For some
queries, fast answers may be required, while for
others, the response time may not be predominant.
Search Space and Algorithms
Solving the problem of selecting views for materializa-
tion involves addressing two main tasks: (a) generating
a search space of alternative view sets for materialization
and (b) designing optimization algorithms that select an
optimal or near-optimal view set from the search space.
A DW is usually organized according to a star schema
where a fact table is surrounded by a number of dimen-
sion tables. The dimension tables define hierarchies of
aggregation levels. Typical OLAP queries involve star
joins (key/foreign key joins between the fact table and
the dimension tables) and grouping and aggregation at
different levels of granularity. For queries of this type,
the search space can be formed in an elegant way as a
multidimensional lattice (Baralis et al., 1997; Harinarayan
et al., 1996).
Gupta (1997) states that the view selection problem is
NP-hard. Most of the approaches on view selection
problems avoid exhaustive algorithms. The adopted al-
gorithms fall into two categories: deterministic and ran-
domized. In the first category belong greedy algorithms
with performance guarantee (Gupta, 1997; Harinarayan
et al., 1996), 0-1 integer programming algorithms (Yang et
al., 1997), A* algorithms (Gupta & Mumick, 1999), and
various other heuristic algorithms (Baralis et al., 1997;
Ross et al., 1996; Shukla et al., 1998; Theodoratos &
Sellis, 1999). In the second category belong simulated
annealing algorithms (Kalnis et al., 2002; Theodoratos et
al., 2001), iterative improvement algorithms (Kalnis et al.,
2002) and genetic algorithms (Lee & Hammer, 2001). Both
categories of algorithms exploit the particularities of the
specific view selection problem and the restrictions of
the class of queries considered.
FUTURE TRENDS
The view selection problem has been addressed for
different types of queries. Research has focused mainly
on queries over star schemas. Newer applications (e.g.,
XML or Web-based applications) require different types
of queries. This topic has only been partially investi-
gated (Golfarelli et al., 2001; Labrinidis & Roussopoulos,
2000).
A relevant issue that needs further investigation is
the construction of the search space of alternative view
719
TEAM LinG
 Materialized View Selection for Data Warehouse Design
sets for materialization. Even though the construction of
such a search space for grouping and aggregation queries
is straightforward (Harinarayan et al., 1966), it becomes an
intricate problem for general queries (Golfarelli & Rizzi, 2001).
Indexes can be seen as special types of views. Gupta,
et al. (1997) show that a two-step process that divides the
space available for materialization and picks views first
and then indexes can perform very poorly. More work
needs to be done on the problem of automating the
selection of views and indexes together.
DWs are dynamic entities that evolve continuously
over time. As time passes, new queries need to be satis-
fied. A dynamic version of the view selection problem
chooses additional views for materialization and avoids
the design of the DW from scratch (Theodoratos & Sellis,
2000). A system that dynamically materializes views in the
DW at multiple levels of granularity in order to match the
workload (Kotidis & Roussopoulos, 2001) is a current
trend in the design of a DW.
CONCLUSION
A DW can be seen as a set of materialized views. A central
problem in the design of a DW is the selection of views to
materialize in it. Depending on the requirements of the
prospective users of the DW, the materialized view selec-
tion problem can be formulated with various design goals
that comprise the minimization of cost functions and the
satisfaction of user- and system-oriented constraints.
Because of its importance, different versions of it have
been the focus of attention of many researchers in recent
years. Papers in the literature deal mainly with the issue
of determining a search space of alternative view sets for
materialization and with the issue of designing optimiza-
tion algorithms that avoid examining exhaustively the
usually huge search space. Some results of this research
have been used already in commercial database manage-
ment systems (Agrawal et al., 2000).
REFERENCES
Agrawal, S., Chaudhuri, S., & Narasayya, V.R. (2000).
Automated selection of materialized views and indexes in
SQL databases. International Conference on Very Large
Data Bases (VLDB), Cairo, Egypt.
Akinde, M.O., Jensen, O.G., & Bhlen, H.M. (1998). Mini-
mizing detail data in data warehouses. International
Conference on Extending Database Technology (EDBT),
Valencia, Spain.
720
Baralis, E., Paraboschi, S., & Teniente, E. (1997). Materi-
alized views selection in a multidimensional database.
International Conference on Very Large Data Bases,
Athens, Greece.
Golfarelli, M., & Rizzi, S. (2000). View materialization for
nested GPSJ queries. International Workshop on Design
and Management of Data Warehouses (DMDW),
Stockholm, Sweden.
Golfarelli, M., Rizzi, S., & Vrdoljak B. (2001). Data ware-
house design from XML sources. ACM International
Workshop on Data Warehousing and OLAP (DOLAP),
Atlanta, Georgia.
Gupta, H. (1997). Selection of views to materialize in a data
warehouse. International Conference on Database
Theory (ICDT), Delphi, Greece.
Gupta, H., Harinarayan, V., Rajaraman, A., & Ullman, J.D.
(1997). Index selection for OLAP. IEEE International
Conference on Data Engineering, Birmingham, UK.
Gupta, H., & Mumick, I.S. (1999). Selection of views to
materialize under a maintenance cost constraint. Interna-
tional Conference on Database Theory (ICDT), Jerusa-
lem, Israel.
Harinarayan, V., Rajaraman, A., & Ullman, J. (1996). Imple-
menting data cubes efficiently. ACM SIGMOD Interna-
tional Conference on Management of Data (SIGMOD),
Montreal, Canada.
Kalnis, P., Mamoulis, N., & Papadias, D. (2002). View
selection using randomized search. Data & Knowledge
Engineering, 42(1), 89-111.
Kotidis, Y., & Roussopoulos, N. (2001). A case for dy-
namic view management. ACM Transactions on Data-
base Systems, 26(4), 388-423.
Labrinidis, A., & Roussopoulos, N. (2000). WebView
materialization. ACM SIGMOD International Conference
on Management of Data (SIGMOD), Dallas, Texas.
Lee, M., & Hammer, J. (2001). Speeding up materialized
view selection in data warehouses using a randomized
algorithm. International Journal of Cooperative Infor-
mation Systems (IJCIS), 10(3), 327-353.
Liang, W. (1999). Making multiple views self-maintainable
in a data warehouse. Data & Knowledge Engineering,
30(2), 121-134.
Quass, D., Gupta, A., Mumick, I.S., & Widom, J. (1996).
Making views self-maintainable for data warehousing.
International Conference on Parallel and Distributed
Information Systems (PDIS), Florida Beach, Florida.
TEAM LinG
Materialized View Selection for Data Warehouse Design
Ross, K., Srivastava, D., & Sudarshan, S. (1996). Materi-
alized view maintenance and integrity constraint check-
ing: Trading space for time. ACM SIGMOD International
Conference on Management of Data (SIGMOD), Montreal,
Canada.
Shukla, A., Deshpande, P., & Naughton, J. (1998). Mate-
rialized view selection for multidimensional datasets. In-
ternational Conference on Very Large Data Bases
(VLDB), New York.
Theodoratos, D. (2000). Complex view selection for data
warehouse self-maintainability. International Conference
on Cooperative Information Systems (CoopIS), Eilat,
Israel.
Theodoratos, D., Dalamagas, T., Simitsis, A., &
Stavropoulos, M. (2001). A randomized approach for the
incremental design of an evolving data warehouse. Inter-
national Conference on Conceptual Modeling (ER),
Yokohama, Japan.
Theodoratos, D., & Sellis, T. (1999). Designing data ware-
houses. Data & Knowledge Engineering, 31(3), 279-301.
Theodoratos, D., & Sellis, T. (2000). Incremental design of
a data warehouse. Journal of Intelligent Information
Systems (JIIS), 15(1), 7-27.
Yang, J., Karlapalem, K., & Li, Q. (1997). Algorithms for
materialized view design in data warehousing environ-
ment. International Conference on Very Large Data
Bases, Athens, Greece.
KEY TERMS
M
Auxiliary View: A view materialized in the DW exclu-
sively for reducing the view maintenance cost.
Materialized View: A view whose answer is stored in
the DW.
Operational Cost: A linear combination of the query
evaluation and view maintenance cost.
Query Evaluation Cost: The sum of the cost of evalu-
ating each input query rewritten over the materialized
views.
Self-Maintainable View: A materialized view that can
be maintained, for any instance of the source relations,
and for all source relation changes, using only these
changes, the view definition, and the view materialization.
View: A named query.
View Maintenance Cost: The sum of the cost of
propagating each source relation change to the material-
ized views.
721
TEAM LinG
722
Methods for Choosing Clusters in Phylogenetic
Trees
Tom Burr
Los Alamos National Laboratory, USA
INTRODUCTION
One data mining activity is cluster analysis, of which there
are several types. One type deserving special attention is
clustering that arises due to evolutionary relationships
among organisms. Genetic data is often used to infer
evolutionary relations among a collection of species,
viruses, bacterial, or other taxonomic units (taxa). A phy-
logenetic tree (Figure 1, top) is a visual representation of
either the true or the estimated branching order of the taxa,
depending on the context. Because the taxa often cluster
in agreement with auxiliary information, such as geo-
graphic or temporal isolation, a common activity associ-
ated with tree estimation is to infer the number of clusters
and cluster memberships, which is also a common goal in
most applications of cluster analysis. However, tree esti-
mation is unique because of the types of data used and the
use of probabilistic evolutionary models which lead to
computationally demanding optimization problems. Fur-
thermore, novel methods to choose the number of clusters
and cluster memberships have been developed and will be
described here. The methods include a unique application
of model-based clustering, a maximum likelihood plus
bootstrap method, and a Bayesian method based on
obtaining samples from the posterior probability distribu-
tion on the space of possible branching orders.
BACKGROUND
Tree estimation is frequently applied to genetic data of
various types; we focus here on applications involving
DNA data, such as that from HIV. Trees are intended to
convey information about the genealogy of such viruses
and the most genetically similar viruses are most likely to
be most related. However, because the evolutionary pro-
cess includes random effects, there is no guarantee that
closer in genetic distance implies closer in time, for
every pair of sequences.
Sometimes the cluster analysis must be applied to
large numbers of taxa, or applied repeatedly to the same
number of taxa. For example, Burr, Myers, and Hyman
(2001) recently investigated how many subtypes (clus-
ters) arise under a simple model of how the env (gp120)
region of HIV-1, group M sequences (Figure 1, top) are
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
evolving. One question was whether the subtypes of
group M could be explained by the past population
dynamics of the virus. For each of many simulated data
sets each having approximately 100 taxa, model-based
clustering was applied to automate the process of choos-
ing the number of clusters. The novel application of
model-based clustering and its potential for scaling to
large numbers of taxa will be described, along with the two
methods mentioned in the introduction section.
It is well-known that cluster analysis results can de-
pend strongly on the metric. There are at least three
unique metric-related features of DNA data. First, the
DNA data is categorical. Second, a favorable trend in
phylogenetic analysis of DNA data is to choose the
evolutionary model using goodness of fit or likelihood
ratio tests (Huelsenbeck & Rannala, 1997). For nearly all
of the currently used evolutionary models, there is an
associated distance measure. Therefore, there is the po-
tential to make an objective metric choice. Third, the
evolutionary model is likely to depend on the region of the
genome. DNA regions that code for amino acids are more
constrained over time due to selective pressure and there-
fore are expected to have a smaller rate of change than
non-coding sequences.
A common evolutionary model is as follows (readers
who are uninterested in the mathematical detail should
skip this paragraph). Consider a pair of taxa denoted x and
y. Define Fxy as
n AA n AC n AG n AT
nCA nCC nCG nCT ,
NFxy =
nGA nGC nGG nGT
nTA nTC nTG nTT
where N is the number of base pairs (sites) in set of aligned
sequences, nAA is the number of sites with taxa x and y
both having an A, n AC is the number of sites with taxa x
having an A and taxa y having a C, etc. The most general
time-reversible model (GTR) for which a distance measure
has been defined (Swofford, Olsen, Waddell, & Hillis,
1996) defines the distance between taxa x and y as d xy =
-trace{ log( -1Fxy)} where is a diagonal matrix of the
average base frequencies in taxa x and y and the trace is
TEAM LinG
Methods for Choosing Clusters in Phylogenetic Trees

the sum of diagonal elements. The GTR is fully specified follows that Pij(t) = 0.25 + 0.75e-t and that the distance
by 5 relative rate parameters (a, b, c, d, e) and 3 relative between taxa x and y is 3/4 log(1 - 4/3D) where D is the M
frequency parameters (A, C, and G with T determined percentage of sites where x and y differ (regardless of what
via A+ C + G + T = 1) in the rate matrix Q defined as kind of difference because all relative substitution rates
and base frequencies are assumed to be equal). Important
generalizations include allowing unequal relative frequen-
a C b G c T cies and/or rate parameters), and to allow the rate to vary

a A d G e T , across DNA sites. Allowing to vary across sites via a
Q/ =
a A d C f G gamma-distributed rate parameter is one way to model the
fact that sites often have different observed rates. If the rate
a A e C f G
is assumed to follow a gamma distribution with shape
parameter then these gamma distances can be obtained
where is the overall substitution rate. The rate matrix Q from the original distances by replacing the function log(x)
is related to the substitution probability matrix P via Pij(t)= with (1-x-1/) in the dxy = -trace{P log(P-1Fxy)} formula
e Qt, where P ij(t) is the probability of a change from (Swofford et al. 1996). Generally, this rate heterogeneity
nucleotide i to j in time t and P ij(t) satisfies the time and the fact that multiple substitutions at the same site
reversibility and stationarity criteria: iP ij = jP ji. Com- tend to saturate any distance measure make it a practical
monly used models such as Jukes-Cantor (Swofford et challenge to find a metric such that the distance between
al. 1996) assumes that a = b = c = d = e = 1 and A= any two taxa increases linearly with time.
C = G= T= 0.25. For the Jukes-Cantor model, it

Figure 1. HIV Data (env region). (Top) Hierarchical Clustering; (Middle) Principle Coordinate plot; (Bottom)
Results of model-based clustering under six different assumptions regarding volume (V), shape (S), and orientation
(O). E denotes equal among clusters and V denotes varying among clusters, for V, S, and O respectively. For
example, case 6 has varying V, equal S, and varying O among clusters. Models 1 and 2 each assume a spherical shape
(I denotes the identify matrix, so S and O are equal among clusters, while V is equal for case 1 and varying for case
2 ). Note that the B and D subtypes tend to be merged.
0.4
0.2

B
GG

F
F
AA
A

FF
BB
D D
GG

AA

D
G E

AA
A

D D

BC
E A
A

BB

FF
G G
G

EE

AA

B
B
D D

D D
0.0

B
A

B
B
D

C C

F
E
G

E
E
E
E
E
E

F
F
EE

BB
D

CC
C
C
C
D

C
G

B
E
E

D
D
D

C
C
C
C
C
C

EEE E
E E
EE EEEE EE E
A D BB B B
AA
A AAAA A A B B BDB D DD
G G A A BBD
B B DD
D D D B
BDD
B
0.0

G GA A
GG G GG
A
F
x2

G F F F
FFFF
F
-0.2

CC CC
CC
C C CC
C
C

-0.2 -0.1 0.0 0.1 0.2

x1
1400

1
2
3 1
2
3 1
2
3 1
3 1 1
3 1
2 6
5 5
6 3 3 1
3 1
3 1 1 1
6
4 4 5
6 5 5 3 3 3 1
3 1
3 1
4
6 3
5 5 15 EI 5 3
2
5 1
4
6 2 VI 5 5
2 3 5 5
1000

3 EEE 5
BIC

5 5
3 5
4
6 4 VVV 5
2 1
3 5 EEV
1 6 VEV
600

1
2
3
4
5
6 1

5 10 15 20
number of clusters

723

TEAM LinG

MAIN THRUST
Much more can be said about evolutionary models, but
the background should suffice to convey the notion that
relationships among taxa determine the probability (albeit
in a complicated way via the substitution probability
matrix P) of observing various DNA character states
among the taxa. Therefore, it is feasible to turn things
around and estimate taxa relationships on the basis of
observed DNA. The quality of the estimate depends on
the adequacy of the model, the number of observed DNA
sites, the number of taxa, and the complexity of the true
tree. These important issues are discussed for example in
Swofford et al. (1996) and nearly all published methods
now are related to varying degrees to the fundamental
result of Felsenstein (1981) for calculating the likelihood
of a given set of character states for a given tree. Likeli-
hood-based estimation is typically computationally de-
manding and rarely (only for small number of taxa, say 15
or less) is the likelihood evaluated for all possible branch-
ing orders. Instead, various search strategies are used
that begin with a subset of the taxa. More recently, branch
rearrangement strategies are also employed that allow
exploration of the space of likelihoods (Li, Pearl, & Doss,
2000; Simon & Larget, 1998) and approximate probabilities
via Markov Chain Monte Carlo (MCMC, see below) of
each of the most likely trees. Finding the tree that opti-
mizes some criteria such as the likelihood is a large topic
that is largely outside our scope here (see Salter, 2000).
Our focus is on choosing groups of taxa because
knowledge of group structure adds considerably to our
understanding of how the taxa is evolving (Korber et al.
2000) and perhaps also leads to efficient ways to estimate
trees containing a large number of taxa (Burr, Skourikhine,
Macken & Bruno, 1999). Here we focus on choosing
groups of taxa without concern for the intended use of the
estimated group structure, and consider three strategies.
One option for clustering taxa is to construct trees that
have maximum likelihood (not necessarily a true maximum
likelihood because of incomplete searches used), identify
groups, and then repeat using resampled data sets. The
resampled (bootstrap) data sets are obtained from the
original data sets by sampling DNA sites with replace-
ment. This resampling captures some of the variation
inherent in the evolutionary process. If, for example, 999
of 1000 bootsrapped ML trees each show a particular
group being monophyletic, (coalesce to a common ances-
tor before any taxa from outside the group) then the
consensus is that this group is strongly supported
(Efron, Halloran, & Holmes, 1996).
A second option is to evaluate the probability of each
of the most likely branching orders. This is computationally
demanding and relies heavily on efficient branch rear-
rangement methods (Li, Pearl, & Doss, 2000) to implement
724
Methods for Choosing Clusters in Phylogenetic Trees
MCMC as a way to evaluate the likelihood of many
different branching orders. In MCMC, the likelihood ratio
of branching orders (and branch lengths) is used to
generate candidate branching orders according to their
relative probabilities; and therefore, to evaluate the rela-
tive probability of many different branching orders.
A third option (Figure 1, middle) is to represent the
DNA character data using: (a) the substitution probabil-
ity matrix P to define distances between each pair of taxa
(described above), (b) multi-dimensional scaling to re-
duce the data dimension and represent each taxa in two to
four or five dimensions in such a way that the pairwise
distances are closely approximated by distances com-
puted using the low-dimensional representation; and (c)
model-based clustering of the low-dimensional data. Sev-
eral clustering methods could be applied to the data if the
data (A, C, T, and G) were coded so that distances could
be computed. An effective way to do this is to represent
the pairwise distance data via multidimensional scaling.
Multidimensional scaling represents the data in new co-
ordinates such that distances computed in the new coor-
dinates very closely approximate the original distances
computed using the chosen metric. For n taxa with an n-
by-n distance matrix, the result of multidimensional scal-
ing is an n-by-p matrix (p coordinates) that can be used to
closely approximate the original distances. Therefore,
multidimensional scaling provides a type of data com-
pression with the new coordinates being suitable for
input to clustering methods. For example, one could use
the cmdscale function in S-PLUS (2003) to implement
classical multidimensional scaling. The implementation of
model-based clustering we consider includes kmeans as
a special case as we will describe.
In model-based clustering, it is assumed that the data
are generated by a mixture of probability distributions in
which each component of the mixture represents a cluster.
Given n p-dimensional observations x = (x1,x2,,xn), as-
sume there are G clusters and let fk(xi|k) be the probability
density for cluster k. The model for the composite of
clusters is typically formulated in one of two ways. The
classification likelihood approach maximizes
LC(1,G;1,,n | x) = i fi(xi | i),
where the i are discrete labels satisfying i = k if xi belongs
to cluster k. The mixture likelihood approach maximizes
LM(1,G;1,,G | x) = i k kfk(xi | I),
where k is the probability that an observation belongs to
cluster k.
Fraley and Raftery (1999) describe their latest version
of model-based clustering where the f k are assumed to be
multivariate Gaussian with mean k and covariance matrix
TEAM LinG
Methods for Choosing Clusters in Phylogenetic Trees
k. Banfield and Raftery (1993) developed a model-based
framework by parameterizing the covariance matrix in
terms of its eigenvalue decomposition in the form
k = k D k A k D k T , where D k is the orthonormal matrix of
eigenvectors, Ak is a diagonal matrix with elements propor-
tional to the eigenvalues of k and k is a scalar, and under
one convention is the largest eigenvalue of k. The
orientation of cluster k is determined by Dk, Ak determines
the shape, while k specifies the volume. Each of the
volume, shape, and orientation (VSO) can be variable
among groups, or fixed at one value for all groups. One
advantage of the mixture-model approach is that it allows
the use of approximate Bayes factors to compare models,
giving a means of selecting the model parameterization
(which of V, S, and O are variable among groups) and the
number of clusters (Figure 1, bottom). The Bayes factor is
the posterior odds for one model against another model
assuming that neither model is favored a priori (uniform
prior). When the EM algorithm (estimation-maximum like-
lihood, Dempster, Laird, & Rubin, 1977) is used to find the
maximum mixture likelihood, the most reliable approxima-
tion to twice the log Bayes factor (called the Bayesian
Information Criterion, BIC) is BIC = 2lM ( x,) mM log( n) ,
where lM ( x,) is the maximized mixture loglikelihood for
the model and mM is the number of independent param-
eters to be estimated in the model. A convention for
calibrating BIC differences is that differences less than 2
correspond to weak evidence, differences between 2 and
6 are positive evidence, differences between 6 and 10 are
strong evidence, and differences more than 10 are very
strong evidence.
The two clustering methods ML + bootstrap and
model-based clustering have been compared on the
same data sets (Burr, Gattiker, & LaBerge, 2002b) and the
differences were small. For example, ML + bootstrap
suggested 7 clusters for 95 HIV env sequences and 6
clusters in HIV gag (p17) sequences while model-based
clustering suggests 6 for env (tends to merge the so-
called B and D subtypes see Figure 1) and 6 for gag. Note
from Figure 1(top) that only 7 of the 10 recognized sub-
types were included among the 95 sequences. However,
it is likely that case-specific features will determine the
extent of difference between the methods. Also, model-
based clustering provides a more natural and automatic
way to identify candidate groups. Once these candidate
groups have been identified, then either method is rea-
sonable for assigning confidence measures to the result-
ing cluster assignments. The two clustering methods
ML + bootstrap and an MCMC-based estimate of the
posterior probability on the space of branching orders
have also been compared on the same data sets (Burr,
Doak, Gattiker, & Stanbro, 2002a) with respect to the
confidence that each method assigns to particular groups
that were chosen in advance of the analysis. The plot of
the MCMC-based estimate versus the ML + bootstrap M
based estimate was consistent with the hypothesis that
both methods assign (on average) the same probability to
a chosen group, unless the number of DNA sites was very
small, in which case there can be a non-negligible bias in
the ML method, resulting in bias in the ML + bootstrap
results. Because the group was chosen in advance, the
method for choosing groups was not fully tested, so there
is a need for additional research.
FUTURE TRENDS
The recognized subtypes of HIV-1 were identified using
informal observations of tree shapes followed by ML +
bootstrap (Korber & Myers, 1992). Although identifying
such groups is common in phylogenetic trees, there have
been only a few attempts to formally evaluate clustering
methods for the underlying genetic data. ML + boot-
strap remains the standard way to assign confidence to
hypothesized groups and the group structure is usually
hypothesized either by using auxiliary information (geo-
graphic, temporal, or other) or by visual inspection of
trees (which often display distinct groups). A more thor-
ough evaluation could be performed using realistic simu-
lated data with known branching orders. Having known
branching order is almost the same as having known
groups; however, choosing the number of groups is likely
to involve arbitrary decisions even when the true branch-
ing order is known.
CONCLUSION
One reason to cluster taxa is that evolutionary processes
can sometimes be revealed once the group structure is
recognized. Another reason is that phylogenetic trees are
complicated objects that can often be effectively summa-
rized by identifying the major groups together with a
description of the typical between and within group
variation. Also, if we correctly choose the number of
clusters present in the tree for a large number of taxa (100
or more), then we can then use these groups to rapidly
construct a good approximation to the true tree very
rapidly. One strategy for doing this is to repeatedly apply
model-based clustering to relatively small numbers of taxa
(100 or fewer) and check for consistent indications for the
number of groups. We described two other clustering
strategies (ML + bootstrap, and MCMC-based) and
note that two studies have made limited comparisons of
these three methods on the same genetic data.
725
TEAM LinG

REFERENCES
Banfield, J., & Raftery, A. (1993). Model-based gaussian
and non-gaussian clustering. Biometrics, 49, 803-821.
Burr, T., Charlton, W., & Stanbro, W. (2000). Comparison
of signature pattern analysis methods in molecular epide-
miology. Proc. Mathematical and Engineering Methods
in Medicine and Biological Sciences, 1 (pp. 473-479).
Burr, T., Doak, J., Gattiker, J., & Stanbro, W. (2002a).
Assessing confidence in phylogenetic trees: Bootstrap
versus Markov Chain Monte Carlo. Mathematical and
Engineering Methods in Medicine and Biological Sci-
ences, 1, 181-187.
Burr, T., Gattiker, J., & LaBerge, G. (2002b). Genetic
subtyping using cluster analysis. Special Interest Group
on Knowledge Discovery and Data Mining Explora-
tions, 3, 33-42.
Burr, T., Myers, G., & Hyman, J. (2001). The origin of AIDS
Darwinian or Lamarkian? Phil. Trans. R. Soc. Lond. B.,
356, 877-887.
Burr, T., Skourikhine, A.N., Macken, C., & Bruno, W.
(1999). Confidence measures for evolutionary trees: Ap-
plications to molecular epidemiology. Proc. of the 1999
IEEE Inter. Conference on Information, Intelligence and
Systems (pp. 107-114).
Dempster, A., Laird, N., & Rubin, D. (1977). Maximum
likelihood for incomplete data via the EM algorithm.
Journal of the Royal Statistical Society, Series B, 39,1-
38.
Efron, B., Halloran, E., & Holmes, S. (1996). Bootstrap
confi-dence levels for phylogenetic trees. Proc. Natl.
Acad. Sci. USA, 93, 13429.
Felsenstein, J. (1981). Evolutionary trees from DNA se-
quences: A maximum likelihood approach. Journal of
Molecular Evolution, 17, 368-376.
Fraley, C., & Raftery, A. (1999). MCLUST: Software for
model-based cluster analysis. Journal of Classification,
16, 297-306.
Huelsenbeck, J., & Rannala, B. (1997). Phylogenetic meth-
ods come of age: testing hypotheses in an evolutionary
context. Science, 276, 227-232.
Korber, B., Muldoon, N., Theiler, J., Gao, R., Gupta, R.
Lapedes, A, Hahn, B., Wolinsky, W., & Bhattacharya, T.
(2000). Timing the ancestor of the HIV-1 pandemic Strains.
Science, 288, 1788-1796.
726
Methods for Choosing Clusters in Phylogenetic Trees
Korber, B., & Myers, G. (1992). Signature pattern analysis:
A method for assessing viral sequence Relatedness. AIDS
Research and Human Retroviruses, 8, 1549-1560.
Li, S., Pearl, D., & Doss, H. (2000). Phylogenetic tree
construction using Markov Chain Monte Carlo. Journal
of the American Statistical Association, 95(450), 493-
508.
Salter, L. (2000). Algorithms for phylogenetic tree recon-
struction. Proc. Mathematical and Engineering Meth-
ods in Medicine and Biological Sciences, 2 (pp. 459-465).
S-PLUS - Statical Programming Lanugage. (2003). Insight-
ful Corp, Seattle, Washington.
Swofford, D.L., Olsen, G.J., Waddell, P.J., & Hillis, D.M.
(1996). Phylogenetic inference in molecular systematics
(2nd ed.) (pp. 407-514). (Hillis et al., Eds.) Sunderland,
Massachusetts: Sinauer Associates.
KEY TERMS
Bayesian Information Criterion: An approximation
to the Bayes Factor which can be used to estimate the
Bayesian posterior probability of a specified model.
Bootstrap: A resampling scheme in which surrogate
data is generated by resampling the original data or
sampling from a model that was fit to the original data.
Coalesce: In the context of phylogenetic trees, two
lineages coalesce at the time that they most recently share
a common ancestor (and hence, come together in the
tree).
Estimation-Maximization Algorithm: An algorithm
for computing maximum likelihood estimates from incom-
plete data. In the case of fitting mixtures, the group labels
are the missing data.
HIV: Human-Immunodeficiency Virus.
Markov Chain Monte Carlo: A stochastic method to
approximate probabilities, available in many situations for
which analytical methods are not available. The method
involves generating observations from the probability
distribution by evaluating the likelihood ratio of any two
candidate solutions.
Mixture of Distributions: A combination of two or
more distributions in which observations are generated
from distribution i with probability pi and (pi =1).
Model-Based Clustering: A clustering method with
relatively flexible assumptions regarding the volume,
shape, and orientation of each cluster.
TEAM LinG
Methods for Choosing Clusters in Phylogenetic Trees
Phylogenetic Tree: A representation of the branching
order and branch lengths of a collection of taxa, which, in
its most common display form, looks like the branches of
a tree.
Probability Density Function: A function that can be
summed (for discrete-valued random variables) or inte-
grated (for interval-valued random variables) to give the
probability of observing values in a specified set. M
Substitution Probability Matrix: A matrix whose i, j
entry is the probability of substituting DNA character j (C,
G, T or A) for character i over a specified time period.
727
TEAM LinG
728
Microarray Data Mining
Li M. Fu
University of Florida, USA
INTRODUCTION
Based on the concept of simultaneously studying the
expression of a large number of genes, a DNA microarray
is a chip on which numerous probes are placed for
hybridization with a tissue sample. Biological complex-
ity encoded by a deluge of microarray data is being
translated into all sorts of computational, statistical, or
mathematical problems bearing on biological issues
ranging from genetic control to signal transduction to
metabolism. Microarray data mining is aimed to iden-
tify biologically significant genes and find patterns that
reveal molecular network dynamics for reconstruction
of genetic regulatory networks and pertinent metabolic
pathways.
BACKGROUND
The idea of microarray-based assays seemed to emerge
as early as of the 1980s (Ekins & Chu, 1999). In that
period, a computer-based scanning and image-process-
ing system was developed to quantify the expression
level in tissue samples of each cloned complementary
DNA sequences spotted in a two-dimensional array on
strips of nitrocellulose, which could be the first proto-
type of the DNA microarray. The microarray-based
gene expression technology was actively pursued in the
mid-1990s (Schena, Heller, & Theriault, 1998) and has
seen rapid growth since then.
Microarray technology has catalyzed the develop-
ment of the field known as functional genomics by
offering high-throughput analysis of the functions of
genes on a genomic scale (Schena et al., 1998). There
are many important applications of this technology,
including elucidation of the genetic basis for health and
disease, discovery of biomarkers of therapeutic re-
sponse, identification and validation of new molecular
targets and modes of action, and so on. The accomplish-
ment of decoding human genome sequence together
with recent advances in the biochip technology has
ushered in genomics-based medical therapeutics, diag-
nostics, and prognostics.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
MAIN THRUST
The laboratory information management system (LIMS)
keeps track of and manages data produced from each
step in a microarray experiment, such as hybridization,
scanning, and image processing. As microarray experi-
ments generate a vast amount of data, the efficient
storage and use of the data require a database manage-
ment system. Although some databases are designed to
be data archives only, other databases such as ArrayDB
(Ermolaeva, Rastogi, & Pruitt, 1998) and Argus
(Comander, Weber, Gimbrone, & Garcia-Cardena,
2001) allow information storage, query, and retrieval,
as well as data processing, analysis, and visualization.
These databases also provide a means to link microarray
data to other bioinformatics databases (e.g., NCBI Entrez
systems, Unigene, KEGG, and OMIM). The integration
with external information is instrumental to the inter-
pretation of patterns recognized in the gene-expression
data. To facilitate the development of microarray data-
bases and analysis tools, there is a need to establish a
standard for recording and reporting microarray gene
expression data. The MIAME (Minimum Information
about Microarray Experiments) standard includes a de-
scription of experimental design, array design, samples,
hybridization, measurements, and normalization con-
trols (Brazma, Hingamp, & Quackenbush, 2001).
Data Mining Objectives
Data mining addresses the question of how to discover
a gold mine from historical or experimental data, par-
ticularly in a large database. The goal of data mining and
knowledge discovery algorithms is to extract implicit
and previously unknown nontrivial patterns, regulari-
ties, or knowledge from large data sets that can be used
to improve strategic planning and decision making. The
discovered knowledge capturing the relations among
the variables of interest can be formulated as a function
for making prediction and classification or as a model
for understanding the problem in a given domain. In the
context of microarray data, the objectives are identify-
ing significant genes and finding gene expression pat-
TEAM LinG
Microarray Data Mining
terns associated with known or unknown categories.
Microarray data mining is an important topic in
bioinformatics, dealing with information processing on
biological data, particularly genomic data.
Practical Factors Prior to Data Mining
Some practical factors should be taken into account
prior to microarray data mining. First of all, microarray
data produced by different platforms vary in their for-
mats and may need to be processed differently. For
example, one type of microarray with cDNA as probes
produces ratio data from two channel outputs, whereas
another type of microarray using oligonucleotide probes
generates nonratio data from a single channel. Not only
may different platforms pick up gene expression activ-
ity with different levels of sensitivity and specificity,
but also different data processing techniques may be
required for different data formats.
Normalizing data to allow direct array-to-array com-
parison is a critical issue in array data analysis, because
several variables in microarray experiments can affect
measured mRNA levels (Schadt, Li, Ellis, & Wong,
2001; Yang, Dudoit, & Luu, 2002). Variations may
occur during sample handling, slide preparation, hybrid-
ization, or image analysis. Normalization is essential
for correct microarray data interpretation. In simple
ways, data can be normalized by dividing or subtracting
expression values by a representative value (e.g., mean
or median in an array) or by taking a linear transforma-
tion to zero mean and unit variance. As an example, data
normalization in the case of cDNA arrays may proceed
as follows: The local background intensity is subtracted
from the value of each spot on the array; the two chan-
nels are normalized against the median values on that
array; and the Cy5/Cy3 fluorescence ratios and log 10-
transformed ratios are calculated from the normalized
values. In addition, genes that do not change signifi-
cantly can be removed through a filter in a process
called data filtration.
Differential Gene Expression
To identify genes differentially expressed across two
conditions is one of the most important issues in
microarray data mining. In cancer research, for ex-
ample, we wish to understand what genes are abnormally
expressed in a certain type of cancer, so we conduct a
microarray experiment and collect the gene expression
profiles of normal and cancer tissues, respectively, as
the control and test samples. The information regarding
differential expression is derived from comparing the
test against the control sample.
To determine which genes are differentially expressed,
a common approach is based on fold-change; in this M
approach, we simply decide a fold-change threshold (e.g.,
2C) and select genes associated with changes greater
than that threshold. If a cDNA microarray is used, the ratio
of the test over control expression in a single array can be
converted easily to fold change in both cases of up-
regulation (induction) and down-regulation (suppres-
sion). For oligonucleotide chips, fold-change is com-
puted from two arrays, one for test and the other for
control sample. In this case, if multiple samples in each
condition are available, the statistical t-test or Wilcoxon
tests can be applied, but the catch is that the Bonferroni
adjustment to the level of significance on hypothesis
testing would be necessary to account for the presence of
multiple genes. The t-test determines the difference in
mean expression values between two conditions and
identifies genes with significant difference. The nonpara-
metric Wilcoxon test is a good alternative in the case of
non-Gaussian data distribution. SAM (Significance Analy-
sis of Microarrays) (Tusher, Tibshirani, & Chu, 2001) is a
state-of-the-art technique based on balanced perturba-
tion of repeated measurements and minimization of the
false discovery rate.
Coordinated Gene Expression
Identifying genes that are co-expressed across multiple
conditions is an issue with significant implications in
microarray data mining. For example, given gene ex-
pression profiles measured over time, we are interested
in knowing what genes are functionally related. The
answer to this question also leads us to deduce the
functions of unknown genes from their correlation with
genes of known functions. Equally important is the
problem of organizing samples based on their gene
expression profiles so that distinct phenotypes or dis-
ease processes may be recognized or discovered.
The solutions to both problems are based on so-
called cluster analysis, which is meant to group objects
into clusters according to their similarity. For example,
genes are clustered by their expression values across
multiple conditions; samples are clustered by their ex-
pression values across genes. The issue is the question
of how to measure the similarity between objects. Two
popular measures are the Euclidean distance and
Pearsons correlation coefficient. Clustering algorithms
can be divided into hierarchical and nonhierarchical
(partitional). Hierarchical clustering is either
agglomerative (starting with singletons and progres-
sively merging) or divisive (starting with a single clus-
ter and progressively breaking). Hierarchical
agglomerative clustering is most commonly used in the
729
TEAM LinG
 Microarray Data Mining

cluster analysis of microarray data. In this method, two
most similar clusters are merged at each stage until all
the objects are included in a single cluster. The result is
a dendrogram (a hierarchical tree) that encodes the
relationships among objects by showing how clusters
merge at each stage. Partitional clustering algorithms
are best exemplified by k-means and self-organization
maps (SOMs).
Gene Selection for Discriminant
Analysis
Taking an action based on the category of the pattern
recognized in microarray gene expression data is an
increasingly important approach to medical diagnosis
and management (Furey, Cristianini, & Duffy, 2000;
Golub, Slonim, & Tamayo, 1999; Khan, Wei, & Ringner,
2001). A class predictor derived on this basis can auto-
matically discover the distinction between different
classes of samples, independent of previous biological
knowledge (Golub et al., 1999). Gene expression infor-
mation appears to be a more reliable indicator than
phenotypic information for categorizing the underlying
causes of diseases. The microarray approach has offered
hope for clinicians to arrive at more objective and accu-
rate cancer diagnoses and hence choose more appropri-
ate forms of treatment (Tibshirani, Hastie, Narasimhan,
& Chu, 2002).
The central question is how to construct a reliable
classifier that predicts the class of a sample on the basis
of its gene expression profile. This is a pattern recogni-
tion problem, and the type of analysis involved is re-
ferred to as discriminant analysis. In practice, given a
limited number of samples, correct discriminant analy-
sis must rely on the use of an effective gene selection
technique to reduce the gene number and, hence, the
data dimensionality. The objective of gene selection is
to select genes that most contribute to classification as
well as provide biological insight. Approaches to gene
selection range from statistical analysis (Golub et al.,
1999) and a Bayesian model (Lee, Sha, Dougherty,
Vannucci, & Mallick, 2003) to Fishers linear dis-
criminant analysis (Xiong, Li, Zhao, Jin, & Boerwinkle,
2001) and support vector machines (SVMs) (Guyon,
Weston, Barnhill, & Vapnik, 2002). This is one of the
most challenging areas in microarray data mining. De-
spite good progress, the reliability of selected genes
should be further improved. Table 1 summarizes some
of most important microarray data-mining problems
and their solutions.
Microarray Data-Mining Applications
Microarray technology permits a large-scale analysis
of gene functions in a genomic perspective and has
brought about important changes in how we conduct
basic research and practice clinical medicine. There
have existed an increasing number of applications with
this technology. Here, the role of data mining in discov-
ering biological and clinical knowledge from microarray
data is examined.
Consider that only the minority of all the yeast (Sac-
charomyces cerevisiae) open reading frames in the ge-
nome sequence could be functionally annotated on the

Table 1. Three common computational problems in microarray data mining

Problem 1: To identify differentially expressed genes, given microarray gene expression

data collected in two conditions, types, or states.

[Solutions:]
Fold change
t-test or Wilcoxon rank sum test (with Bonferronis correction)
Significance analysis of microarrays

Problem 2: To identify genes expressed in a coordinated manner, given microarray gene

expression data collected across a set of conditions or time points.

[Solutions:]
Hierarchical clustering
Self-organization
k-means clustering

Problem 3: To select genes for discriminant analysis, given microarray gene expression data
of two or more classes.

[Solutions:]
Neighborhood analysis
Support vector machines
Principal component analysis
Bayesian analysis
Fishers linear discriminant analysis

730

TEAM LinG
Microarray Data Mining

basis of sequence information alone (Zweiger, 1999),
although microarray results showed that nearly 90% of all
yeast mRNAs (messenger RNAs) are observed to be
present (Wodicka, Dong, Mittmann, Ho, & Lockhart,
1997). Functional annotation of a newly discovered gene
based on sequence comparison with other known gene
sequences is sometimes misleading. Microarray-based
genome-wide gene expression analysis has made it pos-
sible to deduce the functions of novel or poorly charac-
terized genes from co-expression with already known
genes (Eisen, Spellman, Brown, & Botstein, 1998). The
microarray technology is a valuable tool for measuring
whole-genome mRNA and enables system-level explora-
tion of transcriptional regulatory networks (Cho, Campbell,
& Winzeler, 1998; DeRisi, Iyer, & Brown, 1997; Laub,
McAdams, Feldblyum, Fraser, & Shapiro, 2000; Tavazoie,
Hughes, Campbell, Cho, & Church, 1999). Hierarchical
clustering can help us recognize genes whose cis-regula-
tory elements are bound by the same proteins (transcrip-
tion factors) in vivo. Such a set of coregulated genes is
known as a regulon. Statistical characterization of known
regulons is used to derive criteria for inferring new
regulatory elements. To identify regulatory elements
and associated transcription factors is fundamental to
building a global gene regulatory network essential for
understanding the genetic control and biology in living
cells. Thus, determining gene functions and gene net-
works from microarray data is an important application
of data mining.
The limitation of the morphology-based approach to
cancer classification has led to molecular classifica-
tion. Techniques such as immunohistochemistry and
RT-PCR are used to detect cancer-specific molecular
markers, but pathognomonic molecular markers are
unfortunately unavailable for most solid tumors
(Ramaswamy, Tamayo, & Rifkin, 2001). Furthermore,
molecular markers do not guarantee a definitive diagno-
sis, owing to possible failure of detection or presence of
marker variants. The approach of constructing a classifier M
based on gene expression profiles has gained increasing
interest, following the success in demonstrating that
microarray data differentiated between two types of leu-
kemia (Golub et al., 1999). In this application, the two data-
mining problems are to identify gene expression patterns
or signatures associated with each type of leukemia and
to discover subtypes within each. The first problem is
dealt with by gene selection, and the second one by
cluster analysis. Table 2 illustrates some applications of
microarray data mining.
FUTURE TRENDS
The future challenge is to realize biological networks that
provide qualitative and quantitative understanding of
molecular logic and dynamics. To meet this challenge,
recent research has begun to focus on leveraging prior
biological knowledge and integration with biological analy-
sis in quest of biological truth. In addition, there is
increasing interest in applying statistical bootstrapping
and data permutation techniques to mining microarray
data for appraising the reliability of leaned patterns.
CONCLUSION
Microarray technology has rapidly emerged as a power-
ful tool for biological research and clinical investiga-
tion. However, the large quantity and complex nature of
data produced in microarray experiments often plague
researchers who are interested in using this technology.
Microarray data mining uses specific data processing and
normalization strategies and has its own objectives, re-

Table 2. Examples of microarray data mining applications

Classical Work:

v Identified functional related genes and their genetic control upon metabolic shift from fermentation to respiration
(DeRisi et al., 1997).
v Explored co-expressed or coregulated gene families by cluster analysis (Eisen et al., 1998).
v Determined genetic network architecture based on coordinated gene expression analysis and promoter motif analysis
(Tavazoie et al., 1999).
v Differentiated acute myeloid leukemia from acute lymphoblastic leukemia by selecting genes and constructing a
classifier for discriminant analysis (Golub et al., 1999).
v Selected genes differentially expressed in response to ionizing radiation based on significance analysis (Tusher et al.,
2001).

Recent Work:

v Analyzed gene expression in the Arabidopsis genome (Yamada, Lim, & Dale, 2003).
v Discovered conserved genetic modules (Stuart, Segal, Koller, & Kim, 2003).
v Elucidated functional properties of genetic networks and identified regulatory genes and their target genes (Gardner, di
Bernardo, Lorenz, & Collins, 2003).
v Identified genes associated with Alzheimers disease (Roy Walker, Smith, & Liu, 2004).

731

TEAM LinG

quiring effective computational algorithms and statistical
techniques to arrive at valid results. The microarray tech-
nology has been perceived as a revolutionary technology
in biomedicine, but the hardware device does not pay off
unless backed up with sound data-mining software.
ACKNOWLEDGMENT
This work is supported by the National Science Founda-
tion under Grant IIS-0221954.
REFERENCES
Brazma, A., Hingamp, P., & Quackenbush, J. (2001). Mini-
mum information about a microarray experiment
(MIAME) toward standards for microarray data. Nat
Genet, 29(4), 365-371.
Cho, R. J., Campbell, M. J., & Winzeler, E. A. (1998). A
genome-wide transcriptional analysis of the mitotic
cell cycle. Mol Cell, 2(1), 65-73.
Comander, J., Weber, G. M., Gimbrone, M. A., Jr., & Garcia-
Cardena (2001). Argus: A new database system for Web-
based analysis of multiple microarray data sets. Genome
Res, 11(9), 1603-1610.
DeRisi, J. L., Iyer, V. R., & Brown, P. O. (1997). Exploring
the metabolic and genetic control of gene expression on
a genomic scale. Science, 278(5338), 680-686.
Eisen, M. B., Spellman, P. T., Brown, P. O., & Botstein
(1998). Cluster analysis and display of genome-wide
expression patterns. Proceedings of the National Acad
Sci, USA, 95(25), 14863-14868.
Ekins, R., & Chu, F. W. (1999). Microarrays: Their origins
and applications. Trends Biotechnol, 17(6), 217-218.
Ermolaeva, O., Rastogi, M., & Pruitt, K. D., (1998). Data
management and analysis for gene expression arrays.
Nat Genet, 20(1), 19-23.
Furey, T. S., Cristianini, N., & Duffy, N., (2000). Support
vector machine classification and validation of cancer
tissue samples using microarray expression data.
Bioinformatics, 16(10), 906-914.
Gardner, T. S., di Bernardo, D., Lorenz, D., & Collins (2003).
Inferring genetic networks and identifying compound
mode of action via expression profiling. Science, 301(5629),
102-105.
Golub, T. R., Slonim, D. K., & Tamayo, P. (1999). Molecular
classification of cancer: Class discovery and class predic-
732
Microarray Data Mining
tion by gene expression monitoring. Science, 286(5439),
531-537.
Guyon, I., Weston, J., Barnhill, S., & Vapnik (2002). Gene
selection for cancer classification using support vector
machines. Machine Learning, 46(1/3), 389-422.
Khan, J., Wei, J. S., & Ringner, M. (2001). Classification
and diagnostic prediction of cancers using gene expres-
sion profiling and artificial neural networks. Nat Med,
7(6), 673-679.
Laub, M. T., McAdams, H. H., Feldblyum, T., Fraser &
Shapiro (2000). Global analysis of the genetic network
controlling a bacterial cell cycle. Science, 290(5499),
2144-2148.
Lee, K. E., Sha, N., Dougherty, E. R., Vannucci & Mallick
(2003). Gene selection: A Bayesian variable selection
approach. Bioinformatics, 19(1), 90-97.
Ramaswamy, S., Tamayo, P., & Rifkin, R. (2001). Multiclass
cancer diagnosis using tumor gene expression signa-
tures. Proceedings of the National Acad Sci, USA, 98(26),
15149-15154.
Roy Walker, P., Smith, B., & Liu, Q. Y. (2004). Data mining
of gene expression changes in Alzheimer brain. Artif Intell
Med, 31(2), 137-154.
Schadt, E. E., Li, C., Ellis, B., & Wong (2001). Feature
extraction and normalization algorithms for high-den-
sity oligonucleotide gene expression array data. Jour-
nal of Cell Biochemistry, (Suppl. 37), 120-125.
Schena, M., Heller, R. A., & Theriault, T. P. (1998). Microarrays:
Biotechnologys discovery platform for functional genomics.
Trends Biotechnol, 16(7), 301-306.
Stuart, J. M., Segal, E., Koller, D., & Kim (2003). A gene-
coexpression network for global discovery of conserved
genetic modules. Science, 302(5643), 249-255.
Tavazoie, S., Hughes, J. D., Campbell, M. J., Cho & Church
(1999). Systematic determination of genetic network ar-
chitecture. Nat Genet, 22(3), 281-285.
Tibshirani, R., Hastie, T., Narasimhan, B., & Chu (2002).
Diagnosis of multiple cancer types by shrunken cen-
troids of gene expression. Proceedings of the National
Acad Sci, USA, 99(10), 6567-6572.
Tusher, V. G., Tibshirani, R., & Chu, G. (2001). Significance
analysis of microarrays applied to the ionizing radiation
response. Proceedings of the National Acad Sci, USA,
98(9), 5116-5121.
Wodicka, L., Dong, H., Mittmann, M., Ho & Lockhart
(1997). Genome-wide expression monitoring in Saccharo-
myces cerevisiae. Nat Biotechnol, 15(13), 1359-1367.
TEAM LinG
Microarray Data Mining
Xiong, M., Li, W., Zhao, J., Jin & Boerwinkle (2001).
Feature (gene) selection in gene expression-based tu-
mor classification. Mol Genet Metab, 73(3), 239-247.
Yamada, K., Lim, J., & Dale, J. M. (2003). Empirical analysis
of transcriptional activity in the Arabidopsis genome.
Science, 302(5646), 842-846.
Yang, Y. H., Dudoit, S., & Luu, P. (2002). Normalization for
cDNA microarray data: A robust composite method ad-
dressing single and multiple slide systematic variation.
Nucleic Acids Res, 30(4), e15.
Zweiger, G. (1999). Knowledge discovery in gene-expres-
sion-microarray data: Mining the information output of the
genome. Trends Biotechnol, 17(11), 429-436.
KEY TERMS
Bioinformatics: All aspects of information pro-
cessing on biological data, in particular genomic data.
The rise of bioinformatics is driven by the genomic
projects.
Cis-Regulatory Element: The genetic region that af-
fects the activity of a gene on the same DNA molecule. M
Clustering: The process of grouping objects accord-
ing to their similarity. This is an important approach to
microarray data mining.
Functional Genomics: The study of gene functions
on a genomic scale, especially based on microarrays.
Gene Expression: Production of mRNA from DNA
(a process known as transcription) and production of
protein from mRNA (a process known as translation).
Microarrays are used to measure the level of gene
expression in a tissue or cell.
Genomic Medicine: Integration of genomic and
clinical data for medical decision.
Microarray: A chip on which numerous probes are
placed for hybridization with a tissue sample to analyze
its gene expression.
Postgenome Era: The time after the complete hu-
man genome sequence is decoded.
Transcription Factor: A protein that binds to the
cis-element of a gene and affects its expression.
733
TEAM LinG
734
Microarray Databases for Biotechnology
Richard S. Segall
Arkansas State University, USA
INTRODUCTION
Microarray informatics is a rapidly expanding disci-
pline in which large amounts of multi-dimensional data
are compressed into small storage units. Data mining of
microarrays can be performed using techniques such as
drill-down analysis rather than classical data analysis on
a record-by-record basis. Both data and metadata can be
captured in microarray experiments. The latter may be
constructed by obtaining data samples from an experi-
ment. Extractions can be made from these samples and
formed into homogeneous arrays that are needed for
higher level analysis and mining.
Biologists and geneticists find microarray analy-
sis as both a practical and appropriate method of storing
images, together with pixel or spot intensities and iden-
tifiers, and other information about the experiment.
BACKGROUND
A Microarray has been defined by Schena (2003) as an
ordered array of microscopic elements in a planar sub-
strate that allows the specific binding of genes or gene
products. Schena (2003) claims microarray databases
as a widely recognized next revolution in molecular
biology that enables scientists to analyze genes, pro-
teins, and other biological molecules on a genomic
scale.
According to an article (2004) on the National Cen-
ter for Biotechnology Information (NCBI) Web site,
because microarrays can be used to examine the ex-
pression of hundreds or thousands of genes at once, it
promises to revolutionize the way scientists examine
gene expression, and this technology is still consid-
ered to be in its infancy.
The following Figure 1 is from a presentation by
Kennedy (2003) of CSIRO (Commonwealth Scientific
& Industrial Research Organisation) in Australia as
available on the Web, and illustrates an overview of the
microarray process starting with sequence data of indi-
vidual clones that can be organized into libraries. Indi-
vidual samples are taken from the library as spots and
arranged by robots onto slides that are then scanned by
lasers. The image scanned by lasers is than quantified
according to the color generated by each individual spot
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
that are then organized into a results set as a text file that
can then be subjected to analyses such as data mining.
Jagannathan (2002) of the Swiss Institute of
Bioinformatics (SIB) described databases for
microarrays including their construction from
microarray experiments such as gathering data from
cells subjected to more than one conditions. The latter
are hybridized to a microarray that is stored after the
experiment by methods such as scanned images. Hence
data is to be stored both before and after the experi-
ments, and the software used must be capable of dealing
with large volumes of both numeric and image data.
Jagannathan (2002) also discussed some of the most
promising existing non-commercial microarray data-
bases of ArrayExpress, which is a public microarray
gene expression repository, the Gene Express Omnibus
(GEO), which is a gene expression database hosted at
the National Library of Medicine, and GeneX, which is
an open source database and integrated tool set released
by the National Center for Genome Resources (NCGR)
in Santa Fe, New Mexico.
Grant (2001) wrote an entire thesis on microarray
databases describing the scene for its application to
genetics and the human genome and its sequence of the
three billion-letter sequences of genes. Kim (2002)
presented improved analytical methods for micro-array
based genome composition analysis by selecting a sig-
nal value that is used as a cutoff to discriminate present
Figure 1. Overview of the microarray process (Kennedy,
(2003)
GENAdb
Genomics Array Database
Overview of the Microarray Process
>m6kp10a06f1 (xseqid=j10a06f1)
NAATTCCCGACCGTGAAAGTAAACCTAAAA Plants
Sequences GCCTATTTATTTCACCTCTCTCTCTCTCTC
TCTGCAACTAATCACTTGTTCNATCTCGAA
GCTGAAGCTAAAGCTTTCGCTAATTTGCTT
Samples
Libraries
Slides
Scans
Result Sets
Analyses
TEAM LinG
Microarray Databases for Biotechnology
and divergent genes. Do et al. (2003) provided compara-
tive evaluation of microarray-based gene expression
databases by analyzing the requirements for microarray
data management, and Sherlock (2003) discussed storage
and retrieval of microarray data for molecular biology.
Kemmeren (2001) described a bioinformatics pipe-
line for supporting microarray analysis with example of
production and analysis of DNA (Deoxyribonucleic
Acid) microarrays that require informatics support.
Gonclaves & Marks (2002) discussed roles and require-
ments for a research microarray database.
An XML description language called MAML
(Microarray Annotation Markup Language) has been
developed to allow communication with other databases
worldwide (Cover Pages 2002). Liu (2004) discusses
microarray databases and MIAME (Minimal Informa-
tion about a Microarray Experiment) that defines what
information at least should be stored. For example, the
MIAME for array design would be the definite structure
and definition of each array used and their elements. The
Microarray Gene Expression Database Group (MGED)
composed and developed the recommendations for
microarray data annotations for both MAIME and MAML
in 2000 and 2001 respectively in Cambridge, United
Kingdom.
Jonassen (2002) presents a microarray informatics
resource Web page that includes surveys and introduc-
tory papers on informatics aspects, and database and
software links. Another resourceful Web site is that
from the Lawrence Livermore National Labs (2003)
entitled Microarray Links that provides an extensive list
of active Web links for the categories of databases,
microarray labs, and software and tools including data
mining tools.
University-wide database systems have been estab-
lished such as at Yale as the Yale Microarray Database
(YMD) to support large-scale integrated analysis of
large amounts of gene expression data produced by a
wide variety of microarray experiments for different
organisms as described by Cheung (2004), and similarly
at Stanford with Stanford Microarray Database (SMD)
as described by both Sherlock (2001) and Selis (2003).
Microarray Image analysis is currently included in
university curricula, such as in Rouchka (2003) Intro-
duction to Bioinformatics graduate course at University
of Louisville.
In relation to the State of Arkansas, the medical
school is situated in Little Rock and is known as the
University of Arkansas for Medical Sciences (UAMS).
A Bioinformatics Center is housed within UAMS that is
involved with the management of microarray data. The
software utilized at UAMS for microarray analysis in-
cludes BASE (BioArray Software Environment) and
AMAD, which is a Web driven database system written
entirely in PERL and JavaScript (UAMS, Bioinformatics M
Center, 2004).
MAIN THRUST
The purpose of this article is to help clarify the meaning
of microarray informatics. The latter is addressed by
summarizing some illustrations of applications of data
mining to microarray databases specifically for bio-
technology.
First, it needs to be stated which data mining tools
are useful in data mining of microarrays. SAS Enterprise
Miner, which was used in Segall et al. (2003, 2004a,
2004b) as discussed below contains the major data
mining tools of decisions trees, regression, neural net-
works, and clustering, and also other data mining tools
such as association rules, variable selection, and link
analysis. All of these are useful data mining tools for
microarray databases regardless if using SAS Enter-
prise Miner or not. In fact, an entire text has been written
by Draghici (2003) on data analysis tools for DNA
microarrays that includes these data mining tools as
well as numerous others tools such as analysis of func-
tional categories and statistical procedure of correc-
tions for multiple comparisons.
Scientific and Statistical Data Mining
and Visual Data Mining for Genomes
Data mining of microarray databases has been discussed
by Deyholos (2002) for bioinformatics by methods that
include correlation of patterns and identifying the sig-
nificance analysis of microarrays (SAM) for genes within
DNA. Visual data mining was utilized to distinguish the
intensity of data filtering and the effect of normaliza-
tion of the data using regression plots.
Tong (2002) discusses supporting microarray stud-
ies for toxicogenomic databases through data integra-
tion with public data and applying visual data mining
such as ScatterPlot viewer.
Chen et al. (2003) presented a statistical approach
using a Gene Expression Analysis Refining System
(GEARS).
Piatetsky-Shapiro and Tamayo (2003) discussed the
main types of challenges for microarrray data mining as
including gene selection, classification, and clustering.
According to Piatetsky-Shapiro and Tamayo (2003), one
of the important challenges for data mining of microarrays
is that the difficulty of collecting microarray samples
causes the number of samples to remain small and while
735
TEAM LinG

the number of fields corresponding to the number of genes
is typically in the thousands this creates a high likeli-
hood of finding false positives.
Piatetsky-Shapiro and Tamayo (2003) identify areas in
which micorarrays and data mining tools can be im-
proved that include better accuracy, more robust mod-
els and estimators as well as better appropriate biologi-
cal interpretation of the computational or statistical
results for those microarrays constructed from bio-
medical or DNA data.
Piatetsky-Shapiro and Tamayo (2003) summarize up
the areas in which microarray and microarry data mining
tools can be improved by stating:
Typically a computational researcher will apply his or
her favorite algorithm to some microarray dataset and
quickly obtain a voluminous set of results. These results
are likely to be useful but only if they can be put in
context and followed up with more detailed studies, for
example by a biologist or a clinical researcher. Often
this follow up and interpretation is not done carefully
enough because of the additional significant research
involvement, the lack of domain expertise or proper
collaborators, or due to the limitations of the
computational analysis itself.
Draghici (2003) discussed in-depth other challenges
in using microarrays specifically for gene expression
studies, such as being very noisy or prone to error after
the scanning and image processing steps, consensus as to
how to perform normalization, and the fact that
microarrays are not necessarily able to substitute com-
pletely other biological factors or tools in the realm of
the molecular biologist.
Mamitsuka et al. (2003) mined biological active pat-
terns in metabolic pathways using microarray expres-
sion profiles. Mamitsuka (2003) utilized microarray
data sets of gene expressions on yeast proteins.
Curran et al. (2003) performed statistical methods
for joint data mining of gene expressions and DNA
sequence databases. The statistical methods used in-
clude linear mixed effect model, cluster analysis, and
logistic regression.
Zaki et al. (2003) reported on an overview of the
papers on data mining in bioinformatics as presented at
the International Conference on Knowledge Discovery
and Data Mining held in Washington, DC in August 2003.
Some of the novel data mining techniques discussed in
papers at this conference included gene expression analy-
sis, protein/RNA (ribonucleic acid) structure predic-
tion, and gene finding.
736
Microarray Databases for Biotechnology
Scientific and Statistical Data Mining
and Visual Data Mining for Plants
Segall et al. (2003, 2004a, 2004b) performed data mining
for assessing the impact of environmental stresses on
plant geonomics and specifically for plant data from the
Osmotic Stress Microarray Information Database
(OSMID). The latter databases are considered to be
representative of those that could be used for biotech
application such as the manufacture of plant-made-phar-
maceuticals (PMP) and genetically modified (GM) foods.
The Osmotic Stress Microarray Information Data-
base (OSMID) database that was used in the data mining
in Segall et al. (2003, 2004a, 2004b) contains the
results of approximately 100 microarray experiments
performed at the University of Arizona as part of a
National Science Foundation (NSF) funded project
named the The Functional Genomics of Plant Stress
whose data constitutes a data warehouse.
The OSMID microarray database is available for
public access on the Web hosted by Universite
Montpellier II (2003) in France, and the OSMID con-
tains information about the more than 20,000 ESTs
(Experimental Stress Tolerances) that were used to
produce these arrays. These 20,000 ESTs could be
considered as components of data warehouse of plant
microarray databases that was subjected to data mining
in Segall et al. (2003, 2004a, 2004b). The data mining
was performed using SAS Enterprise Miner and its
cluster analysis module that yielded both scientific and
statistical data mining as well as visual data mining. The
conclusions of Segall et al. (2003, 2004a, 2004b)
included the facts about the twenty-five different varia-
tions or levels of the environmental factor of salinity
on plant of corn, as also evidenced by the visualization
of the clusters formed as a result of the data mining.
Other Useful Sources of Tools and
Projects for Microarray Informatics
A bibliography on microarray data analysis cre-
ated as available on the Web by Li (2004) that
includes book and reprints for the last ten years.
The Rosalind Franklin Centre for Genomics Re-
search (RFCGR) of the Medical Research Council
(MRC) (2004) in the UK provides a Web site with
links for data mining tools and descriptions of
their specific applications to gene expressions and
microarray databases for genomics and genetics.
Reviews of data mining software as applied to
genetic microarray databases are included in an
TEAM LinG
Microarray Databases for Biotechnology
annotated list of references for microarray soft
ware review compiled by Leung et al. (2002).
Web links for the statistical analysis of microarray
data are provided by van Helden (2004).
Reid (2004) provides Web links of software tools
for microarray data analysis including image
analysis.
Bio-IT World Journal Web site has a Microarray
Resource Center that includes a link of extensive
resources for microarray informatics at the Eu-
ropean Bioinformatics Institute (EBI).
FUTURE TRENDS
The wealth of resources available on the Web for
microarray informatics only supports the premise that
microarray informatics is a rapidly expanding field.
This growth is in both software and methods of analysis
that includes techniques of data mining.
Future research opportunities in microarray
informatics include the biotech applications for manu-
facture of plant-made-pharmaceuticals (PMP) and ge-
netically modified (GM) foods.
CONCLUSION
Because data within genome databases is composed of
micro-level components such as DNA, microarray data-
bases are a critical tool for analysis in biotechnology.
Data mining of microarray databases opens up this field
of microarray informatics as multi-facet tools for knowl-
edge discovery.
ACKNOWLEDGMENT
The author wishes to acknowledge the funding provided
by a block grant from the Arkansas Biosciences Institute
(ABI) as administered by Arkansas State University
(ASU) to encourage development of a focus area in
Biosciences Institute Social and Economic and Regula-
tory Studies (BISERS) for which he served as Co-
Investigator (Co-I) in 2003, and with which funding the
analyses of the Osmotic Stress Microarray Information
Database (OSMID) discussed within this article were
performed.
The author also wishes to acknowledge a three-year
software grant from SAS Incorporated to the College of
Business at Arkansas State University for SAS Enterprise
Miner that was used in the data mining of the OSMID
microarrays discussed within this article.
Finally, the author also wishes to acknowledge the
useful reviews of the three anonymous referees of the M
earlier version of this article without whose construc-
tive comments the final form of this article would not
have been possible.
REFERENCES
Bio-IT World Inc. (2004). Microarray resources and
articles. Retrieved from http://www.bio-itworld.com/
resources/microarray/
Chen, C.H. et al. (2003). Gene expression analysis refining
system (GEARS) via statistical approach: A preliminary
report. Genome Informatics, 14, 316-317.
Cheung, K.H. et al. (2004). Yale Microarray Database
System. Retrieved from http://crcjs.med.utah.edu/
bioinfo/abstracts/Cheung,%20Kei.doc
Curran, M.D., Liu, H., Long, F., & Ge, N. (2003,Decem-
ber). Machine learning in low-level microarray analy-
sis. SIGKDD Explorations, 5(2),122-129.
Deyholos, M. (2002). An introduction to exploring ge-
nomes and mining microarrays. In OReilly
Bioinformatics Technology Conference, January 28-31,
2002, Tucson, AZ. Retrieved from http://
conferences.oreillynet.com/cs/bio2002/view/e_sess/
1962 - 11k - May 7, 2004
Do, H., Toralf, K., & Rahm, E. (2003). Comparative
evaluation of microarray-based gene expression da-
tabases. Retrieved from http://www.btw2003.de/pro-
ceedings/paper/96.pdfDraghici, S. (2003). Data analy-
sis tools for DNA microarrays. Boca Raton, FL: Chapman
& Hall/CRC.
Goncalves, J., & Marks, W.L. (2002). Roles and re-
quirements for a research microarray Database. IEEE
Engineering Medical Biol Magazine, 21(6), 154-157.
Grant, E. (2001, September). A microarray database. The-
sis for Masters of Science in Information Technology.
The University of Glasgow.
Jagannathan, V. (2002). Databases for microarrays. Pre-
sentation at Swiss Institute of Bioinformatics (SIB), Uni-
versity of Lausanne, Switzerland. Retrieved from http://
www.ch.embnet.org/CoursEMBnet/CHIP02/ppt/
Vidhya.ppt
Jonassen, I. (2002). Microarray informatics resource page.
Retrieved from http://www.ii.uib.no/~inge/micro
Kemmeren, P.C., & Holstege, F.C. (2001). A bioinformatics
pipeline for supporting microarray analysis. Retrieved
737
TEAM LinG

from http://www.genomics.med.uu.nl/presentations/
Bioinformatics-2001-Patrick2.ppt
Kennedy, G. (2003). GENAdb: Genomics Array Database.
CSIRO (Commonwealth Scientific & Industrial Research
Organisation) Plant Industry, Australia. Retrieved from
http://www.pi.csiro.au/gena/repository/GENAdb.ppt
Kim, C.K., Joyce E.A., Chan, K., & Falkow, S. (2002).
Improved analytical methods for microarray-based ge-
nome composition analysis. Genome Biology, 3(11). Re-
trieved from http://genomebiology.com/2002/3/11/re-
search/0065
Lawrence Livermore National Labs. (2003). Microarray
Links. Retrieved from http://microarray.llnl.gov/
links.html.
Leung, Y.F. (2002). Microarray software review. In D.
Berrar, W. Dubitzky & M. Granzow (Eds.), A practical
approach to microarray data analysis (pp. 326-344).
Boston: Kluwer Academic Publishers.
Li, W. (2004). Bibliography on microarray data
analysis.Retrieved from http://www.nslij-genetics.org.
microarray/2004.html
Liu, Y. (2004). Microarray Databases and MIAME (Mini-
mum Information About a Microarray Experiment). Re-
trieved from http://titan.biotec.uiuc.edu/cs491jh/slides/
cs491jh-Yong.ppt
Mamitsuka, H., Okuno, Y., & Yamaguchi, A. (2003, Decem-
ber). Mining biological active patterns in metabolic path-
ways using microarray expression profiles. SIGKDD Ex-
plorations, 5(2), 113-121.
Medical Research Council. (2004). Genome Web: Gene
expression and microarrays. Retrieved from http://
www.rfcgr.mrc.ac.uk/GenomeWeb/nuc-genexp.html
Microarray Markup Language (MAML). (2002, Febru-
ary 8). Cover Pages Technology Reports. Retrieved
from http://xml.coverpages.org/maml.html
National Center for Biotechnology Information (NCBI).
(2004, March 30). Microarrays: Chipping away at the
mysteries of science and medicine. National Library of
Medicine (NLM), National Institutes of Health (NIH).
Retrieved from http://www.ncbi.nlm.nih.gov/About/
primer/microarrys.html
Piatetsky-Shapiro, G., & Tamayo, P. (2003, December).
Microarray data mining: Facing the challenges. SIGKDD
Explorations, 5(2), 1-5.
Reid, J.F. (2004). Software tools for microarray data analy-
sis. Retrieved from http://www.ifom-firc.it/
MICROARRAY/data_analysis.htm
738
Microarray Databases for Biotechnology
Rouchka, E. (2003). CECS 694 Introduction to
Bioinformatics. Lecture 12. Microarray Image Analy-
sis, University of Louisville. Retrieved from http://
kbrin.a-bldg.louisville.edu/~rouchka/CECS694_ 2003/
Week12.html
Schena, M. (2003). Microarray analysis. New York: John
Wiley & Sons.
Segall, R.S., Guha, G.S., & Nonis, S. (2003). Data mining
for analyzing the impact of environmental stress on
plants: A case study using OSMID. Manuscript in prepa-
ration for journal submission.
Segall, R.S., Guha, G.S., & Nonis, S. (2004b, May). Data
mining for assessing the impact of environmental stresses
on plant geonomics. In Proceedings of the Thirty-Fifth
Meeting of the Southwest Decision Sciences Institute
(pp. 23-31). Orlando, FL.
Segall, R.S., & Nonis, S. (2004a, February). Data mining for
analyzing the impact of environmental stress on plants: A
case study using OSMID. Accepted for publication in
Acxiom Working Paper Series of Acxiom Laboratory of
Applied Research (ALAR) and presented at Acxiom Con-
ference on Applied Research and Information Technol-
ogy, University of Arkansas at Little Rock (UALR).
Selis, S. (2003, February 15). Stanford researcher advo-
cates far-reaching microarray data exchange. News
release of Stanford University School of Medicine. Re-
trieved from http://www.stanfordhospital.com/
newsEvents/mewsReleases/2003/02/aaasSherlock.html
Sherlock, G. et al. (2001). The Stanford microarray data-
base. Nucleic Acids Research, 29(1), 152-155.
Sherlock, G., & Ball, C.A. (2003). Microarray databases:
Storage and retrieval of microarray data. In M.J. Brownstein
& A. Khodursky (Eds.), Functional genomics: Methods
and protocols (pp. 235-248). Methods in Molecular Biol-
ogy Series (Vol. 224). Totowa, NJ, Humana Press.
Tong, W. (2002, December). ArrayTrack-Supporting
microarray studies through data integration. U. S. Food
and Drug Administration (FDA)/National Center for Toxi-
cological Research (NCTR) Toxioinformatics Workshop:
Toxicogenomics Database, Study Design and Data Analy-
sis.
Universite Montpellier II. (2003). The Virtual Library of
Plant-Array: Databases. Retrieved from http://www.univ-
montp2.fr/~plant_arrays/databases.html
University of Arkansas for Medical Sciences (UAMS)
Bioinformatics Center. (2004). Retrieved from http://
bioinformatics.uams.edu/microarray/database.html
TEAM LinG
Microarray Databases for Biotechnology
Van Helden, J. (2004). Statistical analysis of microarray
data: Links.Retrieved from http://www.scmbb.ulb.ac.be/
~jvanheld/web_course_microarrays/links.html
Zaki, M.J., Wang, H.T., & Toivonen, H.T. (2003, Decem-
ber). Data mining in bioinformatics. SIGKDD Explora-
tions, 5(2), 198-199.
KEY TERMS
Data Warehouses: A huge collection of consistent
data that is both subject-oriented and time variant, and
used in support of decision-making.
Genomic Databases: Organized collection of data
pertaining to the genetic material of an organism.
Metadata: Data about data, for example, data that
describes the properties or characteristics of other data.
MIAME (Minimal Information about a Microarray
Experiment): Defines what information at least should be stored. M
Microarray Databases: Store large amounts of com-
plex data as generated by microarray experiments (e.g., DNA).
Microarray Informatics: The study of the use of
microarray databases to obtain information about experi-
mental data.
Microarray Markup Language (MAML): An XML
(Extensible Markup Language)-based format for com-
municating information about data from microarray
experiments.
Scientific and Statistical Data Mining: The use of
data and image analyses to investigate knowledge dis-
covery of patterns in the data.
Visual Data Mining: The use of computer gener-
ated graphics in both 2-D and 3-D for the use in knowl-
edge discovery of patterns in data.
739
TEAM LinG
740
Mine Rule
Rosa Meo
Universit degli Studi di Torino, Italy
Giuseppe Psaila
Universit degli Studi di Bergamo, Italy
INTRODUCTION
Mining of association rules is one of the most adopted
techniques for data mining in the most widespread appli-
cation domains. A great deal of work has been carried
out in the last years on the development of efficient
algorithms for association rules extraction. Indeed, this
problem is a computationally difficult task, known as
NP-hard (Calders, 2004), which has been augmented by
the fact that normally association rules are being ex-
tracted from very large databases. Moreover, in order to
increase the relevance and interestingness of obtained
results and to reduce the volume of the overall result,
constraints on association rules are introduced and must
be evaluated (Ng et al.,1998; Srikant et al., 1997).
However, in this contribution, we do not focus on the
problem of developing efficient algorithms but on the
semantic problem behind the extraction of association
rules (see Tsur et al. [1998] for an interesting generali-
zation of this problem).
We want to put in evidence the semantic dimensions
that characterize the extraction of association rules;
that is, we describe in a more general way the classes of
problems that association rules solve. In order to ac-
complish this, we adopt a general-purpose query lan-
guage designed for the extraction of association rules
from relational databases. The operator of this lan-
guage, MINE RULE, allows the expression of con-
straints, constituted by standard SQL predicates that
make it suitable to be employed with success in many
diverse application problems. For a comparison be-
tween this query language and other state-of-the-art
languages for data mining, see Imielinski, et al. (1996);
Han, et al. (1996); Netz, et al. (2001); Botta, et al.
(2004).
In Imielinski, et al. (1996), a new approach to data
mining is proposed, which is constituted by a new gen-
eration of databases called Inductive Databases (IDBs).
With an IDB, the user/analyst can use advanced query
languages for data mining in order to interact with the
knowledge discovery (KDD) system, extract data min-
ing descriptive and predictive patterns from the data-
base, and store them in the database. Boulicaut, et al.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
(1998) and Baralis, et al. (1999) discuss the usage of MINE
RULE in this context.
We want to show that, thanks to a highly expressive
query language, it is possible to exploit all the semantic
possibilities of association rules and to solve very
different problems with a unique language, whose state-
ments are instantiated along the different semantic di-
mensions of the same application domain. We discuss
examples of statements solving problems in different
application domains that nowadays are of a great impor-
tance. The first application is the analysis of a retail
data, whose aim is market basket analysis (Agrawal et
al., 1993) and the discovery of user profiles for cus-
tomer relationship management (CRM). The second
application is the analysis of data registered in a Web
server on the accesses to Web sites by users. Cooley, et
al. (2000) present a study on the same application
domain. The last domain is the analysis of genomic
databases containing data on micro-array experiments
(Fayyad, 2003). We show many practical examples of
MINE RULE statements and discuss the application
problems that can be solved by analyzing the association
rules that result from those statements.
BACKGROUND
An association rule has the form B H, where B and H are
sets of items, respectively called body (the antecedent)
and head (the consequent). An association rule (also
denoted for short with rule) intuitively means that items
in B and H often are associated within the observed data.
Two numerical parameters denote the validity of the rule:
support is the fraction of source data for which the rule
holds; confidence is the conditional probability that H
holds, provided that B holds. Two minimum thresholds for
support and confidence are specified before rules are
extracted, so that only significant rules are extracted.
This very general definition, however, is incomplete
and very ambiguous. For example, what is the meaning
of fraction of source data for which the rule holds? Or
what are the items associated by a rule? If we do not
answer these basic questions, an association rule does
TEAM LinG
Mine Rule
not have a precise meaning. Consider, for instance, the
original problem for which association rules were ini-
tially proposed in Agrawal, et al. (1993)the market
baskets analysis. If we have a database collecting single
purchase transactions (i.e., transactions performed by
customers in a retail store), we might wish to extract
association rules that associate items sold within the
same transactions. Intuitively, we are defining the se-
mantics of our problemitems are associated by a rule
if they appear together in the same transaction. Support
denotes the fraction of the total transactions that con-
tain all the items in the rule (both B and H), while
confidence denotes the conditional probability that,
found B in a transaction, also H is found in the same
transaction. Thus a rule
{pants, shirt} {socks, shoes}
support=0.02 confidence=0.23
means that the items pants, shirt, socks, and shoes
appear together in 2% of the transactions, while having
found items pants and shirt in a transaction, the prob-
ability that the same transaction also contains socks and
shoes is 23%.
Semantic Dimensions
MINE RULE puts in evidence the semantic dimensions
that characterize the extraction of association rules
from within relational databases and force users (typi-
cally analysts) to understand these semantic dimen-
sions. Indeed, extracted association rules describe the
most recurrent values of certain attributes that occur in
the data (in the previous example, the names of the
purchased product). This is the first semantic dimension
that characterizes the problem. These recurrent values are
observed within sets of data grouped by some common
features (i.e., the transaction identifier in the previous
example but, in general, the date, the customer identifier,
etc.). This constitutes the second semantic dimension of
the association rule problem. Therefore, extracted asso-
ciation rules describe the observed values of the first
dimension, which are recurrent in entities identified by the
second dimension.
When values belonging to the first dimension are
associated, it is possible that not every association is
suitable, but only a subset of them should be selected,
based on a coupling condition on attributes of the ana-
lyzed data (e.g., a temporal sequence between events
described in B and H). This is the third semantic dimen-
sion of the problem; the coupling condition is called
mining condition.
It is clear that MINE RULE is not tied to any particular
application domain, since the semantic dimensions allow
the discovery of significant and unexpected information
in very different application domains. M
The main features and clauses of MINE RULE are as
follows (see Meo, et al. [1998] for a detailed description):
Selection of the relevant set of data for a data
mining process: This feature is specified by the
FROM clause.
Selection of the grouping features w.r.t., which
data are observed: These features are expressed
by the GROUP BY clause.
Definition of the structure of rules and cardi-
nality constraints on body and head, specified
in the SELECT clause: Elements in rules can be
single values or tuples.
Definition of coupling constraints: These are
constraints applied at the rule level (mining condi-
tion instantiated by a WHERE clause associated to
SELECT) for coupling values.
Definition of rule evaluation measures and
minimum thresholds: These are support and con-
fidence (even if, theoretically, other statistical
measures also would be possible). Support of a
rule is computed on the total number of groups in
which it occurs and satisfies the given constraints.
Confidence is the ratio between the rule support
and the support of the body satisfying the given
constraints. Thresholds are specified by clause
EXTRACTING RULES WITH.
MAIN THRUST
In this section, we introduce MINE RULE in the context
of the three application domains. We describe many
examples of queries that can be conceived as a sort of
template, because they are instantiated along the rel-
evant dimensions of an application domain and solve
some frequent, similar, and critical situations for users
of different applications.
First Application: Retail Data Analysis
We consider a typical data warehouse gathering infor-
mation on customers purchases in a retail store:
FactTable (TransId, CustId, TimeId, ItemId, Num,
Discount)
Customer (CustId, Profession, Age, Sex)
Rows in FactTable describe sales. The dimensions of
data are the customer (CustId), the time (TimeId), and the
purchased item (ItemId); each sale is characterized by the
741
TEAM LinG

number of sold pieces (Num) and the discount (Discount);
the transaction identifier (TransId) is reported, as well. We
also report table Customer.
Example 1: We want to extract a set of association
rules, named FrequentItemSets, that finds the asso-
ciations between sets of items (first dimension of
the problem) purchased together in a sufficient
number of dates (second dimension), with no spe-
cific coupling condition (third dimension). These
associations provide the business relevant sets of
items, because they are the most frequent in time.
The MINE RULE statement is now reported.
MINE RULE FrequentItemSets AS
SELECT DISTINCT 1..n ItemId AS BODY,
1..n ItemId AS HEAD, SUPPORT, CONFIDENCE
FROM FactTable
GROUP BY TimeId
EXTRACTING RULES WITH SUPPORT:0.2,
CONFIDENCE:0.4
The first dimension of the problem is specified in the
SELECT clause that specifies the schema of each ele-
ment in association rules, the cardinality of body and
head (in terms of lower and upper bound), and the statis-
tical measures for the evaluation of association rules
(support and confidence); in the example, body and head
are not empty sets of items, and their upper bound is
unlimited (denoted as 1..n).
The GROUP BY clause provides the second dimen-
sion of the problem: since attribute TimeId is specified,
rules denote that associated items have been sold in the
same date (intuitively, rows are grouped by values of
TimeId, and rules associate values of attribute ItemId
appearing in the same group).
Support of an association rule is computed in terms
of the number of groups in which any element of the rule
co-occurs; confidence is computed analogously. In this
example, support is computed over the different instants
of time, since grouping is made according to the time
identifier. Support and confidence of rules must not be
lower than the values in EXTRACTING clause (respec-
tively 0.2 and 0.4).
Example 2: Customer profiling is a key problem in
CRM applications. Association rules allow to ob-
tain a description of customers (e.g., w.r.t. age and
profession) in terms of frequently purchased prod-
ucts. To do that, values coming from two distinct
dimensions of data must be associated.
742
Mine Rule
MINE RULE CustomerProfiles AS
SELECT DISTINCT 1..1 Profession, Age AS BODY,
1..n Item AS HEAD, SUPPORT, CONFIDENCE
FROM FactTable JOIN Customer
ON FactTable.CustId=Customer.CustId
GROUP BY CustId
EXTRACTING RULES WITH SUPPORT:0.6,
CONFIDENCE:0.9
The observed entity is the customer (first dimen-
sion of data) described by a single pair in the body
(cardinality constraint 1..1); the head associates prod-
ucts frequently purchased by customers (second di-
mension of data) with the profile reported in the body
(see the SELECT clause). Thus a rule
{(employee, 35)} {socks, shoes}
support=0.7 confidence=0.96
means that customers that are employees and 35 years
old often (96% of cases) buy socks and shoes. Support
tells about the absolute frequency of the profile in the
customer base (GROUP BY clause). This solution can
be generalized easily for any profiling problem.
Second Application: Web Log Analysis
Typically, Web servers store information concerning
access to Web sites stored in a standard log file. This is
a relational table (WebLogTable) that typically con-
tains at least the following attributes:
RequestID: identifier of the request;
IPcaller: IP address from which the request is origi-
nated;
Date: date of the request;
TS: time stamp;
Operation: kind of operation (for instance, get or put);
Page URL: URL of the requested page;
Protocol: transfer protocol (such as TCP/IP);
Return Code: code returned by the Web server;
Dimension: dimension of the page (in Bytes).
Example 1: To discover Web communities of users
on the basis of the pages they visited frequently,
we might find associations between sets of users
(first dimension) that have all visited a certain
number of pages (second dimension); no coupling
conditions are necessary (third dimension). Users
are observed by means of their IP address, Ipcaller,
whose values are associated by rules (see SE-
LECT). In this case, support and confidence of
association rules are computed, based on the num-
TEAM LinG
Mine Rule
ber of pages visited by users in rules (see GROUP
BY). Thus, rule
{Ip1, Ip2} {Ip3, Ip4}
support=0.4 confidence=0.45
means that users operating from Ip1, Ip2, Ip3 and Ip4
visited the same set of pages, which constitute 40% of
the total pages in the site.
MINE RULE UsersSamePages AS
SELECT DISTINCT 1..n IPcaller AS BODY,
1..n IPcaller AS HEAD, SUPPORT, CONFIDENCE
FROM WebLogTable
GROUP BY PageUrl
EXTRACTING RULES WITH SUPPORT:0.2,
CONFIDENCE:0.4
Example 2: In Web log analysis, it is interesting
to discover the most frequent crawling paths.
MINE RULE FreqSeqPages AS
SELECT DISTINCT 1..n PageUrl AS BODY,
1..n PageUrl AS HEAD, SUPPORT, CONFIDENCE
WHERE BODY.Date < HEAD.Date
FROM WebLogTable
GROUP BY IPcaller
EXTRACTING RULES WITH SUPPORT:0.3,
CONFIDENCE:0.4
Rows are grouped by user (IPcaller) and sets of pages
frequently visited by a sufficient number of users are
associated. Furthermore, pages are associated only if
they denote a sequential pattern (third dimension); in
fact, the mining condition WHERE BODY.Date <
HEAD.Date constrains the temporal ordering between
pages in antecedent and consequent of rules. Conse-
quently, rule
{P1, P2} {P3, P4, P5} support=0.5 confidence=0.6
means that 50% of users visit pages P3, P4, and P5 after
pages P1 and P2. This solution can be generalized easily
for any problem requiring the search for sequential
patterns.
Many other examples are possible, such as rules that
associate users to frequently visited Web pages (high-
light the fidelity of the users to the service provided by
a Web site) or frequent requests of a page by a browser
that cause an error in the Web server (interesting be-
cause it constitutes a favorable situation to hackers
attacks).
Third Application: Genes Classification
by Micro-Array Experiments M
We consider information on a single micro-array experi-
ment containing data on several samples of biological
tissue tied to correspondent probes on a silicon chip.
Each sample is treated (or hybridized) in various ways
and under different experimental conditions; these can
determine the over-expression of a set of genes. This
means that the sets of genes are active in the experimen-
tal conditions (or inactive, if, on the contrary, they are
under-expressed). Biologists are interested in discov-
ering which sets of genes are expressed similarly and
under what conditions.
A micro-array typically contains hundreds of samples,
and for each sample, several thousands of genes are
measured. Thus, input relation, called MicroArrayTable,
contains the following information:
SampleID: identifier of the sample of biological
tissue tied to a probe on the microchip;
GeneId: identifier of the gene measured in the
sample;
TreatmentConditionId: identifier of the experi-
mental conditions under which the sample has
been treated;
LevelOfExpression: measured valueif higher
than a threshold T2, the genes are over-expressed;
if lower than another threshold T1, genes are un-
der-expressed.
Example: This analysis discovers sets of genes
(first dimension of the problem) that, in the same
experimental conditions (second dimension), are
expressed similarly (third dimension).
MINE RULE SimilarlyCorrelatedGenes AS
SELECT DISTINCT 1..n GeneId AS BODY,
1..n GeneId AS HEAD, SUPPORT, CONFIDENCE
WHERE BODY.LevelOfExpression < T1 AND
HEAD.LevelOfExpression < T1 OR
BODY.LevelOfExpression > T2 AND
HEAD.LevelOfExpression > T2
FROM MicroArrayTable
GROUP BY SampleId, TreatmentConditionId
EXTRACTING RULES WITH SUPPORT:0.95,
CONFIDENCE:0.8
The mining condition introduced by WHERE con-
strains both the sets of genes to be similarly expressed
in the same experimental conditions (i.e., samples of
tissue treated in the same conditions). Support thresh-
743
TEAM LinG

old (0.95) determines the proportion of samples in which
the sets of genes must be expressed similarly; confidence
determines how strongly the two sets of genes are corre-
lated.
This statement might help biologists to discover the
sets of genes that are involved in the production of
proteins involved in the development of certain diseases
(e.g., cancer).
FUTURE TRENDS
This contribution wants to evaluate the usability of a
mining query language and its resultsassociation
rulesin some practical applications. We identified
many useful patterns that corresponded to concrete user
problems. We showed that the exploitation of the nug-
gets of information embedded in the databases and of
the specialized mining constructs provided by the query
languages enables the rapid customization of the mining
procedures leading to the real users needs. Given our
experience, we also claim that, independently of the
application domain, the use of queries in advanced lan-
guages, as opposed to ad-hoc heuristics, eases the speci-
fication and the discovery of a large spectrum of pat-
terns. This motivates the need for powerful query lan-
guages in KDD systems.
For the future, we believe that a critical point will be
the availability of powerful query optimizers, such as
the one proposed in Meo (2003). This one is able to
solve data mining queries incrementally; that is, by
modification of the previous queries results, material-
ized in the database.
CONCLUSION
In this contribution, we focused on the semantic prob-
lem behind the extraction of association rules. We put
in evidence the semantic dimensions that characterize
the extraction of association rules; we did this by apply-
ing a general purpose query language designed for the
extraction of association rules, named MINE RULE, to
three important application domains.
The query examples we provided show that the min-
ing language is powerful and, at the same time, versatile
because its operational semantics seems to be the basic
one. Indeed, these experiments allow us to claim that
Imielinski and Mannilas (1996) initial view on induc-
tive databases was correct: <<There is no such thing as
real discovery, just a matter of the expressive power of
the query languages>>.
744
Mine Rule
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
association rules between sets of items in large data-
bases. Proceedings of the International Conference on
Management of Data, Washington, D.C.
Baralis, E., & Psaila, G. (1999). Incremental refinement of
mining queries. Proceedings of the First International
Conference on Data Warehousing and Knowledge Dis-
covery, Florence, Italy.
Botta, M., Boulicaut, J.-F., Masson, C., & Meo, R. (2004).
Query languages supporting descriptive rule mining: A
comparative study. In R. Meo, P. Lanzi, & M. Klemettinen
(Eds.), Database support for data mining applications.
(pp. 24-51). Berlin: Springer-Verlag.
Boulicaut, J.-F., Klemettinen, M., & Mannila, H. (1998).
Querying inductive databases: A case study on the MINE
RULE operator Proceedings of the International Con-
ference on Principles of Data Mining and Knowledge
Discovery, Nantes, France.
Calders, T. (2004). Computational complexity of itemset
frequency satisfiability. Proceedings of the Sympo-
sium on Principles Of Database Systems, Paris, France.
Cooley, R., Tan, P.N., & Srivastava, J. (2000). Discovery
of interesting usage patterns from Web data. In Proceed-
ings of WEBKDD-99 International Workshop on Web
Usage Analysis and User Profiling, San Diego, California.
Berlin: Springer Verlag.
Fayyad, U.M. (2003). Special issue on microarray data
mining. SIGKDD Explorations, 5(2), 1-139.
Han, J., Fu, Y., Wang, W., Koperski, K., & Zaiane, O.
(1996). DMQL: A data mining query language for rela-
tional databases. Proceedings of the Workshop on Re-
search Issues on Data Mining and Knowledge Discov-
ery, Montreal, Canada.
Imielinski, T., & Mannila, H. (1996). A database per-
spective on knowledge discovery. Communications of
the ACM, 39(11), 58-64.
Imielinski, T., Virmani, A., & Abdoulghani, A. (1996).
DataMine: Application programming interface and query
language for database mining. Proceedings of the Inter-
national Conference on Knowledge Discovery and
Data Mining, Portland, Oregon.
Meo, R. (2003). Optimization of a language for data
mining. Proceedings of the Symposium on Applied
Computing, Melbourne, Florida.
TEAM LinG
Mine Rule
Meo, R., Psaila, G., & Ceri, S. (1998). An extension to SQL
for mining association rules. Journal of Data Mining and
Knowledge Discovery, 2(2), 195-224.
Netz, A., Chaudhuri, S., Fayyad, U.M., & Bernhardt, J.
(2001). Integrating data mining with SQL databases:
OLE DB for data mining Proceedings of the International
Conference on Data Engineering, Heidelberg, Germany.
Ng, R.T., Lakshmanan, V.S., Han, J., & Pang, A. (1998).
Exploratory mining and pruning optimizations of con-
strained associations rules. Proceedings of the Inter-
national Conference Management of Data, Seattle,
Washington.
Srikant, R., Vu, Q., & Agrawal, R. (1997). Mining asso-
ciation rules with item constraints. Proceedings of the
International Conference on Knowledge Discovery
from Databases, Newport Beach, California.
Tsur, D. et al. (1998). Query flocks: A generalization of
association-rule mining. Proceedings of the International
Conference Management of Data, Seattle, Washington.
KEY TERMS
M
Association Rule: An association between two sets
of items co-occurring frequently in groups of data.
Constraint-Based Mining: Data mining obtained by
means of evaluation of queries in a query language allow-
ing predicates.
CRM: Management, understanding, and control of
data on the customers of a company for the purposes of
enhancing business and minimizing the customers churn.
Inductive Database: Database system integrating
in the database source data and data mining patterns
defined as the result of data mining queries on source
data.
KDD: Knowledge Discovery Process from the data-
base, performing tasks of data pre-processing, transfor-
mation and selection, and extraction of data mining
patterns and their post-processing and interpretation.
Semantic Dimension: Concept or entity of the
studied domain that is being observed in terms of other
concepts or entities.
Web Log: File stored by the Web server containing
data on users accesses to a Web site.
745
TEAM LinG
746
Mining Association Rules on a NCR Teradata
System
Soon M. Chung
Wright State University, USA
Murali Mangamuri
Wright State University, USA
INTRODUCTION
Data mining from relations is becoming increasingly im-
portant with the advent of parallel database systems. In
this paper, we propose a new algorithm for mining asso-
ciation rules from relations. The new algorithm is an
enhanced version of the SETM algorithm (Houtsma &
Swami 1995), and it reduces the number of candidate
itemsets considerably. We implemented and evaluated
the new algorithm on a parallel NCR Teradata database
system. The new algorithm is much faster than the SETM
algorithm, and its performance is quite scalable.
BACKGROUND
Data mining, also known as knowledge discovery from
databases, is the process of finding useful patterns from
databases. One of the useful patterns is the association
rule, which is formally described in Agrawal, Imielinski,
and Swami (1993) as follows: Let I = {i1, i2, . . . , i m} be a set
of items. Let D represent a set of transactions, where each
transaction T contains a set of items, such that T I. Each
transaction is associated with a unique identifier, called
transaction identifier (TID). A set of items X is said to be
in transaction T if X T. An association rule is an
implication of the form X => Y, where X I, Y I and X
Y = . The rule X => Y holds in the database D with
confidence c if c% of the transactions in D that contain X
also contain Y. The rule X => Y has a support s if s% of
the transactions in D contain X U Y. For example, beer and
disposable diapers are items such that beer => diapers is
an association rule mined from the database if the co-
occurrence rate of beer and disposable diapers (in the
same transaction) is not less than the minimum support,
and the occurrence rate of diapers in the transactions
containing beer is not less than the minimum confidence.
The problem of mining association rules is to find all
the association rules that have support and confidence
greater than or equal to the user-specified minimum sup-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
port and minimum confidence, respectively. This problem
can be decomposed into the following two steps:
1. Find all sets of items (called itemsets) that have
support above the user-specified minimum support.
These itemsets are called frequent itemsets or large
itemsets.
2. For each frequent itemset, all the association rules
that have minimum confidence are generated as
follows: For every frequent itemset f, find all non-
empty subsets of f. For every such subset a, gener-
ate a rule of the form a => (f - a) if the ratio of
support(f) to support(a) is at least the minimum
confidence.
Finding all the frequent itemsets is a very resource-
consuming task, but generating all the valid association
rules from the frequent itemsets is quite straightforward.
There are many association rule-mining algorithms
proposed (Agarwal, Aggarwal & Prasad, 2000; Agrawal,
Imielinski & Swami, 1993; Agrawal & Srikant, 1994; Bayardo,
1998; Burdick, Calimlim & Gehrke, 2001; Gouda & Zaki,
2001; Holt & Chung, 2001, 2002; Houtsma & Swami, 1995;
Park, Chen & Yu, 1997; Savasere, Omiecinski & Navathe,
1995; Zaki, 2000). However, most of these algorithms are
designed for data stored in file systems. Considering that
relational databases are used widely to manage the corpo-
ration data, integrating the data mining with the relational
database system is important. A methodology for tightly
coupling a mining algorithm with relational database
using user-defined functions is proposed in Agrawal and
Shim (1996), and a detailed study of various architectural
alternatives for coupling mining with database systems is
presented in Sarawagi, Thomas, and Agrawal (1998).
The SETM algorithm proposed in Houtsma and Swami
(1995) was expressed in the form of SQL queries. Thus, it
can be applied easily to relations in the relational data-
bases and can take advantage of the functionalities pro-
vided by the SQL engine, such as the query optimization,
efficient execution of relational algebra operations, and
indexing. SETM also can be implemented easily on a
TEAM LinG
Mining Association Rules on a NCR Teradata System
parallel database system that can execute the SQL queries
in parallel on different processing nodes. By processing
the relations directly, we can easily relate the mined
association rules to other information in the same data-
base, such as the customer information.
In this paper, we propose a new algorithm named
Enhanced SETM (ESETM), which is an enhanced version
of the SETM algorithm. We implemented both ESETM and
SETM on a parallel NCR Teradata database system and
evaluated and compared their performance for various
cases. It has been shown that ESETM is considerably
faster than SETM.
MAIN THRUST
NCR Teradata Database System
The algorithms are implemented on an NCR Teradata
database system. It has two nodes, where each node
consists of 4 Intel 700MHz Xeon processors, 2GB shared
memory, and 36GB disk space. The nodes are intercon-
nected by a dual BYNET interconnection network sup-
porting 960Mbps of data bandwidth for each node. More-
over, nodes are connected to an external disk storage
subsystem configured as a level-5 RAID (Redundant
Array of Inexpensive Disks) with 288GB disk space.
The relational DBMS used here is Teradata RDBMS
(version 2.4.1), which is designed specifically to function
in the parallel environment. The hardware that supports
Teradata RDBMS software is based on off-the-shelf Sym-
metric Multiprocessing (SMP) technology. The hardware
is combined with a communication network (BYNET) that
connects the SMP systems to form Massively Parallel
Processing (MPP) systems, as shown in Figure 1 (NCR
Teradata Division, 2002).
The versatility of the Teradata RDBMS is based on
virtual processors (vprocs) that eliminate the depen-
dency on specialized physical processors. Vprocs are a
set of software processes that run on a node within the
multitasking environment of the operating system. Each
vproc is a separate, independent copy of the processor
Figure 1. Teradata system architecture
software, isolated from other vprocs but sharing some of
the physical resources of the node, such as memory and M
CPUs (NCR Teradata Division, 2002).
Vprocs and the tasks running under them communi-
cate using the unique-address messaging, as if they were
physically isolated from one another. The Parsing Engine
(PE) and the Access Module Processor (AMP) are two
types of vprocs. Each PE executes the database software
that manages sessions, decomposes SQL statements into
steps, possibly parallel, and returns the answer rows to
the requesting client. The AMP is the heart of the Teradata
RDBMS. The AMP is a vproc that performs many data-
base and file-management tasks. The AMPs control the
management of the Teradata RDBMS and the disk sub-
system. Each AMP manages a portion of the physical disk
space and stores its portion of each database table within
that disk space, as shown in Figure 2 (NCR Teradata
Division, 2002 ).
SETM Algorithm
The SETM algorithm proposed in (Houtsma & Swami,
1995) for finding frequent itemsets and the corresponding
SQL queries used are as follows:
// SALES = <trans_id, item>
k := 1;
sort SALES on item;
F1 := set of frequent 1-itemsets and their counts;
R 1 := filter SALES to retain supported items;
repeat
k := k + 1;
sort R k-1 on trans_id, item1, . . . , itemk-1;
R k := merge-scan Rk-1, R 1;
sort R k on item1, . . . , item k ;
F k := generate frequent k-itemsets from the sorted Rk;
Rk := filter R k to retain supported k-itemsets;
until R k = {}
In this algorithm, initially, all frequent 1-itemsets and
their respective counts (F1=<item, count>) are generated
by a simple sequential scan over the SALES table. After
creating F1, R1 is created by filtering SALES using F1. A
merge-scan is performed for creating Rk table using R k-1
Figure 2. Query processing in the Teradata system
747
TEAM LinG

and R1 tables. R k table can be viewed as the set of candidate
k-itemsets coupled with their transaction identifiers.
SQL query for generating Rk:
INSERT INTO R k
SELECT p.trans_id, p.item 1, . . . , p.item k-1, q.item
FROM Rk-1 p, R1 q
WHERE q.trans_id = p.trans_id AND q.item > p.itemk-1
Frequent k-itemsets are generated by a sequential scan
over Rk and selecting only those itemsets that meet the
minimum support constraint.
SQL query for generating Fk:
INSERT INTO F k
SELECT p.item1, . . . , p.itemk, COUNT(*)
FROM R k p
GROUP BY p.item 1, . . . , p.itemk
HAVING COUNT(*) >= :minimum_support
Rk table is created by filtering Rk table using Fk. Rk table
can be viewed as a set of frequent k-itemsets coupled with
their transaction identifiers. This step is performed to
ensure that only the candidate k-itemsets (R k) relative to
frequent k-itemsets are used to generate the candidate
(k+1)-itemsets.
SQL query for generating R k:
INSERT INTO R k
SELECT p.trans_id, p.item 1, . . . , p.itemk
FROM R k p, Fk q
WHERE p.item 1 = q.item 1 AND
. p.item k-1 = q.itemk-1 AND
. p.item k < q.itemk
p.item k-1 = q.item k-1 AND
p.item k = q.item k
ORDER BY p.trans_id, p.item1, . . . , p.itemk
A loop is used to implement the procedure described
above, and the number of iterations depends on the size of
the largest frequent itemset, as the procedure is repeated
until Fk is empty.
Enhanced SETM (ESETM)
The Enhanced SETM (ESETM) algorithm has three modi-
fications to the original SETM algorithm:
1. Create frequent 2-itemsets without materializing R1
and R2.
2. Create candidate (k+1)-itemsets in Rk+1 by joining Rk
with itself.
3. Use a subquery to generate Rk rather than material-
izing it, thereby generating Rk+1 directly from Rk.
The number of candidate 2-itemsets can be very large,
so it is inefficient to materialize R2 table. Instead of creat-
ing R2 table, ESETM creates a view or a subquery to
748
Mining Association Rules on a NCR Teradata System
generate candidate 2-itemsets and directly generates
frequent 2-itemsets. This view or subquery is used also
to create candidate 3-itemsets.
CREATE VIEW R 2 (trans_id, item 1, item 2) AS
SELECT P1.trans_id, P1.item, P2.item
FROM (SELECT p.trans_id, p.item FROM SALES p, F 1 q
WHERE p.item = q.item) AS P1,
(SELECT p.trans_id, p.item FROM SALES p, F 1 q WHERE p.item = q.item) AS P2
WHERE P1.trans_id = P2.trans_id AND
P1.item < P2.item
Note that R1 is not created, since it will not be used for
the generation of Rk. The set of frequent 2-itemsets, F 2,
can be generated directly by using this R2 view.
INSERT INTO F 2
SELECT item 1 , item 2 , COUNT(*)
FROM R2
GROUP BY item 1, item2
HAVING COUNT(*) >= :minimum_support
The second modification is to generate Rk+1 using the
join of Rk with itself, instead of the merge-scan of Rk with
R1.
SQL query for generating Rk+1:
INSERT INTO R k+1
SELECT p.trans_id, p.item1, . . . , p.item k, q.itemk
FROM R k p, Rk q
WHERE p.trans_id = q.trans_id AND
p.item 1 = q.item 1 AND
.
.
This modification reduces the number of candidates
(k+1)-itemsets generated compared to the original SETM
algorithm. The performance of the algorithm can be
improved further if candidate (k+1)-itemsets are gener-
ated directly from candidate k-itemsets using a subquery
as follows:
SQL query for R k+1 using R k:
INSERT INTO R k+1
SELECT P1.trans_id, P1.item 1, . . . , P1.itemk, P2.item k
FROM
(SELECT p.* FROM Rk p, Fk q WHERE p.item 1 = q.item1
AND . . . AND p.item k = q.item k) AS P1,
(SELECT p.* FROM Rk p, Fk q WHERE p.item 1 = q.item1
AND . . . AND p.item k = q.item k) AS P2
WHERE P1.trans_id = P2.trans_id AND
P1.item 1 = P2.item 1 AND
.
.
P1.itemk-1 = P2.itemk-1 AND
P1.item k < P2.itemk
Rk is generated as a derived table using a subquery,
thereby saving the cost of materializing R k table.
TEAM LinG
Mining Association Rules on a NCR Teradata System
ESETM with Pruning (PSETM)
In the ESETM algorithm, candidate (k+1)-itemsets in Rk+1
are generated by joining Rk with itself on the first k-1 items,
as described previously. For example, a 4-itemset {1, 2, 3,
9} becomes a candidate 4-itemset only if {1, 2, 3} and {1,
2, 9} are frequent 3-itemsets. It is different from the subset-
infrequency-based pruning of the candidates used in the
Apriori algorithm, where a (k+1)-itemset becomes a can-
didate (k+1)-itemset, only if all of its k-subsets are fre-
quent. So, {2, 3, 9} and {1, 3, 9} also should be frequent
for {1, 2, 3, 9} to be a candidate 4-itemset. The above SQL-
query for generating Rk+1 can be modified such that all the
k-subsets of each candidate (k+1)-itemset can be checked.
To simplify the presentation, we divided the query into
subqueries. Candidate (k+1)-itemsets are generated by
the Subquery Q1 using Fk.
Subquery Q0:
SELECT item 1,item2, . . . , itemk FROM F k
Subquery Q1:
SELECT p.item1, p.item 2, . . . , p.item k, q.item k
FROM Fk p, Fk q
WHERE p.item1 = q.item 1 AND
.
. because the effect of the reduced number of candidates in
p.itemk-1 = q.item k-1 AND
p.item k < q.item k AND
(p.item2, . . . , p.itemk, q.itemk) IN (Subquery Q0) AND
.
. 0.25% is shown in Figure 4. The second pass execution
(p.item1, . . . , p.itemj-1, p.itemj+1, . . . , p.item k, q.itemk) IN
(Subquery Q0 ) AND
.
.
(p.item1, . . . , p.item k-2, p.item k, q.item k) IN (Subquery Q0)
Subquery Q2:
SELECT p.* FROM R k p, F k q
WHERE p.item 1 = q.item 1 AND . . . . AND p.itemk = q.item k
INSERT INTO R k+1
SELECT p.trans_id, p.item1, . . . , p.item k, q.itemk
FROM (Subquery Q 2) p, (Subquery Q 2) q
WHERE p.trans_id = q.trans_id AND
p.item 1 = q.item1 AND
. Figure 3. Total execution times (for T10.I4.D100K)
.
p.itemk-1 = q.item k-1 AND
p.itemk < q.itemk AND
(p.item1, . . . , p.itemk, q.item k) IN (Subquery Q1)
The Subquery Q1 joins Fk with itself to generate the
candidate (k+1)-itemsets, and all candidate (k+1)-itemsets
having any infrequent k-subset are pruned. The Subquery
Q2 derives Rk, and Rk+1 is generated as: Rk+1 = (Rk JOIN Rk)
JOIN (Subquery Q1).
However, it is not efficient to prune all the candidates
in all the passes, since the cost of pruning the candidates
in the Subquery Q1 is too high when there are not many
candidates to be pruned. In our implementation, the prun- M
ing is performed until the number of rows in Fk becomes
less than 1,000, or up to five passes. The difference
between the total execution times with and without prun-
ing was very small for most of the databases we tested.
Performance Analysis
In this section, the performance of the Enhanced SETM
(ESETM), ESETM with pruning (PSETM), and SETM are
evaluated and compared. We used synthetic transaction
databases generated according to the procedure de-
scribed in (Agrawal & Srikant, 1994).
The total execution times of ESETM, PSETM and
SETM are shown in Figure 3 for the database T10.I4.D100K,
where Txx.Iyy.DzzzK indicates that the average number of
items in a transaction is xx, the average size of maximal
potential frequent itemset is yy, and the number of trans-
actions in the database is zzz in thousands.
ESETM is more than three times faster than SETM for
all minimum support levels, and the performance gain
increases as the minimum support level decreases. ESETM
and PSETM have almost the same total execution time,
PESTM is offset by the extra time required for the pruning.
The time taken for each pass by the algorithms for the
T10.I4.D100K database with the minimum support of
time of ESTM is much smaller than that of SETM, because
R2 table (containing candidate 2-itemsets together with
the transaction identifiers) and R2 table (containing fre-
quent 2-itemsets together with the transaction identifiers)
are not materialized. In the later passes, the performance
of ESETM is much better than that of SETM, because
ESTM has much less candidate itemsets generated and
does not materialize Rk tables, for k > 2.
In Figure 5, the size of Rk table containing candidate
k-itemsets is shown for each pass when the T10.I4.D100K
database is used with the minimum support of 0.25%. From
the third pass, the size of Rk table for ESETM is much
SETM ESETM PSETM
1200
1000
800
Time (sec)
600
400
200
0
1.00% 0.50% 0.25% 0.10%
Minimum Support
749
TEAM LinG
 Mining Association Rules on a NCR Teradata System

smaller than that of SETM because of the reduced number
of candidate itemsets. PSETM performs additional prun-
ing of candidate itemsets, but the difference in the number
of candidates is very small in this case.
The scalability of the algorithms is evaluated by in-
creasing the number of transactions and the average size
of transactions. Figure 6 shows how the three algorithms
scale up as the number of transactions increases. The
database used here is T10.I4, and the minimum support is
0.5%. The number of transactions ranges from 100,000 to
400,000. SETM performs poorly as the number of transac-
tions increases, because it generates much more candi-
date itemsets than others.
The effect of the transaction size on the performance
is shown in Figure 7. In this case, the size of the database
wasnt changed by keeping the product of the average
transaction size and the number of transactions constant.
The number of transactions was 20,000 for the average
transaction size of 50 and 100,000 for the average transac-
tion size of 10. We used the fixed minimum support count
of 250 transactions, regardless of the number of transac-
tions. The performance of SETM deteriorates as the
transaction size increases, because the number of candi-
date itemsets generated is very large. On the other hand,
the total execution times of ESETM and PSETM are stable,
because the number of candidate itemsets generated in
the later passes is small.
FUTURE TRENDS
Relational database systems are used widely, and the size
of existing relational databases grows quite rapidly. Thus,
mining the relations directly without transforming them
into certain file structures is very useful. However, due to
the high operation complexity of the mining processes,
parallel data mining is essential for very large databases.
Currently, we are developing an algorithm for mining
association rules across multiple relations using our par-
allel NCR Teradata database system.
CONCLUSION
In this paper, we proposed a new algorithm, named En-
hanced SETM (ESETM) for mining association rules from
relations. ESETM is an enhanced version of the SETM
algorithm (Houtsma & Swami, 1995), and its performance
is much better than SETM, because it generates much less

Figure 4. Per pass execution times (for T10.I4.D100K) Figure 5. Size of R'k (for T10.I4.D100K)

SETM ESETM PSETM

SETM ESETM PSETM

250 2500
No. of Tuples (in 1000s)

200 2000
Time (sec)

150 1500

100 1000

50 500

0
0
R3 R4 R5 R6 R7 R8 R9
1 2 3 4 5 6 7 8 9
Number of Passes

Figure 6. Effect of the number of transactions Figure 7. Effect of the transaction size

SETM ESETM PSETM SETM ESETM PSETM

1200 5000

1000 4000
800
Time (sec)
Time (sec)

3000
600
2000
400
200 1000

0 0
100 200 300 400 10 20 30 40 50
Number of Transactions (in 1000s) Average Transaction Size

750

TEAM LinG
Mining Association Rules on a NCR Teradata System
candidate itemsets to count. ESTM and SETM are imple-
mented on a parallel NCR database system, and we evalu-
ated their performance in various cases. ESTM is at least
three times faster than SETM in most of our test cases, and
its performance is quite scalable.
ACKNOWLEDGMENTS
This research was supported in part by NCR, LexisNexis,
Ohio Board of Regents (OBR), and AFRL/Wright Broth-
ers Institute (WBI)
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
association rules between sets of items in large data-
bases. Proceedings of the ACM SIGMOD International
Conference on Management of Data, Washington, D.C.,
USA.
Agrawal, R., & Shim, K. (1996). Developing tightly-coupled
data mining applications on a relational database system.
Proceedings of the International Conference on Knowl-
edge Discovery and Data Mining, Portland, OR, USA.
Agrawal. R., & Srikant, R. (1994). Fast algorithms for
mining association rules. Proceedings of the VLDB Confer-
ence.
Agarwal, R.C., Aggarwal, C.C., & Prasad, V.V.V. (2000).
Depth first generation of long patterns. Proceedings of
the International Conference on Knowledge Discovery
and Data Mining, Boston, MS, USA.
Bayardo, R.J. (1998). Efficient mining long patterns from
databases. Proceedings of the ACM SIGMOD Interna-
tional Conference on Management of Data, Seattle, WA,
USA.
Burdick, D., Calimlim, M., & Gehrke, J. (2001). MAFIA: A
maximal frequent itemset algorithm for transaction data-
bases. Proceedings of the International Conference on
Data Engineering, Heidelberg, Germany.
Gouda, K., & Zaki, M.J. (2001). Efficiently mining maximal
frequent itemsets. Proceedings of the 1st IEEE Interna-
tional Conference on Data Mining, San Jose, CA, USA.
Holt, J.D., & Chung, S.M. (2001). Multipass algorithms for
mining association rules in text databases. Knowledge M
and Information Systems, 3(2), 168-183.
Holt, J.D., & Chung, S.M. (2002). Mining association rules
using inverted hashing and pruning. Information Pro-
cessing Letters, 83(4), 211-220.
Houtsma, M., & Swami, A. (1995). Set-oriented mining for
association rules in relational databases. Proceedings of
the International Conference on Data Engineering,
Taipei, Taiwan.
NCR Teradata Division (2002). Introduction to Teradata
RDBMS.
Park, J.S., Chen, M.S., & Yu, P.S. (1997). Using a hash-
based method with transaction trimming for mining asso-
ciation rules. IEEE Trans. on Knowledge and Data Engi-
neering, 9(5), 813-825.
Sarawagi, S., Thomas, S., & Agrawal, R. (1998). Integrat-
ing association rule mining with relational database sys-
tems: Alternatives and implications. Proceedings of the
ACM SIGMOD International Conference on Manage-
ment of Data, Seattle, WA, USA
Savasere, A., Omiecinski, E., & Navathe, S. (1995). An
eficient algorithm for mining association rules in large
databases. Proceedings of the VLDB Conference, Zurich,
Switzerland.
Zaki, M.J. (2000). Scalable algorithms for association
mining. IEEE Trans. on Knowledge and Data Engineer-
ing, 12(3), 372-390.
KEY TERMS
Association Rule: Implication of the form X => Y,
meaning that database tuples satisfying the conditions of
X are also likely to satisfy the conditions of Y.
Data Mining: Process of finding useful data patterns
hidden in large data sets.
Parallel Database System: Database system support-
ing the parallel execution of the individual basic database
operations, such as relational algebra operations and
aggregate operations.
751
TEAM LinG
752
Mining Association Rules Using Frequent
Closed Itemsets
Nicolas Pasquier
Universit de Nice-Sophia Antipolis, France
INTRODUCTION
In the domain of knowledge discovery in databases and
its computational part called data mining, many works
addressed the problem of association rule extraction that
aims at discovering relationships between sets of items
(binary attributes). An example association rule fitting in
the context of market basket data analysis is cereal milk
sugar (support 10%, confidence 60%). This rule states
that 60% of customers who buy cereals and sugar also buy
milk, and that 10% of all customers buy all three items.
When an association rule support and confidence exceed
some user-defined thresholds, the rule is considered
relevant to support decision making. Association rule
extraction has proved useful to analyze large databases in
a wide range of domains, such as marketing decision
support; diagnosis and medical research support; tele-
communication process improvement; Web site manage-
ment and profiling; spatial, geographical, and statistical
data analysis; and so forth.
The first phase of association rule extraction is the data
selection from data sources and the generation of the data
mining context that is a triplet D = (O, I, R), where O and I
are finite sets of objects and items respectively, and R O
I is a binary relation. An item is most often an attribute
value or an interval of attribute values. Each couple (o, i)
R denotes the fact that the object o O is related to the item
i I. If an object o is in relation with all items of an itemset
I (a set of items) we say that o contains I.
This phase helps to improve the extraction efficiency
and enables the treatment of all kinds of data, often mixed
in operational databases, with the same algorithm. Data-
mining contexts are large relations that do not fit in
main memory and must be stored in secondary memory.
Consequently, each context scan is very time consuming.
Table 1. Example context
OID Items
1 ACD
2 BCE
3 ABCE
4 BE
5 ABCE
6 BCE
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
The support of an itemset I is the proportion of objects
containing I in the context. An itemset is frequent if its
support is greater or equal to the minimal support thresh-
old defined by the user. An association rule r is an
implication with the form r: I1 I2 - I1 where I1 and I2 are
frequent itemsets such that I1 I2. The confidence of r is
the number of objects containing I2 divided by the number
of objects containing I1. An association rule is generated
if its support and confidence are at least equal to the
minsupport and minconfidence thresholds. Association
rules with 100% confidence are called exact association
rules; others are called approximate association rules.
The natural decomposition of the association rule-mining
problem is:
1. Extracting frequent itemsets and their support from
the context.
2. Generating all valid association rules from frequent
itemsets and their support.
The first phase is the most computationally expensive
part of the process, since the number of potential frequent
itemsets 2|I| is exponential in the size of the set of items, and
context scans are required. A trivial approach would
consider all potential frequent itemsets at the same time,
but this approach cannot be used for large databases
where I is large. Then, the set of potential frequent
itemsets that constitute a lattice called itemset lattice
must be decomposed into several subsets considered one
at a time.
Level-Wise Algorithms for Extracting
Frequent Itemsets
These algorithms consider all itemsets of a given size (i.e.,
all itemsets of a level in the itemset lattice) at a time. They
are based on the properties that all supersets of an
infrequent itemset are infrequent and all subsets of a
frequent itemset are frequent (Agrawal et al., 1995).
Using this property, the candidate k-itemsets (itemsets of
size k) of the kth iteration are generated by joining two
frequent (k-1)-itemsets discovered during the preceding
TEAM LinG
Mining Association Rules Using Frequent Closed Itemsets

Figure 1. Itemset lattice These algorithms perform an iterative search in the

itemset lattice advancing during each iteration by one M
A BC D E level from the bottom upwards, as in A PRIORI, and by one
or more levels from the top downwards. Compared to
preceding algorithms, both the number of iterations and,
A BC E A BC D A BD E ACDE BCDE
thus, the number of context scans and the number of CPU
operations carried out are reduced. The most well known
A BC A BE ACE BCE A BD ACD BCD ADE BDE CDE
algorithms based on this approach are PINCER-SEARCH (Lin
& Kedem, 1998) and MAX-MINER (Bayardo, 1998).

AB AC AE BE BC CE AD CD BD DE Relevance of Extracted Association

Rules
A B C E D
For many datasets, a huge number of association rules is
extracted, even for high minsupport and minconfidence
values. This problem is crucial with correlated data, for
which several million association rules sometimes are
extracted. Moreover, a majority of these rules bring the
same information and, thus, are redundant. To illustrate
this problem, nine rules extracted from the mushroom
iteration, if their k-1 first items are identical. Then, one dataset (ftp://ftp.ics.uci.edu/pub/machine-learning-data-
database scan is performed to count the supports of the bases/mushroom/) are presented in the following. All
candidates, and infrequent ones are pruned. This process have the same support (51%) and confidence (54%), and
is repeated until no new candidate can be generated. the item free gills in the antecedent:
This approach is used in the well known APRIORI and
OCD algorithms. Both carry out a number of context scans 1. free_gills edible
equal to the size of the largest frequent itemsets. Several 2. free_gills edible, partial_veil
optimizations have been proposed to improve the effi- 3. free_gills edible, white_veil
ciency by avoiding several context scans. The COFI* (El- 4. free_gills edible, partial_veil, white_veil
Hajj & Zaane, 2004) and FP-GROWTH (Han et al., 2004) 5. free_gills, partial_veil edible
algorithms use specific data structures for that, and the 6. free_gills, partial_veil edible, white_veil
PASCAL algorithm (Bastide et al., 2000) uses a method 7. free_gills, white_veil edible
called pattern counting inference to avoid counting all 8. free_gills, white_veil edible, partial_veil
supports. 9. free_gills, partial_veil, white_veil edible

The most relevant rule from the viewpoint of the user is

Algorithms for Extracting Maximal rule 4, since all other rules can be deduced from this one,
Frequent Itemsets including support and confidence. This rule is a non-redun-
dant association rule with minimal antecedent and maximal
Maximal and minimal itemsets are defined according to the consequent, or minimal non-redundant rule, for short.
inclusion relation. Maximal frequent itemsets are frequent
itemsets of which all supersets are infrequent. They form Association Rules Reduction Methods
a border under which all itemsets are frequent; knowing
all maximal frequent itemsets, we can deduce all frequent
Several approaches for reducing the number of rules and
itemsets, but not their support. Then, the following ap-
selecting the most relevant ones have been proposed.
proach for mining association rules was proposed:
The application of templates (Baralis & Psaila, 1997)
or Boolean operators (Bayardo, Agrawal & Gunopulos,
1. Extracting maximal frequent itemsets and their sup-
2000) allows selecting rules according to the users
ports from the context.
preferences.
2. Deriving frequent itemsets from maximal fre-
When taxonomies of items exist, generalized asso-
quent itemsets and counting their support in the
ciation rules (Han & Fu, 1999) (i.e., rules between
context during one final scan.
items of different levels of taxonomies) can be extracted.
3. Generating all valid association rules from fre-
This produces fewer but more general associations. Other
quent itemsets.
statistical measures, such as Pearsons correlation or c 2,

753

TEAM LinG
 Mining Association Rules Using Frequent Closed Itemsets
also can be used instead of the confidence to determine the
rule precision (Silverstein, Brin & Motwani, 1998).
Several methods to prune similar rules by analyzing
their structures also have been proposed. This allows the
extraction of rules only, with maximal antecedents among
those with the same support and the same consequent
(Bayardo & Agrawal, 1999), for instance.
MAIN THRUST
Algorithms for Extracting Frequent
Closed Itemsets
In contrast with the (maximal) frequent itemsets-based
approaches, the frequent closed itemsets approach
(Pasquier et al., 1998; Zaki & Ogihara, 1998) is based on the
closure operator of the Galois connection. This operator
associates with an itemset I the maximal set of items
common to all the objects containing I (i.e., the intersection
of these objects). The frequent closed itemsets are fre-
quent itemsets with (I) = I. An itemset C is a frequent
closed itemset, if no other item i C is common to all
objects containing C.
The frequent closed itemsets, together with their sup-
ports, constitute a generating set for all frequent itemsets
and their supports and, thus, for all association rules, their
supports, and their confidences (Pasquier et al., 1999a).
This property relies on the properties that the support of
a frequent itemset is equal to the support of its closure and
that the maximal frequent itemsets are maximal frequent
closed itemsets. Using these properties, a new approach
for mining association rules was proposed:
1. Extracting frequent closed itemsets and their sup-
ports from the context.
2. Deriving frequent itemsets and their supports from
frequent closed itemsets.
3. Generating all valid association rules from frequent
itemsets.
The search space in the first phase is reduced to the
closed itemset lattice, which is a sublattice of the itemset
lattice.
The first algorithms based on this approach proposed
are CLOSE (Pasquier et al., 1999a) and A-CLOSE (Pasquier
et al., 1999b). To improve the extraction efficiency, both
perform a level-wise search for generators of frequent
closed itemsets. The generators of a closed itemset C are
the minimal itemsets whose closure is C; an itemset G is
a generator of C, if there is no other itemset G G whose
closure is C.
During an iteration k, CLOSE considers a set of candi-
date k-generators. One context scan is performed to com-
754
Figure 2. Closed itemset lattice
ABCDE
ACD ABCE
AC BCE
C BE
pute their supports and closures; for each generator G,
the intersection of all objects containing G gives its
closure, and counting them gives its support. Then,
infrequent generators and generators of frequent closed
itemsets previously discovered are pruned. During the
(k+1)th iteration, candidate (k+1)-generators are con-
structed by joining two frequent k-generators having
identical k-1 first items.
In the A-CLOSE algorithm, generators are identified by
comparing supports only, since the support of a genera-
tor is different from the supports of all its subsets. Then,
one more context scan is performed at the end of the
algorithm to compute closures of all frequent generators
discovered.
Recently, the CHARM (Zaki & Hsiao, 2002), CLOSET +
(Wang, Han & Pei, 2003) and BIDE (Wang & Han, 2004)
algorithms have been proposed. These algorithms effi-
ciently extract frequent closed itemsets but not their
generators. The TITANIC algorithm (Stumme et al., 2002)
can extract frequent closed sets according to different
closures, such as functional dependencies or Galois
closures, for instance.
Comparing Execution Times
Experiments conducted on both synthetic and opera-
tional datasets showed that (maximal) frequent itemsets-
based approaches are more efficient than closed itemsets-
based approaches on weakly correlated data, such as
market-basket data. In such data, nearly all frequent
itemsets also are frequent closed itemsets (i.e., closed
itemset lattice and itemset lattice are nearly identical),
and closure computations add execution times.
Correlated data constitute a challenge for efficiently
extracting association rules, since the number of fre-
quent itemsets is most often very important, even for
TEAM LinG
Mining Association Rules Using Frequent Closed Itemsets
high minsupport values. On these data, few frequent
itemsets are also frequent closed itemsets. Thus, the
closure helps to reduce the search space; fewer itemsets
are tested, and the number of context scans is reduced. On
such data, maximal frequent itemsets-based approaches
suffer from the time needed to compute frequent itemset
supports that require accessing the dataset. With the
closure, these supports are derived from the supports of
frequent closed itemsets without accessing the dataset.
Extracting Bases for Association Rules
Bases are minimal sets, with respect to some criteria, from
which all rules can be deduced with support and confi-
dence. The Duquenne-Guigues and the Luxenburger ba-
sis for global and partial implications were adapted to
association rule framework in Pasquier et al. (1999c) and
Zaki (2000). These bases are minimal regarding the number
of rules; no smaller set allows the deduction of all rules
with support and confidence. However, they do not
contain the minimal non-redundant rules.
An association rule is redundant, if it brings the same
information or less general information than those con-
veyed by another rule with identical support and confi-
dence. Then, an association rule r is a minimal non-
redundant association rule, if there is no association rule
r with the same support and confidence whose anteced-
ent is a subset of the antecedent of r and whose conse-
quent is a superset of the consequent of r. An inference
system based on this definition was proposed in Cristofor
and Simovici (2002).
The Min-Max basis for exact association rules con-
tains all rules G g(G) - G between a generator G and its
closure (G) such that (G) G. The Min-Max basis for
approximate association rules contains all rules G C -
G between a generator itemset G and a frequent closed
itemset C that is a superset of its closure: (G) C. These
bases, also called informative bases, contain, respec-
tively, the minimal non-redundant exact and approximate
association rules. Their union constitutes a basis for all
association rules: They all can be deduced with their
support and confidence (Bastide et al., 2000). The objec-
tive is to capture the essential knowledge in a minimal
number of rules without information loss.
Algorithms for determining generators, frequent closed
itemsets, and the min-max bases from frequent itemsets
and their supports are presented in Pasquier et al. (2004).
Comparing Sizes of Association Rule
Sets
Results of experiments conducted on both synthetic and
operational datasets show that the generation of the
bases can reduce substantially the number of rules.
For weakly correlated data, very few exact rules are
extracted, and the reduction for approximate rules is in the M
order of five for both the min-max and the Luxenburger
bases.
For correlated data, the Duquenne-Guigues basis re-
duces exact rules to a few tens; for the min-max exact basis,
the reduction factor is about some tens. For approximate
association rules, both the Luxenburger and the min-max
bases reduce the number of rules by a factor of some
hundreds.
If the number of rules can be reduced from several
million to a few hundred or a few thousand, visualization
tools such as templates and/or generalization tools such
as taxonomies are required to explore so many rules.
FUTURE TRENDS
Most recent researches on association rules extraction
concern applications to natural phenomena modeling,
gene expression analysis (Creighton & Hanash, 2003),
biomedical engineering (Gao, Cong et al., 2003), and
geospatial, telecommunications, Web and semi-struc-
tured data analysis (Han et al., 2002). These applications
most often require extending existing methods. For in-
stance, to extract only rules with low support and high
confidence in semi-structured (Cohen et al., 2001) or
medical data (Ordonez et al., 2001), to extract temporal
association rules in Web data (Yang & Parthasarathy,
2002) or adaptive sequential association rules in long-
term medical observation data (Brisson et al., 2004). Fre-
quent closed itemsets extraction also is applied as a
conceptual analysis technique to explore biological (Pfaltz
& Taylor, 2002) and medical data (Cremilleux, Soulet &
Rioult, 2003).
These domains are promising fields of application for
association rules and frequent closed itemsets-based
techniques, particularly in combination with other data
mining techniques, such as clustering and classification.
CONCLUSION
Next-generation data-mining systems should answer the
analysts requirements for high-level ready-to-use knowl-
edge that will be easier to exploit. This implies the integra-
tion of data-mining techniques in DBMS and domain-
specific applications (Ansari et al., 2001). This integration
should incorporate the use of knowledge visualization
and exploration techniques, knowledge consolidation by
cross-analysis of results of different techniques, and the
incorporation of background knowledge, such as taxono-
mies or gene annotations for gene expression data, for
example, in the process.
755
TEAM LinG
 Mining Association Rules Using Frequent Closed Itemsets
REFERENCES
Agrawal, R., Mannila, H., Srikant, R., Toivonen, H., &
Verkamo, A.I. (1995). Fast discovery of association rules.
Advances in knowledge discovery and data mining.
AAAI/MIT Press.
Ansari, S., Kohavi, R., Mason, L., & Zheng, Z. (2001).
Integrating e-commerce and data mining: Architecture
and challenges. Proceedings of the ICDM Conference.
Baralis, E., & Psaila, G. (1997). Designing templates for
mining association rules. Journal of Intelligent Informa-
tion Systems, 9(1), 7-32.
Bastide, Y., Pasquier, N., Taouil, R., Lakhal, L., & Stumme,
G. (2000). Mining minimal non-redundant association
rules using frequent closed itemsets. Proceedings of the
DOOD Conference.
Bastide, Y., Taouil, R., Pasquier, N., Stumme, G., & Lakhal,
L. (2000). Mining frequent closed itemsets with counting
inference. SIGKDD Explorations, 2(2), 66-75.
Bayardo, R.J. (1998). Efficiently mining long patterns from
databases. Proceedings of the SIGMOD Conference.
Bayardo, R.J., & Agrawal, R. (1999). Mining the most
interesting rules. Proceedings of the KDD Conference.
Bayardo, R.J., Agrawal, R., & Gunopulos, D. (2000). Con-
straint-based rule mining in large, dense databases. Data
Mining and Knowledge Discovery, 4(2/3), 217-240.
Brisson, L., Pasquier, N., Hebert, C., & Collard, M. (2004).
HASAR: Mining sequential association rules for athero-
sclerosis risk factor analysis. Proceedings of the PKDD
Discovery Challenge.
Cohen, E. et al. (2001). Finding interesting associations
without support pruning. IEEE Transaction on Knowl-
edge and Data Engineering, 13(1), 64,78.
Creighton, C., & Hanash, S. (2003). Mining gene expres-
sion databases for association rules. Bioinformatics,
19(1), 79-86.
Cremilleux, B., Soulet, A., & Rioult, F. (2003). Mining the
strongest emerging patterns characterizing patients af-
fected by diseases due to atherosclerosis. Proceedings of
the PKDD Discovery Challenge.
Cristofor, L., & Simovici, D.A. (2002). Generating an infor-
mative cover for association rules. Proceedings of the
ICDM Conference.
El-Hajj, M., & Zaane, O.R. (2004). COFI approach for
mining frequent itemsets revisited. Proceedings of the
SIGMOD/DMKD Workshop.
756
Gao Cong, F.P., Tung, A., Yang, J., & Zaki, M.J. (2003).
CARPENTER: Finding closed patterns in long biological
datasets. Proceedings of the KDD Conference.
Han, J., & Fu, Y. (1999). Mining multiple-level association
rules in large databases. IEEE Transactions on Knowl-
edge and Data Engineering, 11(5), 798-804.
Han, J., Pei, J., Yin, Y., & Mao, R. (2004). Mining frequent
patterns without candidate generation: A frequent-pat-
tern tree approach. Data Mining and Knowledge Discov-
ery, 8(1), 53-87.
Han, J., Russ, B., Kumar, V., Mannila, H., & Pregibon, D.
(2002). Emerging scientific applications in data mining.
Communications of the ACM, 45(8), 54-58.
Lin, D., & Kedem, Z.M. (1998). P INCER-SEARCH: A new
algorithm for discovering the maximum frequent set. Pro-
ceedings of the EBDT Conference.
Ordonez, C. et al. (2001). Mining constrained association
rules to predict heart disease. Proceedings of the ICDM
Conference.
Pasquier, N., Bastide, Y., Taouil, R., & Lakhal, L. (1998).
Pruning closed itemset lattices for association rules. Pro-
ceedings of the BDA Conference.
Pasquier, N., Bastide, Y., Taouil, R., & Lakhal, L. (1999a).
Efficient mining of association rules using closed itemset
lattices. Information Systems, 24(1), 25-46.
Pasquier N., Bastide, Y., Taouil, R., & Lakhal, L. (1999b).
Discovering frequent closed itemsets for association
rules. Proceedings of the ICDT Conference.
Pasquier, N., Bastide, Y., Taouil, R., & Lakhal, L. (1999c).
Closed set based discovery of small covers for associa-
tion rules. Proceedings of the BDA Conference.
Pasquier, N., Taouil, R., Bastide, Y., Stumme, G., & Lakhal,
L. (2004). Generating a condensed representation for
association rules. Journal of Intelligent Information
Systems.
Pfaltz J., & Taylor C. (2002, July). Closed set mining of biological
data. Proceedings of the KDD/BioKDD Conference.
Silverstein, C., Brin, S., & Motwani, R. (1998). Beyond
market baskets: Generalizing association rules to de-
pendence rules. Data Mining and Knowledge Discov-
ery, 2(1), 39-68.
Stumme, G., Taouil, R., Bastide, Y., Pasquier, N., &
Lakhal, L. (2002). Computing iceberg concept lattices with
TITANIC. Data and Knowledge Engineering, 42(2), 189-
222.
TEAM LinG
Mining Association Rules Using Frequent Closed Itemsets
Wang, J., & Han, J. (2004). BIDE: Efficient mining of
frequent closed sequences. Proceedings of the ICDE
Conference.
Wang, J., Han, J., & Pei, J. (2003). CLOSET+: Searching for
the best strategies for mining frequent closed itemsets.
Proceedings of the KDD Conference.
Yang, H., & Parthasarathy, S. (2002). On the use of con-
strained associations for Web log mining. Proceedings of
the KDD/WebKDD Conference.
Zaki, M.J. (2000). Generating non-redundant association
rules. Proceedings of the KDD Conference.
Zaki, M.J., & Hsiao, C.-J. (2002). CHARM: An efficient
algorithm for closed itemset mining. Proceedings of the
SIAM International Conference on Data Mining.
Zaki, M.J., & Ogihara, M. (1998). Theoretical foundations
of association rules. Proceedings of the SIGMOD/DMKD
Workshop.
KEY TERMS
M
Association Rules: An implication rule between two
itemsets with statistical measures of range (support) and
precision (confidence).
Basis for Association Rules: A set of association
rules that is minimal with respect to some criteria and from
which all association rules can be deduced with support
and confidence.
Closed Itemset: An itemset that is a maximal set of
items common to a set of objects. An itemset is closed if
it is equal to the intersection of all objects containing it.
Frequent Itemset: An itemset contained in a number
of objects at least equal to some user-defined threshold.
Itemset: A set of binary attributes, each correspond-
ing to an attribute value or an interval of attribute values.
757
TEAM LinG
758

Mining Chat Discussions

Stanley Loh
Catholic University of Pelotas, Brazil, and Lutheran University of Brasil, Brazil

Daniel Licthnow
Catholic University of Pelotas, Brazil

Thyago Borges
Catholic University of Pelotas, Brazil

Tiago Primo
Catholic University of Pelotas, Brazil

Rodrigo Branco Kickhfel

Catholic University of Pelotas, Brazil

Gabriel Simes
Catholic University of Pelotas, Brazil

Gustavo Piltcher
Catholic University of Pelotas, Brazil

Ramiro Saldaa
Catholic University of Pelotas, Brazil

INTRODUCTION cussed and then offer new information or can remember

According to Nonaka and Takeuchi (1995), the majority
of the organizational knowledge comes from interac-
tions between people. People tend to reuse solutions
from other persons in order to gain productivity.
When people communicate to exchange information
or acquire knowledge, the process is named Collabora-
tion. Collaboration is one of the most important tasks
for innovation and competitive advantage within learn-
ing organizations (Senge, 2001). It is important to
record knowledge to later reuse and analysis. If knowl-
edge is not adequately recorded, organized and retrieved,
the consequence is re-work, low productivity and lost of
opportunities.
Collaboration may be realized through synchronous
interactions (e.g., exchange of messages in a chat), asyn-
chronous interactions (e.g., electronic mailing lists or
forums), direct contact (e.g., two persons talking) or indi-
rect contact (when someone stores knowledge and others
can retrieve this knowledge in a remote place or time).
In special, chat rooms are becoming important tools
for collaboration among people and knowledge exchange.
Intelligent software systems may be integrated into chat
rooms in order to help people in this collaboration task.
For example, systems can identify the theme being dis-
people of existing information sources. This kind of sys-
tems is named recommender systems.
Furthermore, chat sessions have implicit knowledge
about what the participants know and how they are view-
ing the world. Analyzing chat discussions allows under-
standing what people are looking for and how people
collaborates one with each other. Intelligent software
systems can analyze discussions in chats to extract
knowledge about the group or about the subject being
discussed.
Mining tools can analyze chat discussions to under-
stand what is being discussed and help people. For ex-
ample, a recommender system can analyze textual mes-
sages posted in a web chat, identify the subject of the
discussion and then look for items stored in a Digital
Library to recommend individually to each participant of
the discussion. Items can be electronic documents, web
pages and bibliographic references stored in a digital
library, past discussions and authorities (people with
expertise in the subject being discussed). Besides that,
mining tools can analyze the whole discussion to map the
knowledge exchanged among the chat participants.
The benefits of such technology include supporting learn-
ing environments, knowledge management efforts within
organizations, advertisement and support to decisions.

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
Mining Chat Discussions
BACKGROUND
Some works has investigated the analysis of online
discussions. Brutlag and Meek (2000) have studied the
identification of themes in e-mails. The work compares
the identification by analyzing only the subject of the e-
mails against analyzing the message bodies. One con-
clusion is that e-mail headers perform so well as mes-
sage bodies, with the additional advantage of reducing
the number of features to be analyzed.
Busemann, Schmeier and Arens, (2000) investigated
the special case of messages registered in call centers.
The work proved possible to identify themes in this kind
of message, although the informality of the language used
in the messages. This informality causes mistakes due to
jargons, misspellings and grammatical inaccuracy.
The work of Durbin, Richter, and Warner (2003) has
shown possible to identify affective opinions about
products and services in e-mails sent by customers, in
order to alert responsible people or to evaluate the
organization and customers satisfaction. Furthermore,
the work identifies the intensity of the rating, allowing
the separation of moderate or intensive opinions.
Tong (2001) investigated the analysis of online dis-
cussions about movies. Messages represent comments
about movies. This work proved to be feasible to find
positive and negative opinions, by analyzing key or cue
words. Furthermore, the work also extracts information
about the movies, like directors and actors, and then
examines opinions about these particular characteristics.
The only work found in the scientific literature that
analyzes chat messages is the one from Khan, Fisher,
Shuler, Wu, and Pottenger (2002). They apply mining
techniques over chat messages in order to find social
interactions among people. The goal is to find who is
related to whom inside a specific area, by analyzing the
exchange of messages in a chat and the subject of the
discussion.
MAIN THRUST
Following, the chapter explains how messages can be
mined, how recommendations can be made and how the
whole discussion (an entire chat session) can be analyzed.
Identifying Themes in Chat Messages
To provide people with useful information during a
collaboration session, the system has to identify what is
being discussed. Textual messages sent by the users in
the chat can be analyzed for this purpose. Texts can lead
to the identification of the subject discussed because
the words and the grammar present in the texts represent
knowledge from people, expressed in written formats M
(Sowa, 2000).
An ontology or thesaurus can be used to help to
identify cue words for each subject. The ontology or
thesaurus has concepts of a domain or knowledge area,
including relations between concepts and the terms
used in written languages to express these concepts
(Gilchrist, 2003). The ontology can be created by ma-
chine learning methods (supervised learning), where
human experts select training cases for each subject
(e.g., texts of positive and negative examples) and an
intelligent software system identifies the keywords that
define each subject. The TFIDF method from Salton and
McGill (1983) is the most used in this kind of task.
If considering that the terms that compose the mes-
sages compose a bag of words (have no difference in
importance), probabilistic techniques can be used to
identify the subject. By other side, natural language
processing techniques can identify syntactic elements
and relations, then supporting more precise subject
identification.
The identification of themes should consider the
context of the messages to determine if the concept
identified is really present in the discussion. A group of
messages is better to infer the subject than a single
message. That avoids misunderstandings due to words
ambiguity and use of synonyms.
Making Recommendations in a Chat
Discussion
A recommender system is a software whose main goal is
to aid in the social collaborative process of indicating
or receiving indications (Resnick & Varian, 1997).
Recommender systems are broadly used in electronic
commerce for suggesting products or providing infor-
mation about products and services, helping people to
decide in the shopping process (Lawrence et al., 2001;
Schafer et al., 2001). The offered gain is that people do
not need to request recommendation or to perform a
query over an information base, but the system decides
what and when to suggest. The recommendation is usu-
ally based on user profiles and reuse of solutions.
When a subject is identified in a message, the
recommender searches for items classified in this sub-
ject. Items can come from different databases. For
example, a Digital Library may provide electronic docu-
ments, links to Web pages and bibliographic references.
A profile database may contain information about
people, including the interest areas of each person, as
well an associated degree, informing the users knowl-
edge level on the subject or how much is his/her compe-
tence in the area (his/her expertise). This can be used to
759
TEAM LinG

indicate the most active user in the area or who is the
authority in the subject.
A database of past discussions records everything
that occurs in the chat, during every discussion session.
Discussions may be stored by sessions, identified by
data and themes discusses and can include who partici-
pated in the session, all the messages exchanged (with a
label indicating who sent it), the concept identified in
each message, the recommendations made during the
session for each user and documents downloaded or read
during the session. Past discussions may be recom-
mended during a chat session, remembering the partici-
pants that other similar discussions have already hap-
pened. This database also allows users to review the
whole discussion later after the session. The great ben-
efit is that users do not re-discuss the same question.
Mining a Chat Session
Analyzing the themes discussed in a chat session can
bring an important overview of the discussion and also of
the subject. Statistical tools applied over the messages
sent and the subjects identified in each message can help
users to understand which were the themes more dis-
cussed. Counting the messages associated with each
subject, it is possible to infer the central point of the
discussion and the peripheral themes.
The list of subjects identified during the chat session
compose an interesting order, allowing users to analyze
the path followed by the participants during the discus-
sion. For example, it is possible to observe which was the
central point of the discussion, whether the discussion
deviated from the main subject and whether the subjects
present in the beginning of the discussion were also
present at the end. The coverage of the discussion may be
identified by the number of different themes discussed.
Furthermore, this analysis allows identifying the depth
of the discussion, that is, whether more specific themes
were discussed or whether the discussion occurred su-
perficially at a higher conceptual level.
Analyzing the messages sent by every participant
allows determining the degree of participation of each
person in the discussion: who participated more and who
did less. Furthermore, it is possible to observe which are
the interesting areas for each person and in someway to
determine the expertise of the group and of the partici-
pants (which are the areas where the group is more
competent).
Association techniques can be used to identify corre-
lations between themes or between themes and persons.
For example, it is possible to find that some theme is
present always when other theme is also present or to find
that every discussion where some person participated had
a certain theme as the principal.
760
Mining Chat Discussions
FUTURE TRENDS
Recommender systems are still an emerging area. There
are some doubts and open issues. For example, whether
is good or bad to recommend items already suggested
in past discussions (re-recommend, as if remembering
the person). Besides that it is important to analyze the
level of the participants in order to recommend only
basic or advanced items.
Collaborative filtering techniques can be used to
recommend items already seen by other users (Resnick,
et al., 1994; Terveen & Hill, 2001). Grouping people
with similar characteristics allows for crossing of
recommended items, for example, to offer documents
read by one person to others.
In the same way, software systems can capture
relevance feedback from users to narrow the list of
recommendations. Users should read some items of
the list and rate them, so that the system can use this
information to eliminate items from the list or to
reorder the items in a new ranking.
The context of the messages needs to be more
studied. To infer the subject being discussed, the sys-
tem can analyze a group of messages, but it is necessary
to determine how many (a fixed number or all messages
sent in the past N minutes?).
An orthographic corrector is necessary to clean the
messages posted to the chat. Lots of linguistic mis-
takes are expected since people are using chats in a
hurry, with little attention to the language, without
revisions and in an informal way. Furthermore, the text
mining tools must analyze special signs like novel
abbreviations, emoticons and slang expressions. Spe-
cial words may be added to the domain ontology in
order to hold the differences in the language.
CONCLUSION
An example of such a system discussed in this chapter
is available in http://gpsi.ucpel.tche.br/sisrec. Cur-
rently, the system uses a domain ontology for com-
puter science, but others can be used. Similarly, the
current digital library only has items related to Com-
puter Science.
The recommendation system facilitates the organi-
zational learning because people receive suggestions
of information sources during online discussions. The
main advantage of the system is to free the user of the
burden to search information sources during the online
discussion. Users do not have to choose attributes or
requirements from a menu of options, in order to retrieve
items of a database; the system decides when and what
information to recommend to the user. This proactive
TEAM LinG
Mining Chat Discussions
approach is useful for non-experienced users that receive
hits about what to read in a specific subject. Users
information needs are discovered naturally during the
conversation.
Furthermore, when the system indicates people who
are authorities in each subject, nave users can meet
these authorities for getting more knowledge.
Other advantage of the system is that part of the
knowledge shared in the discussion can be made explicit
through the record of the discussion for future retrieval.
Besides that, the system allows the posterior analysis of
each discussion, presenting the subjects discussed, the
messages exchanged, the items recommended and the
order in which the subjects were discussed.
An important feature is the statistical analysis of the
discussion, allowing understanding the central point,
the peripheral themes, the order of the discussion, its
coverage and depth.
The benefit of mining chat sessions is of special
interest for Knowledge Management efforts. Organiza-
tions can store tacit knowledge formatted as discus-
sions. The discussions can be retrieved, so that knowl-
edge can be reused. In the same way, the contents of a
Digital Library (or Organizational Memory) can be bet-
ter used through recommendations. People do not have
to search for contents neither to remember items in
order to suggest to others. Recommendations play this
role in a proactive way, examining what people are
discussing and users profiles and selecting interesting
new contents.
In special, such systems (that mine chat sessions)
can be used in e-learning environments, supporting the
construction of knowledge by individuals or groups.
Recommendations help the learning process, suggest-
ing complementary contents (documents and sites stored
in the Digital Library). Recommendations also include
authorities in topics being discussed, that is, people
with high degrees of knowledge.
ACKNOWLEDGMENTS
This research group is partially supported by CNPq, an
entity of the Brazilian government for scientific and
technological development.
REFERENCES
Brutlag, J.D., & Meek, C. (2000). Challenges of the email
domain for text classification. In Proceedings of the 7th
International Conference on Machine Learning (ICML
2000) (pp. 103-110), Stanford University, Stanford, CA,
USA.
Busemann, S., Schmeier, S., & Arens, R.G. (2000) Message
classification in the call center. In Proceedings of the M
Applied Natural Language Processing Conference
ANLP2000 (pp. 159-165), Seattle, WA.
Durbin, S.D., Richter, J.N., & Warner, D. (2003). A system
for affective rating of texts. In Proceedings of the 3rd
Workshop on Operational Text Classification, 9th ACM
International Conference on Knowledge Discovery and
Data Mining (KDD-2003), Washington, DC.
Khan, F.M., Fisher, T.A., Shuler, L., Wu, T., & Pottenger,
W. M. (2002). Mining chat-room conversations for social
and semantic interactions. Technical Report LU-CSE-02-
011, Lehigh University, Bethlehem, Pennsylvania, USA.
Gilchrist, A. (2003). Thesauri, taxonomies and ontologies
an etymological note. Journal of Documentation, 59(1),
7-18.
Lawrence, R.D. et al. (2001). Personalization of supermar-
ket product recommendations. Journal of Data Mining
and Knowledge Discovery, 5(1/2), 11-32.
Nonaka, I., & Takeuchi, T. (1995). The knowledge-creating
company: How Japanese companies create the dynamics of
innovation. Cambridge: Oxford University Press.
Resnick, P. et al. (1994). GroupLens: An open architecture
for collaborative filtering of Netnews. In Proceedings of
the Conference on Computer Supported Cooperative
Work (pp. 175-186).
Resnick, P., & Varian, H. (1997). Recommender systems.
Communications of the ACM, 40(3), 56-58.
Salton, G., & McGill, M.J. (1983). Introduction to modern
information retrieval. New York: McGraw-Hill.
Schafer, J.B. et al. (2001). E-commerce recommendation
applications. Journal of Data Mining and Knowledge
Discovery, 5(1/2), 115-153.
Senge, P.M. (2001). The fifth discipline: The art and
practice of the learning organization (9th ed.). So
Paulo: Best Seller (in Portuguese).
Sowa, J.F. (2000). Knowledge representation: Logical,
philosophical, and computational foundations. Pacific
Grove, CA: Brooks/Cole Publishing Co.
Terveen, L., & Hill, W. (2001). Human-computer col-
laboration in recommended systems. In J. Carroll (Ed.),
Human computer interaction in the new millennium.
Boston: Addison-Wesley.
Tong, R. (2001). Detecting and tracking opinions in online
discussions. In Proceedings of the Workshop on Opera-
tional Text Classification, SIGIR, New Orleans, Louisi-
ana, USA.
761
TEAM LinG

KEY TERMS
Chat: A software system that enables real-time com-
munication among users through the exchange of tex-
tual messages.
Collaboration: The process of communication
among people with the goal of sharing information and
knowledge.
Digital Library: A set of electronic resources (usually
documents) combined with a software system which al-
lows storing, organizing and retrieving the resources.
Knowledge Management: Systems and methods for
storing, organizing and retrieving explicit knowledge.
762
Mining Chat Discussions
Mining: The application of statistical techniques to
infer implicit patterns or rules in a collection of data, in
order to discover new and useful knowledge.
Ontology: A formal and explicit definition of concepts
(classes or categories) and their attributes and relations.
Recommendations: Results of the process of provid-
ing useful resources to a user, like products, services or
information.
Recommender System: A software system that makes
recommendations to a user, usually analyzing the users
interest or need.
Text Mining: The process of discovering new infor-
mation analyzing textual collections.
TEAM LinG

Mining Data with Group Theoretical Means
Gabriele Kern-Isberner
University of Dortmund, Germany
INTRODUCTION
Knowledge discovery refers to the process of extract-
ing new, interesting, and useful knowledge from data and
presenting it in an intelligible way to the user. Roughly,
knowledge discovery can be considered a three-step
process: preprocessing data; data mining, in which the
actual exploratory work is done; and interpreting the
results to the user. Here, I focus on the data-mining step,
assuming that a suitable set of data has been chosen
properly.
The patterns that we search for in the data are plau-
sible relationships, which agents may use to establish
cognitive links for reasoning. Such plausible relation-
ships can be expressed via association rules. Usually,
the criteria to judge the relevance of such rules are
either frequency based (Bayardo & Agrawal, 1999) or
causality based (for Bayesian networks, see Spirtes,
Glymour, & Scheines, 1993). Here, I will pursue a
different approach that aims at extracting what can be
regarded as structures of knowledge relationships
that may support the inductive reasoning of agents and
whose relevance is founded on information theory. The
method that I will sketch in this article takes numerical
relationships found in data and interprets these relation-
ships as structural ones, using mostly algebraic tech-
niques to elaborate structural information.
BACKGROUND
Common sense and expert knowledge is most generally
expressed by rules, connecting a precondition and a
conclusion by an if-then construction. For example, you
avoid puddles on sidewalks because you are aware of the
fact that if you step into a puddle, then your feet might
get wet; similarly, a physician would likely expect a
patient showing the symptoms of fever, headache, and a
sore throat to suffer from a flu, basing his diagnosis on
the rule that if a patient has a fever, headache, and sore
throat, then the ailment is a flu, equipped with a suffi-
ciently high probability.
If-then rules are more formally denoted as condi-
tionals. The crucial point with conditionals is that they
carry generic knowledge that is applicable to different
situations. This fact makes them most interesting ob-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
763
M
jects in artificial intelligence, in a theoretical as well as
in a practical respect. For instance, a sales assistant who
has a general knowledge about the preferences of his or
her customers can use this knowledge when consulting
any new customer.
Typically, two central problems have to be solved in
practical applications: First, where do the rules come
from? How can they be extracted from statistical data?
And second, how should rules be represented? How
should conditional knowledge be propagated and com-
bined for further inferences? Both of these problems
can be dealt with separately, but it is most rewarding to
combine them, that is, to discover rules that are most
relevant with respect to some inductive inference for-
malism and to build up the best model from the discov-
ered rules that can be used for queries.
MAIN THRUST
This article presents an approach to discover associa-
tion rules that are most relevant with respect to the
maximum entropy methods. Because entropy is related
to information, this approach can be considered as
aiming to find the most informative rules in data. The
basic idea is to exploit numerical relationships that are
observed by comparing (relative) frequencies, or ratios
of frequencies, and so forth, as manifestations of inter-
actions of underlying conditional knowledge.
My approach differs from usual knowledge discov-
ery and data-mining methods in various respects:
It explicitly takes the instrument of inductive in-
ference into consideration.
It is based on statistical information but not on
probabilities close to 1; actually, it mostly uses
only structural information obtained from the data.
It is not based on observing conditional indepen-
dencies (as for learning causal structures), but
aims at learning relevant conditional dependen-
cies in a nonheuristic way.
As a further novelty, it does not compute single,
isolated rules, but yields a set of rules by taking
into account highly complex interactions of rules.
Zero probabilities computed from data are inter-
preted as missing information, not as certain
knowledge.
TEAM LinG

The resulting set of rules may serve as a basis for
maximum entropy inference. Therefore, the method
described in this article addresses minimality aspects,
as in Padmanabhan and Tuzhilin (2000), and makes use
of inference mechanisms, as in Cristofor and Simovici
(2002). Different from most approaches, however, it
exploits the inferential power of the maximum entropy
methods in full consequence and in a structural,
nonheuristic way.
Modelling Conditional Knowledge by
Maximum Entropy (ME)
Suppose a set R* = {(B1|A1)[x1], , (Bn|An)[xn]} of
probabilistic conditionals is given. For instance, R*
may describe the knowledge available to a physician
when he has to make a diagnosis. Or R* may express
common sense knowledge, such as Students are young
with a probability of (about) 80% and Singles (i.e.,
unmarried people) are young with a probability of (about)
70%, the latter knowledge being formally expressed by
R* = { (young|student)[0.8], (young|single)[0.7] }.
Usually, these rule bases represent incomplete
knowledge, in that a lot of probability distributions are
apt to represent them. So learning or inductively repre-
senting the rules, respectively, means to take them as a
set of conditional constraints and to select a unique
probability distribution as the best model that can be
used for queries and further inferences. Paris (1994)
investigates several inductive representation techniques
in a probabilistic framework and proves that the prin-
ciple of maximum entropy (ME-principle) yields the
only method to represent incomplete knowledge in an
unbiased way, satisfying a set of postulates describing
sound common sense reasoning. The entropy H(P) of a
probability distribution P is defined as
H(P) = - w P(w) log P(w),
where the sum is taken over all possible worlds, w, and
measures the amount of indeterminateness inherent to
P. Applying the principle of maximum entropy, then,
means to select the unique distribution P* = ME(R*)
that maximizes H(P) among all distributions P that
satisfy the rules in R*. In this way, the ME-method
ensures that no further information is added, so the
knowledge R* is represented most faithfully.
Indeed, the ME-principle provides a most conve-
nient and founded method to represent incomplete
probabilistic knowledge (efficient implementations of
ME-systems are described in Roedder & Kern-Isberner,
2003). In an ME-environment, the expert has to list only
whatever relevant conditional probabilities he or she is
aware of. Furthermore, ME-modelling preserves the
764
Mining Data with Group Theoretical Means
generic nature of conditionals by minimizing the amount
of information being added, as shown in Kern-Isberner
(2001).
Nevertheless, modelling ME-rule bases has to be
done carefully so as to ensure that all relevant depen-
dencies are taken into account. This task can be difficult
and troublesome. Usually, the modelling rules are based
somehow on statistical data. So, a method to compute rule
sets appropriate for ME-modelling from statistical data is
urgently needed.
Structures of Knowledge
The most typical approach to discover interesting rules
from data is to look for rules with a significantly high
(conditional) probability and a concise antecedent
(Bayardo & Agrawal, 1999; Agarwal, Aggarwal, & Prasad,
2000; Fayyad & Uthurusamy, 2002; Coenen,
Goulbourne, & Leng, 2001). Basing relevance on fre-
quencies, however, is sometimes unsatisfactory and
inadequate, particularly in complex domains such as
medicine. Further criteria to measure the interesting-
ness of the rules or to exclude redundant rules have also
been brought forth (Jaroszewicz & Simovici, 2001;
Bastide, Pasquier, Taouil, Stumme, & Lakhal, 2000;
Zaki, 2000). Some of these algorithms also make use of
optimization criteria, which are based on entropy
(Jaroszewicz & Simovici, 2002).
Mostly, the rules are considered as isolated pieces
of knowledge; no interaction between rules can be taken
into account. In order to obtain more structured infor-
mation, one often searches for causal relationships by
investigating conditional independencies and thus
noninteractivity between sets of variables (Spirtes et
al., 1993).
Although causality is undoubtedly most important
for human understanding, the concept seems to be too
rigid to represent human knowledge in an exhaustive way.
For instance, a person suffering from a flu is certainly sick
(P(sick | flu) = 1), and he or she often will complain about
headaches (P(headache | flu) = 0.9). Then you have
P(headache | flu) = P(headache | flu & sick), but you would
surely expect that P(headache | not flu) is different from
P(headache | not flu & sick)! Although the first equality
suggests a conditional independence between sick and
headache, due to the causal dependency between head-
ache and flu, the second inequality shows this to be (of
course) false. Furthermore, a physician might also state
some conditional probability involving sickness and head-
ache, so you obtain a complex network of rules. Each of
these rules will be considered relevant by the expert, but
none will be found when searching for conditional inde-
pendencies! So what, exactly, are the structures of knowl-
edge by which conditional dependencies (not indepen-
TEAM LinG
Mining Data with Group Theoretical Means
dencies! See also Simovici, Cristofor, D., & Cristofor, L.,
2000) manifest themselves in data?
To answer this question, the theory of conditional
structures has been presented in Kern-Isberner (2000).
Conditional structures are an algebraic means to make
the effects of conditionals on possible worlds (i.e.,
possible combinations or situations) transparent, in that
they reflect whether the corresponding world verifies
the conditional or falsifies it, or whether the conditional
cannot be applied to the world because the if-condition is
not satisfied. Consider, for instance, the conditional If
you step in a puddle, then your feet might get wet. In a
particular situation, the conditional is applicable (you
actually step into a puddle) or not (you simply walk
around it), and it can be found verified (you step in a
puddle and indeed, your feet get wet) or falsified (you
step in a puddle, but your feet remain dry because you are
wearing rain boots).
This intuitive idea of considering a conditional as a
three-valued event is generalized in Kern-Isberner (2000)
to handle the simultaneous impacts of a set of condition-
als by using algebraic symbols for positive and negative
impact, respectively. Then for each world, a word of
these symbols can be computed, which shows immedi-
ately how the conditionals interact on this world. The
proper mathematical structure for building words are
(semi)groups, and indeed, group theory provides the basis
for connecting numerical to structural information in an
elegant way. In short, a probability (or frequency) distri-
bution is called (conditionally) indifferent with respect to
a set of conditionals R* iff its numerical information
matches the structural information provided by condi-
tional structures. In particular, each ME-distribution turns
out to be indifferent with respect to a generating set of
conditionals.
Data Mining and Group Theory A
Strange Connection?
The concept of conditional structures, however, is not
only an algebraic means to judge well-behavedness with
respect to conditional information. The link between
numerical and structural information, which is provided
by the concept of conditional indifference, can also be
used in the other direction, that is, to derive structural
information about the underlying conditional relation-
ships from numerical information. More precisely, find-
ing a set of rules with the ability to represent a given
probability distribution P via ME-methods can be done
by elaborating numerical relationships in P, interpreting
them as manifestations of underlying conditional depen-
dencies. The procedure to discover appropriate sets of
rules is sketched in the following steps:
Start with a set B of simple rules, the length of which
is considered to be large enough to capture all M
relevant dependencies.
Search for numerical relationships in P by inves-
tigating which products of probabilities match.
Compute the corresponding conditional struc-
tures with respect to B, yielding equations of
group elements.
Solve these equations by forming appropriate
factor groups.
Building these factor groups corresponds to elimi-
nating and joining the basic conditionals in B to
make their information more concise, in accor-
dance with the numerical structure of P. Actually,
the antecedents of the conditionals in B are short-
ened so as to comply with the numerical relation-
ships in P.
So the basic idea of this algorithm is to start with
long rules and to shorten them in accordance with the
probabilistic information provided by P without losing
information.
Group theory actually provides an elegant frame-
work, on the one hand, to disentangle highly complex
conditional interactions in a systematic way, and on the
other hand, to make operations on the conditionals
computable, which is necessary to make information
more concise.
How to Handle Sparse Knowledge
The frequency distributions calculated from data are
mostly not positive just to the contrary, they would
be sparse, full of zeros, with only scattered clusters of
nonzero probabilities. This overload of zeros is also a
problem with respect to knowledge representation,
because a zero in such a frequency distribution often
merely means that such a combination has not been
recorded. The strict probabilistic interpretation of zero
probabilities, however, is that such a combination does
not exist, which does not seem to be adequate.
The method sketched in the preceding section is
also able to deal with that problem in a particularly
adequate way: The zero values in frequency distribu-
tions are taken to be unknown but equal probabilities,
and this fact can be exploited by the algorithm. So they
actually help to start with a tractable set B of rules right
from the beginning (see also Kern-Isberner & Fisseler,
2004).
In summary, zeros occurring in the frequency dis-
tribution computed from data are considered as miss-
ing information, and in my algorithm, they are treated
as non-knowledge without structure.
765
TEAM LinG

FUTURE TRENDS
Although by and large, the domain of knowledge discov-
ery and data mining is dominated by statistical tech-
niques and the problem of how to manage vast amounts
of data, the increasing need for and popularity of human-
machine interactions will make it necessary to search
for more structural knowledge in data that can be used to
support (humanlike) reasoning processes. The method
described in this article offers an approach to realize
this aim. The conditional relationships that my algo-
rithm reveals can be considered as kind of cognitive
links of an ideal agent, and the ME-technology takes the
task of inductive reasoning to make use of this knowl-
edge. Combined with clustering techniques in large
databases, for example, it may turn out a useful method
to discover relationships that go far beyond the results
provided by other, more standard data-mining techniques.
CONCLUSION
In this article, I have developed a new method for discov-
ering conditional dependencies from data. This method
is based on information-theoretical concepts and group-
theoretical techniques, considering knowledge discov-
ery as an operation inverse to inductive knowledge
representation. By investigating relationships between
the numerical values of a probability distribution P, the
effects of conditionals are analyzed and isolated, and
conditionals are joined suitably so as to fit the knowl-
edge structures inherent to P.
REFERENCES
Agarwal, R. C., Aggarwal, C. C., & Prasad, V. V. V.
(2000). Depth first generation of long patterns. Pro-
ceedings of the Sixth ACM-SIGKDD International
Conference on Knowledge Discovery and Data Mining
(pp. 108-118).
Bastide, Y., Pasquier, N., Taouil, R., Stumme, G. & Lakhal,
L. (2000). Mining minimal non-redundant association rules
using frequent closed itemsets. Proceedings of the First
International Conference on Computational Logic (pp.
972-986).
Bayardo, R. J., & Agrawal, R. (1999). Mining the most
interesting rules. Proceedings of the Fifth ACM SIGKDD
International Conference on Knowledge Discovery
and Data Mining.
766
Mining Data with Group Theoretical Means
Coenen, F., Goulbourne, G., & Leng, P. H. (2001). Comput-
ing association rules using partial totals. Proceedings of
the Fifth European Conference on Principles and Prac-
tice of Knowledge Discovery in Databases (pp. 54-66).
Cristofor, L., & Simovici, D. (2002). Generating an informa-
tive cover for association rules. Proceedings of the IEEE
International Conference on Data Mining (pp. 597-600).
Fayyad, U., & Uthurusamy, R. (2002). Evolving data
mining into solutions for insights. Communications of
the ACM, 45(8), 28-61.
Jaroszewicz, S., & Simovici, D. A. (2001). A general mea-
sure of rule interestingness. Proceedings of the Fifth
European Conference on Principles and Practice of
Knowledge Discovery in Databases (pp. 253-265).
Jaroszewicz, S., & Simovici, D. A. (2002). Pruning redun-
dant association rules using maximum entropy principle.
Proceedings of the Pacific-Asia Conference on Knowl-
edge Discovery and Data Mining.
Kern-Isberner, G. (2000). Solving the inverse representa-
tion problem. Proceedings of the 14th European Confer-
ence on Artificial Intelligence (pp. 581-585).
Kern-Isberner, G. (2001). Conditionals in nonmonotonic
reasoning and belief revision. Lecture Notes in Artifi-
cial Intelligence.
Kern-Isberner, G., & Fisseler, J. (2004). Knowledge
discovery by reversing inductive knowledge representa-
tion. Proceedings of the Ninth International Confer-
ence on the Principles of Knowledge Representation
and Reasoning.
Padmanabhan, B., & Tuzhilin, A. (2000). Small is beau-
tiful: Discovering the minimal set of unexpected pat-
terns. Proceedings of the Sixth ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and Data
Mining (pp. 54-63).
Paris, J. B. (1994). The uncertain reasoners companion:
A mathematical perspective. Cambridge University Press.
Roedder, W., & Kern-Isberner, G. (2003). From informa-
tion to probability: An axiomatic approach. International
Journal of Intelligent Systems, 18(4), 383-403.
Simovici, D.A., Cristofor, D., & Cristofor, L. (2000). Min-
ing for purity dependencies in databases (Tech. Rep. No.
00-2). Boston: University of Massachusetts.
Spirtes, P., Glymour, C., & Scheines, R.. (1993). Causation,
prediction and search. Lecture Notes in Statistics, 81.
TEAM LinG
Mining Data with Group Theoretical Means
Zaki, M. J. (2000). Generating non-redundant association
rules. Proceedings of the Sixth ACM-SIGKDD Interna-
tional Conference on Knowledge Discovery and Data
Mining (pp. 34-43).
KEY TERMS
Conditional: The formal algebraic term for a rule
that need not be strict, but also can be based on plausi-
bility, probability, and so forth.
Conditional Independence: A generalization of
plain statistical independence that allows you to take a
context into account. Conditional independence is of-
ten associated with causal effects.
Conditional Structure: An algebraic expression that
makes the effects of conditionals on possible worlds M
transparent and computable.
Entropy: Measures the indeterminateness inherent to
a probability distribution and is dual to information.
Possible World: Corresponds to the statistical notion
of an elementary event. Probabilities over possible worlds,
however, have a more epistemic, subjective meaning, in
that they are assumed to reflect an agents knowledge.
Principle of Maximum Entropy: A method to complete
incomplete probabilistic knowledge by minimizing the
amount of information added.
Probabilistic Conditional: A conditional that is as-
signed a probability. To match the notation of conditional
probabilities, a probabilistic conditional is written as
(B|A)[x] with the meaning If A holds, then B holds with
probability x.
767
TEAM LinG
768
Mining E-Mail Data
Steffen Bickel
Humboldt-Universitt zu Berlin, Germany
Tobias Scheffer
Humboldt-Universitt zu Berlin, Germany
INTRODUCTION
E-mail has become one of the most important communica-
tion media for business and private purposes. Large
amounts of past e-mail records reside on corporate serv-
ers and desktop clients. There is a huge potential for
mining this data. E-mail filing and spam filtering are well-
established e-mail mining tasks. E-mail filing addresses
the assignment of incoming e-mails to predefined catego-
ries to support selective reading and organize large e-mail
collections. First research on e-mail filing was conducted
by Green and Edwards (1996) and Cohen (1996). Pantel
and Lin (1998) and Sahami, Dumais, Heckerman, and
Horvitz (1998) first published work on spam filtering. Here,
the goal is to filter unsolicited messages. Recent research
on e-mail mining addresses automatic e-mail answering
(Bickel & Scheffer, 2004) and mining social networks from
e-mail logs (Tyler, Wilkinson, & Huberman, 2004).
In Section Background we will categorize common e-
mail mining tasks according to their objective, and give an
overview of the research literature. Our Main Thrust
Section addresses e-mail mining with the objective of
supporting the message creation process. Finally, we
discuss Future Trends and conclude.
BACKGROUND
There are two objectives for mining e-mail data: support-
ing communication and discovering hidden properties of
communication networks.
Support of Communication
The problems of filing e-mails and filtering spam are text
classification problems. Text classification is a well stud-
ied research area; a wide range of different methods is
available. Most of the common text classification algo-
rithms have been applied to the problem of e-mail classi-
fication and their performance has been compared in
several studies. Because publishing an e-mail data set
involves disclosure of private e-mails, there are only a
small number of standard e-mail classification data sets.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Since there is no study that compares large numbers of
data sets, different classifiers and different types of
extracted features, it is difficult to judge which text clas-
sifier performs best specifically for e-mail classification.
Against this background we try to draw some conclu-
sions on the question which is the best text classifier for
e-mail. Cohen (1996) applies rule induction to the e-mail
classification problem and Provost (1999) finds that Nave
Bayes outperforms rule induction for e-mail filing. Nave
Bayes classifiers are widely used for e-mail classification
because of their simple implementation and low computa-
tion time (Pantel & Lin, 1998; Rennie, 2000; Sahami,
Dumais, Heckerman, & Horvitz, 1998). Joachims (1997,
1998) shows that Support Vector Machines (SVMs) are
superior to the Rocchio classifier and Nave Bayes for
many text classification problems. Drucker, Wu, and Vapnik
(1999) compares SVM with boosting on decision trees.
SVM and boosting show similar performance but SVM
proves to be much faster and has a preferable distribution
of errors.
The performance of an e-mail classifier is dependent
on the extraction of appropriate features. Joachims (1998)
shows that applying feature selection for text classifica-
tion with SVM does not improve performance. Hence,
using SVM one can bypass the expensive feature selec-
tion process and simply include all available features.
Features that are typically used for e-mail classification
include all tokens in the e-mail body and header in bag-of-
words representation using TF- or TFIDF-weighting.
HTML tags and single URL elements also provide useful
information (Graham, 2003).
Boykin and Roychowdhury (2004) propose a spam
filtering method that is not based on text classification but
on graph properties of message sub-graphs. All addresses
that appear in the headers of the inbound mails are graph
nodes; an edge is added between all pairs of addresses
that jointly appear in at least one header. The resulting
sub-graphs exhibit graph properties that differ signifi-
cantly for spam and non-spam sub-graphs. Based on this
finding black- and whitelists can be constructed for
spam and non-spam addresses. While this idea is appeal-
ing, it should be noted that the approach is not immedi-
ately practical since most headers of spam e-mails do not
TEAM LinG
Mining E-Mail Data
contain other spam recipients addresses, and most send-
ers addresses are used only once.
Additionally, the semantic e-mail approach
(McDowell, Etzioni, Halevy, & Levy, 2004) aims at sup-
porting communication by allowing automatic e-mail pro-
cessing and facilitating e-mail mining; it is the equivalent
of semantic web for e-mail. The goal is to make e-mails
human- and machine-understandable with a standardized
set of e-mail processes. Each e-mail has to follow a
standardized process definition that includes specific
process relevant information. An example for a semantic
e-mail process is meeting coordination. Here, the indi-
vidual process tasks (corresponding to single e-mails) are
issuing invitations and collecting responses. In order to
work, semantic e-mail would require a global agreement
on standardized semantic processes, special e-mail cli-
ents and training for all users. Additional mining tasks for
support of communication are automatic e-mail answering
and sentence completion. They are described in Section
Main Thrust.
Discovering Hidden Properties of
Communication Networks
E-mail communication patterns reveal much information
about hidden social relationships within organizations.
Conclusions about informal communities and informal
leadership can be drawn from e-mail graphs. Differences
between informal and formal structures in business orga-
nizations can provide clues for improvement of formal
structures which may lead to enhanced productivity. In
the case of terrorist networks, the identification of com-
munities and potential leaders is obviously helpful as
well. Additional potential applications lie in marketing,
where companies especially communication providers
can target communities as a whole.
In social science, it is common practice for studies on
electronic communication within organizations to derive
the network structure by means of personal interviews or
surveys (Garton Garton, Haythornthwaite, & Wellman,
1997; Hinds & Kiesler, 1995). For large organizations, this
is not feasible. Building communication graphs from e-
mail logs is a very simple and accurate alternative pro-
vided that the data is available. Tyler, Wilkinson, and
Huberman (2004) derive a network structure from e-mail
logs and apply a divisive clustering algorithm that decom-
poses the graph into communities. Tyler, Wilkinson, and
Huberman verify the resulting communities by interview-
ing the communication participants; they find that the
derived communities correspond to informal communities.
Tyler et al. also apply a force-directed spring algorithm
(Fruchterman & Rheingold, 1991) to identify leadership
hierarchies. They find that with increasing distance of
vertices from the spring (center) there is a tendency of
decreasing real hierarchy depth. M
E-mail graphs can also be used for controlling virus
attacks. Ebel, Mielsch, and Bornholdt (2002) show that
vertex degrees of e-mail graphs are governed by power
laws. By equipping the small number of highly connected
nodes with anti-virus software the spreading of viruses
can be prevented easily.
MAIN THRUST
In the last section we categorized e-mail mining tasks
regarding their objective and gave a short explanation on
the single tasks. We will now focus on the ones that we
consider to be most interesting and potentially most
beneficial for users and describe them in greater detail.
These tasks aim at supporting the message creation
process. Many e-mail management systems allow the
definition of message templates that simplify the message
creation for recurring topics. This is a first step towards
supporting the message creation process, but past e-
mails that are available for mining are disregarded. We
describe two approaches for supporting the message
creation process by mining historic data: mining ques-
tion-answer pairs and mining sentences.
Mining Question-Answer Pairs
We consider the problem of learning to answer incoming
e-mails from records of past communication. We focus on
environments in which large amounts of similar answers
to frequently asked questions are sent such as call
centers or customer support departments. In these envi-
ronments, it is possible to manually identify equivalence
classes of answers in the records of outbound communi-
cation. Each class then corresponds to a set of semanti-
cally equivalent answers sent in the past; it depends
strongly on the application context which fraction of the
outbound communication falls into such classes. Map-
ping inbound messages to one of the equivalence classes
of answers is now a multi-class text classification problem
that can be solved with text classifiers.
This procedure requires a user to manually group
previously sent answers into equivalence classes which
can then serve as class labels for training a classifier. This
substantial manual labeling effort reduces the benefit of
the approach. Even though it can be reduced by employ-
ing semi-supervised learning (Nigam, McCallum, Thrun,
& Mitchell, 2000; Scheffer, 2004), it would still be much
preferable to learn from only the available data: stored
inbound and outbound messages. Bickel and Scheffer
(2004) discuss an algorithm that learns to answer ques-
769
TEAM LinG
 Mining E-Mail Data

tions from only the available data and does not require
additional manual labeling. The key idea is to replace the
manual assignment of outbound messages to equivalence
classes by a clustering step.
The algorithms for training (learning from message
pairs) and answering a new question are shown in Table 1.
In the training phase, a clustering algorithm identifies
groups of similar outbound messages. Each cluster then
serves as class label; the corresponding questions which
have been answered by a member of the cluster are used
as training examples for a multi-class text classifier. The
medoid of each cluster (the outbound message closest to
the center) is used as an answer template. The classifier
maps a newly incoming question to one of the clusters; this
clusters medoid is then proposed as answer to the ques-
tion. Depending on the user interface, high confidence
messages might be answered automatically, or an answer
is proposed which the user may then accept, modify, or
reject (Scheffer, 2004).
The approach can be extended in many ways. Multiple
topics in a question can be identified to mix different
corresponding answer templates and generate a multi-
topic answer. Question specific information can be ex-
tracted in an additional information extraction step and
automatically inserted into answer templates. In this ex-
traction step also customer identifications can be ex-
tracted and used for a database lookup that provides
customer and order specific information for generating
more customized answers.
Bickel and Scheffer (2004) analyze the relationship of
answer classes regarding the separability of the corre-
sponding questions using e-mails sent by the service
department of an online shop. By analyzing this relation-
ship one can draw conclusions about the amount of addi-
tional information that is needed for answering specific
types of questions. This information can be visualized in
an inseparability graph, where each class of equivalent
answers is represented by a vertex, and an edge is drawn
when a classifier that discriminates between these classes
achieves only a low AUC performance (the AUC perfor-
mance is the probability that, when a positive and a
negative example are drawn at random, a discriminator
assigns a higher value to the positive than to the nega-
tive one). Typical examples of inseparable answers are
your order has been shipped this morning and your
order will be shipped tomorrow. Intuitively, it is not
possible to predict which of these answers a service
employee will send, based on only the question when
will I receive my shipment?
Mining Sentences
The message creation process can also be supported on
a sentence level. Given an incomplete sentence, the task
of sentence completion is to propose parts or the total
rest of the current sentence, based on an application
specific document collection. A sentence completion
user interface can, for instance, display a proposed
completion in a micro window and insert the proposed
text when the user presses the tab key.
The sentence completion problem poses new chal-
lenges for data mining and information retrieval, includ-
ing the problem of finding sentences whose initial frag-
ment is similar to a given fragment in a very large text
corpus. To this end, Grabski and Scheffer (2004) provide
a retrieval algorithm that uses a special inverted indexing
structure to find the sentence whose initial fragment is
most similar to a given fragment, where similarity is

Table 1. Algorithms for learning from message pairs and answering new questions

Learning from message pairs.

Input: Message pairs, variance threshold 2, pruning parameter .
1. Recursively cluster answers of message pairs with bisecting partitioning cluster algorithm, end
recursion when cluster variance lies below 2.
2. Prune all clusters with less than elements. Combine all pruned clusters into one
miscellaneous cluster. Let n be the number of resulting clusters.
3. For all n clusters
a. Construct an answer template by choosing the answer that is most similar to the
centroid of this cluster in vector space representation and remove salutation line.
b. Let the inbound mails that have been answered by a mail in the current cluster be the
positive training examples for this answer class.
2. Train SVM classifier that classifies an inbound message into one of the n answer classes or the
miscellaneous class from these training examples. Return this classifier.

Answering new questions.

Input: New question message, message answering hypothesis, confidence threshold .
1. Classify new message into one of the n answer classes and remember SVM decision function
value.
2. If confidence exceeds the confidence threshold, propose the answer template that corresponds
to the classification result. Perform instantiation operations that typically include formulating
a salutation line.

770

TEAM LinG
Mining E-Mail Data
defined in terms of the greatest cosine similarity of the
TFIDF vectors. In addition, they study an approach that
compresses the data further by identifying clusters of the
most frequently used similar sets of sentences. In order to
evaluate the accuracy of sentence completion algorithms,
Grabski and Scheffer (2004) measure how frequently the
algorithm, when given a sentence fragment drawn from a
corpus, provides a prediction with confidence above ,
and how frequently this prediction is semantically equiva-
lent to the actual sentence in the corpus. They find that
for the sentence mining problem higher precision and
recall values can be obtained than for the problem of
mining question answer pairs; depending on the thresh-
old and the fragment length, precision values of between
80% and 100% and recall values of about 40% can be
observed.
FUTURE TRENDS
Spam filtering and e-mail filing based on message text can be
reduced to the well studied problem of text classification.
The challenges that e-mail classification faces today con-
cern technical aspects, the extraction of spam-specific
features from e-mails, and an arms race between spam filters
and spam senders adapting to known filters. By comparison,
research in the area of automatic e-mail answering and
sentence completion is in an earlier stage; we see a substan-
tial potential for algorithmic improvements to the existing
methods. The technical integration of these approaches into
existing e-mail clients or call-center automation software
provides an additional challenge. Some of these technical
challenges have to be addressed before mining algorithms
that aim at supporting communication can be evaluated
under realistic conditions.
Construction of social network graphs from e-mail
logs is much easier than by surveys and there is a huge
interest in mining social networks see, for instance, the
DARPA program on Evidence Extraction and Link Dis-
covery (EELD). While social networks have been studied
intensely in the social sciences and in physics, we see a
considerable potential for new and better mining algo-
rithms for social networks that computer scientists can
contribute.
CONCLUSION
Some methods that can form the basis for effective spam
filtering have reached maturity (text classification), addi-
tional foundations are being worked on (social network
analysis). Today, technical challenges dominate the de-
velopment of spam filters. The development of methods
that support and automate communication processes is
a research topic and first solutions to some of the prob-
lems involved have been studied. Mining social networks M
from e-mail logs is a new challenge; research on this topic
in computer science is in an early stage.
ACKNOWLEDGMENT
The authors are supported by the German Science Foun-
dation DFG under grant SCHE540/10-1. We would like to
thank the anonymous reviewers.
REFERENCES
Bickel, S., & Scheffer, T. (2004). Learning from message
pairs for automatic email answering. Proceedings of the
European Conference on Machine Learning.
Boykin, P., & Roychowdhury, V. (2004). Personal e-mail
networks: An effective anti-spam tool. Preprint, arXiv id
0402143.
Cohen, W. (1996). Learning rules that classify e-mail.
Proceedings of the IEEE Spring Symposium on Machine
learning for Information Access, Palo Alto, California,
USA.
Drucker, H., Wu, D., & Vapnik, V. (1999). Support vector
machines for spam categorization. IEEE Transactions on
Neural Networks, 10(5), 1048-1055.
Ebel, H., Mielsch, L., & Bornholdt, S. (2002). Scale-free
topology of e-mail networks. Physical Review, E 66.
Fruchterman, T. M., & Rheingold, E. M. (1991). Force-
directed placement. Software Experience and Practice,
21(11).
Garton, L., Haythornthwaite, C., & Wellman, B. (1997).
Studying online social networks. Journal of Computer-
Mediated Communication, 3(1).
Grabski, K., & Scheffer, T. (2004). Sentence completion.
Proceedings of the SIGIR International Conference on
Information Retrieval, Sheffield, UK.
Graham, P. (2003). Better Bayesian filtering. Proceedings
of the First Annual Spam Conference, MIT. Retrieved
from http://www.paulgraham.com/better.html
Green, C., & Edwards, P. (1996). Using machine learning
to enhance software tools for internet information man-
agement. Proceedings of the AAAI Workshop on Internet
Information Management.
Hinds, P., & Kiesler, S. (1995). Communication across
boundaries: Work, structure, and use of communication
771
TEAM LinG

technologies in a large organization. Organization Sci-
ence, 6(4), 373-393.
Joachims, T. (1997). A probabilistic analysis of the Rocchio
algorithm with TFIDF for text categorization. Proceedings
of the International Conference on Machine Learning.
Joachims, T. (1998). Text categorization with support
vector machines: Learning with many relevant features.
Proceedings of the European Conference on Machine
Learning.
McDowell, L., Etzioni, O., Halevy, A., & Levy, H. (2004).
Semantic e-mail. Proceedings of the WWW Conference.
Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. (2000).
Text classification from labeled and unlabeled documents
using EM. Machine Learning, 39(2/3).
Pantel, P., & Lin, D. (1998) Spamcop: a spam classification
and organization program. Proceedings of the AAAI Work-
shop on Learning for Text Categorization.
Provost, J. (1999). Nave Bayes vs. rule-learning in clas-
sification of e-mail. Technical Report AI-TR-99-284, Uni-
versity of Texas at Austin.
Rennie, J. (2000). iFILE: An application of machine learn-
ing to e-mail filtering. Proceedings of the SIGKDD Text
Mining Workshop.
Sahami, M., Dumais, S., Heckerman, D., & Horvitz, E.
(1998). A Bayesian approach to filtering junk e-mail.
Proceedings of AAAI Workshop on Learning for Text
Categorization.
Scheffer, T. (2004). E-mail answering assistance by
semi-supervised text classification. Intelligent Data
Analysis, 8(5).
Tyler, J. R., Wilkinson, D. M., & Huberman, B. A.
(2003). E-mail as spectroscopy: Automated discovery
of community structure within organizations. Proceed-
ings of the International Conference on Communities
and Technologies (pp. 81-95). Kluwer Academic Publish-
ers.
772
Mining E-Mail Data
KEY TERMS
Community: A group of people having mutual rela-
tionships among themselves or having common interests.
Clusters in social network graphs are interpreted as com-
munities.
Mining E-Mails: The application of analytical meth-
ods and tools to e-mail data for a) support of communica-
tion by filing e-mails into folders, filtering spam, answer-
ing e-mails automatically, or proposing completions to
sentence fragments, b) discovery of hidden properties of
communication networks by e-mail graph analysis.
Mining Question-Answer Pairs: Analytical method
for automatically answering question e-mails using knowl-
edge that is discovered in question-answer pairs of past
e-mail communication.
Mining Sentences: Analytical method for interac-
tively completing incomplete sentences using knowledge
that is discovered in a document collection.
Semantic E-Mail: E-mail framework in which the
semantics of e-mails is understandable by both, human
and machine. A standardized definition of semantic e-
mail processes is required.
Spam E-Mail: Unsolicited and unwanted bulk e-
mail. Identifying spam e-mail is a text classification
task.
Text Classification: The task of assigning docu-
ments expressed in natural language to one or more
categories (classes) of a predefined set.
TFIDF: Weighting scheme for document and query
representation in the vector space model. Each dimension
represents a term, its value is the product the frequency
of the term in a document (TF) and the inverse document
frequency (IDF) of the term. The inverse document
frequency of a term is the logarithmic proportion of
documents in which the term occurs. The TFIDF scheme
assigns a high weight to terms which occur frequently in
the focused document, but are infrequent in average
documents.
TEAM LinG

Mining for Image Classification Based on
Feature Elements
Yu-Jin Zhang
Tsinghua University, Beijing, China
INTRODUCTION
Motivation: Image Classification in Web
Search
The growth of the Internet and storage capability not
only increasingly makes images a widespread informa-
tion format on the World Wide Web (WWW), but it also
dramatically expands the number of images on WWW
and makes the search of required images more complex
and time-consuming. To efficiently search images on
the WWW, effective image search engines need to be
developed.
The classification of images plays an important role
both for Web image searching and retrieving, as it is
time-consuming for users to browse through the huge
amount of data on the Web. Classification has been used
to provide access of large image collections in a more
efficient manner, because the classification can reduce
search space by filtering out images in an unrelated
category (Hirata, 2000).
The heterogeneous nature of Web images makes
their classification a challenging task. A functional
classification scheme should take the contents of im-
ages into consideration. The association rule mining,
first proposed by Agrawal (1993), is an appropriate tool
for pattern detection in knowledge discovery and data
mining. Its objective is to extract useful information
from very large databases (Renato, 2002). By using
rules extracted from images, the content of images can
be suitably analyzed, and the information required for
image classification can be obtained.
Highlights of the Article
A novel method for image classification based on fea-
ture element through association rule mining is pre-
sented. The feature elements can capture well the visual
meanings of images according to the subjective percep-
tion of human beings. In addition, feature elements are
discrete entities and are suitable for working with rule-
based classification models. Different from traditional
image classification methods, the proposed classifica-
tion approach, based on a feature element, does not
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
773
M
compute the distance between two vectors in the feature
space. This approach just tries to find associations
between the feature elements and class attributes of the
image. Techniques for mining the association rules are
adapted, and the mined rules are applied to image clas-
sifications. Experiments with real images show that the
new approach not only reduces the classification errors
but also diminishes the time complexity.
The remaining parts of this article are structured as
follows:
Background: (1) Feature Elements vs. Feature
Vectors; (2) Association Rules and Rule Mining;
(3) Classification Based on Association
Main Thrust: (1) Extracting Various Types of
Feature Elements; (2) Feature Element Based Im-
age Classification; (3) Database Used in Test; (4)
Classification and Comparison Results
Direction of Future Research.
Conclusion.
BACKGROUND
Feature Elements vs. Feature Vectors
Traditionally, feature vectors are used for object iden-
tification and classification as well as for content-based
image retrieval (CBIR). In object identification and
classification, different features representing the char-
acteristics of objects are extracted first. These features
mark out an object to a point in the feature space. By
detecting this point in the space, the object can be
identified or classified. In CBIR, the procedure is simi-
lar. Features such as color, texture, and shape are ex-
tracted from images and grouped into feature vectors
(Zhang, 2003). The similarity among images is mea-
sured by distances between corresponding vectors.
However, these feature vectors often are different
from the representation and description adapted by hu-
man beings. For example, when people look at a colorful
image, they hardly figure out its color histogram but
rather are concerned about what particular colors are
contained in certain components of the image. In fact,
TEAM LinG
 Mining for Image Classification Based on Feature Elements
these color components play a great role in perception and
represent useful visual meanings of images. The pixels
belonging to these visual components can be taken to form
perceptual primitive units, by which human beings could
identify the content of images (Xu, 2001).
The feature elements are defined on the basis of these
primitive units. They are discrete quantities, relatively
independent of each other, and have obvious intuitive
visual senses. In addition, they can be considered as sets
of items. Based on feature elements, image classifica-
tion becomes a process of counting the existence of
representative components in images. For this purpose,
it is required to find some association rules between the
feature elements and the class attributes of image.
Association Rules and Rule Mining
The association rule can be represented by an expres-
sion X Y, where X and Y can be any discrete entity. As
we discuss image database, X and Y can be some feature
elements extracted from images. The meaning of X Y
is: Given an image database D, for each image I D, X
Y expresses that whenever an image I contains X then I
probably will also contain Y. The support of association
rule is defined as the probability p(X I, Y I), and the
confidence of association rule is defined as the condi-
tional probability p(X I | Y I). A rule with support
bigger than a specified minimum support and with con-
fidence bigger than a specified minimum confidence is
considered as a significant association rule.
Since the introduction of the association rule mining
by Agrawal (1993), many researches have been con-
ducted to enhance its performance. Most works can be
grouped into the following categories:
1. Works for mining of different rules, such as multi-
dimensional rules (Yang, 2001).
2. Works for taking advantage of particular techniques,
such as, tree projection (Guralnik, 2004), mul-
tiple minimum supports (Tseng, 2001), constraint-
based clustering (Tung, 2001), and association
(Cohen, 2001).
3. Works for developing fast algorithms, such as algo-
rithm based on anti-skew partitioning (Lin, 1998).
4. Works for discovering a temporal database, such
as discovering temporal association rules
(Guimaraes, 2000; Li, 2003).
Currently, the association rule mining (Lee, 2003;
Harms, 2004) is one of the most popular pattern discov-
ery methods in knowledge discovery and data mining. In
contrast to the classification rule mining (Pal, 2003),
the purpose of association rule mining is to find all
significant rules in the database that satisfy some mini-
774
mum support and minimum confidence constraints (Hipp,
2000). It is known that rule-based classification models
often have difficulty dealing with continuous variables.
However, as a feature element is just a discrete entity,
association rules can easily be used for treating images
represented and described by feature elements. In fact,
a decision about whether an image I contains feature
element X and/or feature element Y can be properly
defined and detected.
Classification Based on Association
Classification based on associations (CBA) is an algo-
rithm for integrating classification and association rule
mining (Liu, 1998). Assume that the data set is a normal
relational table that consists of N cases described by
distinct attributes and classified into several known
classes. All the attributes are treated uniformly. For a
categorical attribute, all the possible values are mapped
to a set of consecutive positive integers. With these
mappings, a data case can be treated as a set of (attribute,
integer value) pairs plus a class label. Each (attribute,
integer value) is called an item. Let D be the data set, I the
set of all items in D, and Y the class labels. A class
association rule (CAR) is an implication of the form X
y, where X I, and y Y. A data case d D means d
contains a subset of items; that is, X d and X I. A rule
X y holds in D with confidence C if C percentages of
cases in D that contain X are labeled with class y. The rule
X y has support S in D if the S percentages of cases in
D are contained in X and are labeled with class y.
The objective of CBA is to generate the complete set
of CARs that satisfy the specified minimum supports
and minimum confidence constraints, and to build a
classifier from CARs. It is easy to see that if the right-
hand-side of the association rules is restricted to the
(classification) class attributes, then such rules can be
regarded as classification rules to build classifiers.
MAIN THRUST
Extracting Various Types of Feature
Elements
Various types of feature elements that put emphasis on
different properties will be employed in different applica-
tions. The extractions of feature elements can be carried
out first by locating the perceptual elements and then by
determining their main properties and giving them suit-
able descriptions. Three typical examples are described in
the following.
TEAM LinG
Mining for Image Classification Based on Feature Elements

One process for obtaining feature elements primarily
based on color properties can be described by the follow-
ing steps (Xu, 2001):
1. Images are divided into several clusters with a per-
ceptual grouping based on hue histogram.
2. For each cluster, the central hue value is taken as its
color cardinality named as Androutsos-cardinality
(AC). In addition, color-coherence-vector (CCV) and
color-auto-correlogram (CAC) also are calculated.
3. Additional attributes such as the center coordi-
nates and area of each cluster are recoded to repre-
sent the position and size information of clusters.
One type of feature element highlighting the form
property of clusters is obtained with the help of Zernike
moments (Xu, 2003). They are invariant to similarity
transformations, such as translation, rotation, and scal-
ing of the planar shape (Wee, 2003). Based on Zernike
moments of clusters, different descriptors for express-
ing circularity, directionality, eccentricity, roundness,
symmetry, and so forth, can be directly obtained, which
provides useful semantic meanings of clusters with re-
spect to human perception.
Wavelet feature element is based on wavelet modulus
maxima and invariant moments (Zhang, 2003). Wavelet
modulus maxima can indicate the location of edges in
images. A set of seven invariant moments (Gonzalez,
2002) is used to represent the multi-scale edges in
wavelet-transformed images. Three steps are taken first:
1. Images are decomposed, using dyadic wavelet, into
a multi-scale modulus image.
2. Pixels in the wavelet domain whose moduli are
locally maxima are used to form multi-scale edges.
3. The seven invariant moments at each scale are
computed and combined to form the feature vector
of images.
Then, a process of discretization is followed (Li,
2002). Suppose the wavelet decomposition is performed M
in six levels; for each level, seven moments are computed.
This gives a 42-D vector. It can be split into six groups,
each of them being a 7-D vector that represents seven
moments on one level. On the other side, the whole vector
can be split into seven groups; each of them is a 6-D
vector that represents one moment on all six levels. This
process can be described with the help of Figure 1.
In all these examples, the feature elements have prop-
erty represented by numeric values. As not all of the
feature elements have the same status in the visual
sense, an evaluation of feature elements is required to
select suitable feature elements according to the sub-
jective perception of human beings (Xu, 2002).
Feature Element Based Image
Classification
Feature Element Based Image Classification (FEBIC)
uses CBA to find association rules between feature
elements and class attributes of the images, while the
class attributes of unlabeled images could be predicted
with such rules. In case an unlabeled image satisfies
several rules, which might make this image be classi-
fied into different classes, the support values and con-
fidence values can be used to make the final decision.
In accordance with the assumption in CBA, each
image is considered as a data case, which is described
by a number of attributes. The components of the
feature element are taken as attributes. The labeled
image set can be considered as a normal relational table
that is used to mine association rules for classification.
In the same way, feature elements from unlabeled im-
ages are extracted and form another relational table
without class attributes, on which the classification
rules to predict the class attributes of each unlabeled
image will be applied.
The whole procedure can be summarized as follows:

Figure 1. Splitting and groups feature vectors to 1. Extract feature elements from images.
construct feature elements 2. Form relational table for mining association rules.
3. Use mined rules to predict the class attributes of
m11 m12 m13 m14 m15 m16 m17 unlabeled images.
4. Classify images using the association of feature
m21 m22 m23 m24 m25 m26 m27 elements.
m31 m32 m33 m34 m35 m36 m37
Database Used in Test
m41 m42 m43 m44 m45 m46 m47
The image database for testing consists of 2,558 real-
m51 m52 m53 m54 m55 m56 m57 color images that can be grouped into five different
classes: (1) 485 images with (big) flowers; (2) 565
m61 m62 m63 m64 m65 m66 m67 images with person pictures; (3) 505 images with au-

775

TEAM LinG
 Mining for Image Classification Based on Feature Elements

Figure 2. Typical image examples from different classes

(a) Flower image (b) Person picture (c) Auto image

(d) Scenery (e) Flower cluster

tos; (4) 500 images with different sceneries (e.g., sunset,
sunrise, beach, mountain, forest, etc.); and (5) 503 images
with flower clusters. Among these classes, the first three
have prominent objects, while the other two normally
have no dominant items. Two typical examples from each
class are shown in Figure 2.
Among these images, one-third have been used in the
test set and the rest in the training set. The images in the
training set are labeled manually and then used in the mining
of association rules, while the images in the testing set will
be labeled automatically by these mined rules.
Except the classification error, the time complexity is
another important factor to be counted in Web applica-
tion, as the number of images on the WWW is huge. The
computation times for two methods are compared during
the test experiments. The time needed for FEBIC is only
about 1/100 of the time needed for NFL. Since NFL requires
many arithmetic operations to compute distance func-
tions, while FEBIC needs only a few operations for judg-
ing the existence of feature elements, such a big difference
in computation is well expected.

Classification and Comparison Results FUTURE TRENDS

Classification experiments using two methods with the The detection and description of feature elements play
previously mentioned database are carried out. The pro- an important role in providing suitable information and
posed method FEBIC is compared to another state-of- a basis for association rule mining. How to adaptively
the-art methodnearest feature line (NFL) (Li, 2000). design feature elements that can capture the users
NFL is a classification method based on feature vectors. intention based on perception and interpretation needs
In comparison, the color feature (i.e., AC, CCV, CAC) further research.
and the wavelet feature based on wavelet modulus maxima, The proposed techniques also can be extended to the
and invariant moments are used. content-based retrieval of images over the Internet. As
Two tests are performed. For each test, both meth- feature elements are discrete entities, the similarity
ods use the same training set and testing set. The results between images described by feature elements can be
of these experiments are summarized in Table 1, where computed according to the number of common elements.
the classification error rates for each class and for the
average over the five classes are listed.
The results in Table 1 show that the classification CONCLUSION
error rate of NFL is about 34.5%, while the classifica-
tion error rate of FEBIC is about 25%. The difference is A new approach for image classification that uses fea-
evident. ture elements and employs association rule mining is
proposed. It provides lower classification error and
higher computation efficiency. These advantages make
Table 1. Comparison of classification errors
it quite suitable to be included into a Web search engine
Error rate Test set 1 Test set 2 for images over the Internet.
FEBIC NFL FEBIC NFL
Flower 32.1% 48.8% 36.4% 46.9%
Person 22.9% 25.6% 20.7% 26.1%
Auto 21.3% 23.1% 18.3% 23.1%
Scenery 30.7% 38.0% 32.5% 34.3%
Flower cluster 26.8% 45.8% 20.2% 37.0%
Average 26.6% 35.8% 25.4% 33.2%

776

TEAM LinG
Mining for Image Classification Based on Feature Elements
ACKNOWLEDGMENTS
This work has been supported by the Grants NNSF-
60172025 and TH-EE9906.
REFERENCES
Agrawal, R., Imielinski, T. & Swami, A. (1993). Mining
association rules between sets of items in large data-
bases. Proceeding of the ACM SIGMOD.
Cohen, E. et al. (2001). Finding interesting associations
without support pruning. IEEE Trans. Knowledge and
Data Engineering, 13(1), 64-78.
Gonzalez, R.C., & Woods, R.E. (2002). Digital image
processing. Prentice Hall.
Guimaraes, G. (2000). Temporal knowledge discovery
for multivariate time series with enhanced self-organiz-
ing maps. Proceedings of the International Joint Con-
ference on Neural Networks.
Guralnik, V., & Karypis, G. (2004). Parallel tree-pro-
jection-based sequence mining algorithms. Parallel
Computing, 30(4), 443-472.
Harms, S K., & Deogun, J.S. (2004). Sequential associa-
tion rule mining with time lags. Journal of Intelligent
Information Systems, 22(1), 7-22.
Hipp, J., Guntzer, U., & Nakhaeizadeh, G. (2000). Algo-
rithms for association rule miningA general survey
and comparison. ACM SIGKDD, 2(1), 58-64.
Hirata, K. et al. (2000). Integration of image matching and
classification for multimedia navigation. Multimedia Tools
and Applications, 11, 295309.
Lee, C.H., Chen, M.S., & Lin, C.R. (2003). Progressive
partition miner: An efficient algorithm for mining gen-
eral temporal association rules. IEEE Trans. Knowl-
edge and Data Engineering, 15(4), 1004-1017.
Li, Q., Zhang, Y.J., & Dai, S.Y. (2002). Image search
engine with selective filtering and feature element based
classification. Proceedings of the SPIE of Internet
Imaging III.
Li, S.Z., Chan, K.L., & Wang, C.L. (2000). Performance
evaluation of the nearest feature line method in image
classification and retrieval. IEEE Trans. Pattern Analy-
sis and Machine Intelligence, 22(11), 1335-1339.
Li, Y.J. et al. (2003). Discovering calendar-based temporal
association rules. Data and Knowledge Engineering, M
44(2), 193-218.
Lin, J.L., & Dunham, M.H. (1998). Mining association
rules: Antiskew algorithms. Proceeding of the Interna-
tional Conference on Data Engineering.
Liu, B., Hsu, W., & Ma, Y.M. (1998). Integrating classifi-
cation and association rule mining. Proceedings of the
International Conference on Knowledge Discovery and
Data Mining.
Pal, S.K. (2003). Soft computing pattern recognition, case
generation and data mining. Proceedings of the Interna-
tional Conference on Active Media Technology.
Renato, C. (2002). A theoretical framework for data mining:
The informational paradigm. Computational Statistics
and Data Analysis, 38(4), 501-515.
Tseng, M.C., Lin, W., & Chien, B.C. (2001). Maintenance
of generalized association rules with multiple minimum
supports. Proceedings of the Annual Conference of the
North American Fuzzy Information Processing Society.
Tung, A.K.H. et al. (2001). Constraint-based clustering in
large databases. Proceedings of International Confer-
ence on Database Theory.
Wee, CY. (2003). New computational methods for full and
subset Zernike moments. Information Sciences, 159(3-4),
203-220.
Xu, Y., & Zhang, Y.J. (2001). Image retrieval framework
driven by association feedback with feature element
evaluation built in. Proceedings of the SPIE Storage
and Retrieval for Media Databases.
Xu, Y., & Zhang, Y.J. (2002). Feature element theory
for image recognition and retrieval. Proceedings of the
SPIE Storage and Retrieval for Media Databases.
Xu, Y., & Zhang, Y.J. (2003). Semantic retrieval based
on feature element constructional model and bias com-
petition mechanism. Proceedings of the SPIE Storage
and Retrieval for Media Databases.
Yang, C., Fayyad, U., & Bradley, P.S. (2001). Efficient
discovery of error-tolerant frequent itemsets in high
dimensions. Proceedings of the International Confer-
ence on Knowledge Discovery and Data Mining.
Zhang, Y.J. (2003). Content-based visual information
retrieval. Science Publisher.
777
TEAM LinG
 Mining for Image Classification Based on Feature Elements
KEY TERMS
Classification Error: Error produced by incorrect clas-
sifications, which consists of two types: correct negative
(wrongly classify an item belonging to one class into
another class) and false positive (wrongly classify an item
from other classes into the current class).
Classification Rule Mining: A technique/procedure
aiming to discover a small set of rules in the database to
form an accurate classifier for classification.
Content-Based Image Retrieval (CBIR): A process
framework for efficiently retrieving images from a collec-
tion by similarity. The retrieval relies on extracting the
appropriate characteristic quantities describing the de-
sired contents of images. In addition, suitable querying,
matching, indexing, and searching techniques are re-
quired.
Multi-Resolution Analysis: A process to treat a func-
tion (i.e., an image) at various levels of resolutions and/
or approximations. In such a way, a complicated function
could be divided into several simpler ones that can be
studied separately.
Pattern Detection: Concerned with locating patterns
in the database to maximize/minimize a response variable
778
or minimize some classification error (i.e., supervised
pattern detection), or with not only locating occurrences
of the patterns in the database but also deciding whether
such an occurrence is a pattern (i.e., unsupervised pattern
detection).
Pattern Recognition: Concerned with the classifica-
tion of individual patterns into pre-specified classes (i.e.,
supervised pattern recognition), or with the identification
and characterization of pattern classes (i.e., unsupervised
pattern recognition).
Similarity Transformation: A group of transforma-
tions that will preserve the angles between any two
curves at their intersecting points. It is also called equi-
form transformation, because it preserves the form of
curves. A planar similarity transformation has four de-
grees of freedom and can be computed from a two-point
correspondence.
Web Image Search Engine: A kind of search engine
that starts from several initially given URLs and extends
from complex hyperlinks to collect images on the WWW.
Web search engine is also known as Web crawler.
Web Mining: Concerned with the mechanism for
discovering the correlations among the references to
various files that are available on the server by a given
client visit to the server.
TEAM LinG

Mining for Profitable Patterns in the Stock
Market
Yihua Philip Sheng
Southern Illinois University, USA
Wen-Chi Hou
Southern Illinois University, USA
Zhong Chen
Shanghai JiaoTong University, PR China
INTRODUCTION
The stock market, like other economic phenomena, is a
very complex system. Many factors, such as company
news, interest rates, macro economic data, and inves-
tors hopes and fears, all affect its behavior (Pring,
1991; Sharpe, Alexander, & Bailey, 1999). Investors
have longed for tools and algorithms to analyze and
predict stock market movement. In this study, we com-
bine a financial theory, the market efficiency theory,
and a data mining technique to explore profitable trading
patterns in the stock market. To observe the price oscil-
lation of several consecutive trading days, we examine
the K-lines, each of which represents a stocks one-day
movement. We will use a data mining technique with a
heuristic rating algorithm to mine for reliable patterns
indicating price rise or fall in the near future.
BACKGROUND
Methods of Stock Technical Analysis
Conventional stock market technical analysis is often
done through visually identifying patterns or indicators
on the stock price and volume charts. Indicators like
moving averages and support and resistance level are
easy to implement algorithmatically. Patterns like head-
and-shoulder, inverse head-and-shoulder, broadening
tops and bottoms, and etcetera, are easy for human to
visually identify but difficult for computers to recog-
nize. For such patterns, methods like smoothing estima-
tors and kernel regression can be applied to increase
their machine-readability (Dawson & Steely, 2003; Lo,
Mamaysky, & Wang, 2000).
The advances in data mining technology have pushed
the technical analysis of stock market from simple
indicators, visually recognizable patterns, and linear
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
779
M
statistical models to more complicated nonlinear mod-
els. A great deal of research has focused on the applica-
tions of artificial intelligence (AI) algorithms, such as
artificial neural networks (ANNs) and genetic algo-
rithm (e.g., Allen & Karjalainen, 1999; Chenoweth,
Obradovic, & Stephenlee, 1996; Thawornwong, Enke, &
Dagli, 2003). ANNs equipped with effective learning
algorithms can use different kinds of inputs, handle
noisy data, and identify highly nonlinear models. Ge-
netic algorithms constitute a class of search, adaptation,
and optimization techniques to emulate the principles
of natural evolution. More recent studies tend to em-
brace multiple AI techniques in one approach. Tsaih,
Hsu, & Lai (1998) integrated the rule-based systems
technique and the neural networks technique to predict
the direction of daily price changes in S&P 500 stock
index futures. Armano, Murru, & Roli (2002) employed
ANNs with a genetic algorithm to predict the Italian stock
market. Fuzzy logic, a relatively newer AI algorithm,
has also been used in stock market prediction literature
(e.g., Dourra & Siy, 2001). The increasing popularity of
fuzzy logic is due to its simplicity in constructing the
models and less computation load. Fuzzy algorithms
provide a fairly straightforward translation of the quali-
tative/linguistic statements of rules.
Market Efficiency Theory
According to the market efficiency theory, a stocks
price is a full reflection of market information about
that stock (Fama, 1991; Malkiel, 1996). Therefore, if
there is information out on the market about a stock, the
stocks price will adjust accordingly. Interestingly, evi-
dence shows that price adjustment in response to news
usually does not settle down in one day; it actually takes
some time for the whole market to digest the news. If the
stocks price really adjusted to relevant events in a
timely manner, the stock price chart would have looked
more like what Figure 1 shows.
TEAM LinG

Figure 1. Ideal stock price movement curve under the
market efficiency theory
The flat periods indicate that there were no events
occurring during the periods while the sharp edges
indicate sudden stock price movements in response to
event announcements. However, in reality, most stocks
daily price resembles the curve shown in Figure 2. As
the figure shows, there is no obvious flat period for a
stock and the stock price seemed to keep on changing. In
some cases, the stock price continuously moves down
or up for a relatively long period, for example, the
period of May 17, 2002 to July 2, 2002, and the period
of October 16, 2002 to November 6, 2002. This could
be either there were negative (or positive) events for the
company every day for a long period of time or the stock
price adjustment to events actually spans a period of
time, rather than instantly. The latter means that stock
price adjustment to the event announcements is not
efficient and the semi-form of the market efficiency theory
does not hold. Furthermore, we think the first few days
price adjustments of the stock are crucial, and the price
movements in these early days might contain enough
information to predict whether the rest of price adjustment
in the near future is upwards or downwards.
Knowledge Representation
Knowledge representation holds the key to the success
of data mining. A good knowledge representation should
be able to include all possible phenomena of a problem
domain without complicating it (Liu, 1998). Here, we
use K-lines, a widely used representation method of the
daily stock price in Asian stock markets, to describe the
daily price change of a stock. Figure 3 is examples of K-
Lines.
Figure 2. The daily stock price curve of Intel Corporation (NasdaqNM Symbol: INTC)
780
Mining for Profitable Patterns in the Stock Market
Figure 3(a) is a price-up K-Line, denoted by an empty
rectangle, indicating the closing price is higher than the
opening price. Figure 3(b) is a price-down K-line, de-
noted by a solid rectangle, indicating the closing price is
lower than the opening price. Figure 3(c) and 3(d) are 3-
day K-Lines. Figure 3(c) shows that the price was up for
two consecutive days and the second days opening
price continued on the first days closing price. This
indicates that the news was very positive. The price came
down a little bit on the third day, which might be due to the
correction to the over-valuation of the good news in the
prior two days. Actually, the long price shadow above the
closing price of the second day already shows some
degree of price correction. Figure 3(d) is the opposite of
Figure 3(c). When an event about a stock happens, such
as rumors on merger/acquisition, or change of dividend
policies, the price adjustments might last for several days
till the price finally settles down. As a result, the stocks
price might keep rising or falling or stay the same during
the price adjustment period.
A stock has a K-Line for every trading day, but not
every K-Line is of our interest. Our goal is to identify a
stocks K-Line patterns that reflect investors reactions
to market events such as the releases of good or bad
corporate news, major stock analysts upgrade on the
stock, and etcetera. Such market events usually can
cause the stocks price to oscillate for a period of time.
Certainly, a stocks price sometimes might change with
large magnitude just for one day or two due to transient
market rumors. These types of price oscillations are
regarded as market noises and therefore are ignored.
Whether a stocks daily price oscillates is deter-
mined by examining if the price change on that day is
greater than the average price change of the year. If a
stocks price oscillates for at least three consecutive
days, we regard it as a signal of the occurrence of a
market event. The markets response to the event is
recorded in a 3-day K-Line pattern. Then, we examine
whether this pattern is followed by an up or down trend
of the stocks price a few days later.
TEAM LinG
Mining for Profitable Patterns in the Stock Market
Figure 3. K-Line examples
The relative positions of the K-Lines such as one days
opening/closing prices relative to the prior days closing/
opening prices, the length of the price body, etc. reveal
market reactions to the events. The following bit-represen-
tation method, called Relative Price Movement or RPM for
simplicity, is used to describe the positional relationship
of the K-lines in three days.
Day 1
bit 0: 1 if the days price is up, 0 otherwise
bit 1: 1 if the up shadow is longer than the price
body, 0 otherwise.
bit 2: 1 if the down shadow is longer than the price
body, 0 otherwise
Day 2
bits 0 2: The same as Day 1s representation
bits 3 5:
001, if the price body covers the day 1s price body
010, if the price body is covered by the day 1s
price body
011, if the whole price body is higher than the day
1s price body
100, if the whole price body is lower than the day
1s price body
101, if the price body is partially higher than the
day 1s price body
110, if the price body is partially lower than the day
1s price body
DAY 3
bits 0 2: The same as Day 1s representation
bits 3 7:
M
0000100111, reserved
01000, if the price body covers the day 1 and day
2s price bodies
01001, if the price body covers the day 1s price
body only
01010, if the price body covers the day 2s price
body only
01011, if the price body is covered by the day 1
and day 2s price bodies
01100, if the price body is covered by the day 2s
price body only
01101, if the price body is covered by the day 1s
price only
01110, if the whole price body is higher than the
day 1 and day 2s price bodies
01111, if the whole price body is higher than the
day 1s price body only
10000, if the whole price body is higher than the
day 2s price body only
10001, if the whole price body is lower than the
day 1 and day 2s price bodies
10010, if the whole price body is lower than the
day 1s price body only
10011, if the whole price body is lower than the
day 2s price body only
10100, if the price body is partially higher than
the day 1 and day 2s price bodies
10101, if the price body is partially lower than
the day 1 and day 2s price bodies
10110, if the price body is partially higher than
the day 2s price body only
10111, if the price body is partially higher than
the day 1s price body only
11000, if the price body is partially lower than
the day 2s price body only
11001, if the price body is partially lower than
the day 1s price body only
781
TEAM LinG
 Mining for Profitable Patterns in the Stock Market

Mining for Rules patterns to reduce the ambiguity of the patterns. Using
the price-up pattern as an example, for a pattern to be
The rules we mine for are similar to those by Liu (1998), labeled as a price-up pattern, we think the times it
Siberschatz & Tuzhilin (1996), and Zaki, Parthasatathy, appeared in Pup should be at least twice as many as the
Ogihara, & Li (1997). They have the following format: times it appeared in the Pdown. Within all the patterns
labeled as price-up patterns, they were sorted based on
Rule type (1): a 3-day K-Line pattern the stocks the ratio of the squared root of its total occurrences plus
price rises 10% in 10 days its occurrence as a price-up pattern over the occurrence
as a price-down pattern.
Rule type (2): a 3-day K-Line pattern the stocks
For price-up pattern:
price falls 10 % in 10 days
Pup PO Pup
The search algorithm for finding 3-day K-Line pat- + * Pup , if Pdown > 2
Pdown Pdown

terns that lead to stock price rise or fall is as follows:
Preference =
up ,
1. For every 3-day K-Line pattern in the database P + PO * Pup , if Pup 2
2. Encode it by using the RPM method to get every
Pdown Pdown Pdown
days bit representation, c1, c2, c3;
3. Increase pattern_occurrence[c1][c2][c3] by 1;
4. base_price = the 3rd days closing price; For price-down pattern:
5. if the stocks price rises 10% or more, as com-
pared to the base_price, in 10 days after the Pdown PO Pdown
+ * Pdown , if >2
occurrence of this pattern Pup Pup Pup

Preference =
Increase Pup[c1][c2][c3] by 1; down
P PO Pdown
+ * Pdown , if 2
Pup Pup Pup
6. if the stocks price falls 10% or more, as com-
pared to the base_price, in 10 days after the
occurrence of this pattern The final winning patterns with positive Preference
score are listed in Table 2.
Increase Pdown[c1][c2][c3] by 1;
Performance Evaluation
We used the daily trading data from January 1, 1994,
through December 31, 1998, of the 82 stocks, as shown in
To evaluate the performance of the found winning pat-
Table 1, as the base data set to mine for the price up and
terns listed in Table 2, we applied them to the prices of
down patterns. After applying the above search algorithm
the same 82 stocks for the period from January 1, 1999,
on the base data set, the Pup and Pdown arrays contained the
counts of all the patterns that led price to rise or fall by 10%
in 10 days. In total, the up-patterns occurred 1,377 times,
Table 2. Final winning patterns sorted by preference
among which there were 870 different types of up-pat-
terns; and the down-patterns occurred 1,001 times, among Pattern Code PO Pup Pdown Preference
which there were 698 different types of down-patterns. Up[00][20][91] 46 15 4 81.68
A heuristic, stated below, was applied to all found Up[01][28][68] 17 7 1 77.86
Up[07][08][88] 11 7 1 72.22
Up[00][24][88] 10 7 1 71.14
Table 1. 82 selected stocks Up[00][30][8E] 9 7 1 70.00
ADBE BA CDN F KO MWY S WAG Up[01][19][50] 28 12 3 69.17
ADSK BAANF CEA FON LGTO NETG SAPE WCOM Up[00][30][90] 39 21 9 63.57
ADVS BEAS CHKP GATE LU NKE SCOC WMT Up[00][31][81] 26 8 2 52.40
AGE BEL CLGY GE MACR NOVL SNPS XOM
Up[00][20][51] 18 8 2 48.97
AIT BTY CNET GM MERQ ORCL SUNW YHOO
AMZN BVEW CSCO HYSL MO PRGN SYBS Up[01][19][60] 24 9 3 41.70
AOL CA DD IBM MOB PSDI SYMC Down[01][1D][71] 10 0 6 66.00
ARDT CAL DELL IDXC MOT PSFT T
Down[00][11][71] 17 1 6 60.74
AVNT CBS DIS IFMX MRK RATL TSFW
AVTC CBTSY EIDSY INTU MSFT RMDY TWX
Down[01][19][79] 35 3 10 53.05
AWRE CCRD ERTS ITWO MUSE RNWK VRSN Down[00][20][67] 18 2 5 23.11

782

TEAM LinG
Mining for Profitable Patterns in the Stock Market

through December 31, 1999. A stop-loss of 5% was set to
reduce the risk imposed by a wrong signal. This is a
common practice in the investment industry. If a buy
signal is generated, we buy that stock and hold it. The
stock will be sold when it reaches the 10% profit target
or 10 days of holding period, or when its price goes
down 5%. Same rules were applied to the sell signals
but in an opposite way. Table 3 shows the number of buy
and short sell signals generated by these patterns.
As seen from Table 3, the price-up winning pattern
worked very well. 42.86% predictions were perfectly
correct. Also, 20 of the 84 buy signals assured 6.7%
gain after the signals. If we regard 5% increase also as
making money, then in total we had 70.24% chance to
win money, and 85.71% chance of not losing money.
The price-down patterns did not work as well as the
price-up patterns. It was probably because there were
not as many down trends as the up trends in the U.S.
stock market in 1999. Still, by following the sell
signal, there was 43% chance of gaining money and
87.5% chance of not losing money in 1999. The final
return for the year 1999 was 153.8%, which was supe-
rior as compared to 84% return of Nasdaq composite
and 25% return of Dow Industrial Average.
FUTURE TRENDS
Being able to identify price rise or drop patterns can be
exciting for frequent stock traders. By following the
buy or sell signals generated by these patterns,
frequent stock traders can earn excessive returns over
the simple buy-and-hold strategy (Allen & Karjalainen,
1999; Lo & MacKinlay, 1999). Data mining techniques
combined with financial theories can be a powerful ap- M
proach for discovering price movement patterns in the
financial market. Unfortunately, researchers in the data
mining field often focus exclusively on computational
part of market analysis, not paying attention to the theo-
ries of the target area. In addition, the knowledge repre-
sentation methods and variables chosen were often based
on the common sense, rather than theories. This article
borrows the market efficiency theory to model the problem
and the out-of-sample performance was quite pleasing.
We believe there will be more studies integrating theories
in multiple disciplines to achieve better results in the near
future.
CONCLUSION
This paper combines a knowledge discovery technique
with financial theory, the market efficiency theory, to
solve a classic problem in stock market analysis, that is,
finding stock trading patterns that lead to superior fi-
nancial gains. This study is one of a few efforts that go
across multi-disciplines to study the stock market and
the results were quite good.
There are also some future research opportunities in
this direction. For example, the trading volume is not
considered in this research, which it is an important
factor of the stock market, and we believe it is worth
further investigation. Using four or more days K-Line
patterns, instead of just 3-day K-Line patterns, is also
worth exploring.

Table 3. The performance of the chosen winning patterns

Accumulated
Times Percentage
Total Buy Signals 84
Price is up at least 10% after the signal 36 42.86%
Price is up 2/3 of 10%, i.e. 6.7%, after the signal 20 66.67%
Price is up 1/2 of 10%, i.e. 5%, after the signal 3 70.24%
Price is up only 1/10 of 10% after the signal 13 85.71%
Price drops after the signal 12 100.00%
Accumulated
Times Percentage
Total Sell Signals 16
Price is down at least 10% after the signal 4 25.00%
Price is down 2/3 of 10%, i.e. 6.7%, after the signal 2 37.50%
Price is down 1/2 of 10%, i.e. 5%, after the signal 1 43.75%
Price is down only 1/10 of 10% after the signal 7 87.50%
Price raises after the signal 2 100.00%

783

TEAM LinG

REFERENCES
Allen, F., & Karjalainen, R. (1999). Using genetic algo-
rithms to find technical trading rules. Journal of Finan-
cial Economics, 51, 245-271.
Armano, G., Murru, A., & Roli, F. (2002). Stock market
prediction by a mixture of genetic-neural experts. In-
ternational Journal of Pattern Recognition and Arti-
ficial Intelligence, 16, 501-526.
Chenoweth, T., Obradovic, Z., & Stephenlee, S. (1996).
Embedding technical analysis into neural network based
trading systems. Applied Artificial Intelligence, 10,
523-541.
Dawson, E.R., & Steeley, J.M. (2003). On the existence of
visual technical patterns in the UK stock market. Journal
of Business Finance and Accounting, 20, 263-293.
Dourra, H., & Siy, P. (2001). Stock evaluation using
fuzzy logic. International Journal of Theoretical and
Applied Finance, 4, 585-602.
Fama, E.F. (1991). Efficient capital markets: II. Jour-
nal of Finance, 46, 1575-1617.
Liu, H. (1998). Feature selection for knowledge dis-
covery and data mining. Kluwer Academic Publishers.
Lo, A.W., & MacKinlay, A.C. (1999). A non-random
walk down Wall Street. Princeton, NJ: Princeton Uni-
versity Press.
Lo, A.W., Mamaysky, H., & Wang, J. (2000). Founda-
tions of technical analysis: Computational algorithms,
statistical inference, and empirical implementation. The
Journal of Finance, 55, 1705-1765
Pring, M.J. (1991) Technical analysis explained: The
successful investors guide to spotting investment
trends and turning points (3rd ed.). McGraw-Hill Inc.
Sharpe, W.F., Alexander, G.J., & Bailey, J.V. (1999).
Investments (6th ed.). Prentice-Hall.
Siberschatz, A., & Tuzhilin, A. (1996). What makes
patterns interesting in knowledge discovery system.
IEEE Trans. on Knowledge and Data Engineering, 8,
970-974.
784
Mining for Profitable Patterns in the Stock Market
Thawornwong, S., Enke, D., & Dagli, C. (2003). Neural
networks as a decision maker for stock trading: a technical
analysis approach. International Journal of Smart Engi-
neering System Design, 5, 313-325
Tsaih, R., Hsu, Y., & Lai, C.C. (1998). Forecasting S&P
500 stock index futures with a hybrid AI system. Deci-
sion Support Systems, 23, 161-174.
Zaki, M.J., Parthasatathy, S., Ogiharam M., & Li, W.
(1997). New algorithms for fast discovery of associa-
tion rules. In Proceedings of the 3 rd International
Conference on Knowledge Discovery and Data Min-
ing (pp. 283-286).
KEY TERMS
Buy-And-Hold Strategy: An investment strategy
for buying portfolios of stocks or mutual funds with
solid, long-term growth potential. The underlying value
and stability of the investments are important, rather
than the short or medium-term volatility of the market.
Fuzzy Logic: Fuzzy logic provides an approach to
approximate reasoning in which the rules of inference
are approximate rather than exact. Fuzzy logic is useful
in manipulating information that is incomplete, impre-
cise, or unreliable.
Genetic Algorithm: A genetic algorithm is an op-
timization algorithm based on the mechanisms of Dar-
winian evolution, which uses random mutation, cross-
over and selection procedures to breed better models or
solutions from an originally random starting population
or sample.
K-Line: It is an Asian version of stock price bar
chart where a lower close than open on a day period is
shaded dark and a higher close day period is shaded light.
Market Efficiency Theory: A financial theory that
states that stock market prices reflect all available,
relevant information.
Neural Networks: Neural networks are algorithms
simulating the functioning of human neurons and may be
used for pattern recognition problems, for example, to
establish a quantitative structure-activity relationship.
TEAM LinG

Mining for Web-Enabled E-Business
Applications
Richi Nayak
Queensland University of Technology, Australia
INTRODUCTION
A small shop owner builds a relationship with its custom-
ers by observing their needs, preferences and buying
behaviour. A Web-enabled e-business will like to accom-
plish something similar. It is an easy job for the small shop
owner to serve his customers better in future by learning
from past interactions. But, this may not be easy for Web-
enabled e-businesses when most customers may never
interact personally, and the number of customers is much
higher than of the small shop owner.
Data mining techniques can be applied to understand
and analyse e-business data, and turn into actionable
information, that can support a Web enabled e-business
to improve its marketing, sales and customer support
operations. This seems to be more appealing, when data
is produced and stored with advance electronic data
interchange methods, the computing power is affordable,
the competitive pressure among businesses is strong,
and the efficient and commercial data mining tools are
available for data analysis.
BACKGROUND
Data mining is the process of searching the trends, clus-
ters, valuable links and anomalies in the entire data. The
process benefits from the availability of large amount of
data with rich description. The rich descriptions of data
such as wide customer records with many potentially
useful fields allow data mining algorithms to search be-
yond obvious correlations. Examples of data mining in
Web-enabled e-business applications are generation of
user profiles, enabling customer relationship manage-
ment, and targeting Web advertising based on user ac-
Figure 1. A mining process for Web-enabled e-business data
Data Selection &
Locating &
Retrieving Data Quality check &
Web documents & Data Transformation &
Web access logs Data Distribution
Data Gathering Data Processing Data Modelling
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
785
M
cess patterns extracted from the Web data. With the use
of data mining techniques, e-business companies can
improve the sales and quality of the products by anticipat-
ing problems before they occur.
When dealing with Web-enabled e-business data, a
data mining task is decomposed into many sub tasks
(figure 1). The discovered knowledge is presented to user
in an understandable and useable form. The analysis may
reveal how a Web site is useful in making decision for a
user, resulting in improving the Web site. The analysis
may also lead into business strategies for acquiring new
customers and retaining the existing ones.
DATA MINING OPPORTUNITIES
Data obtained from the Web-enabled e-business transac-
tions can be categorised into (1) primary data that in-
cludes actual Web contents, and (2) secondary data that
includes Web server access logs, proxy server logs,
browser logs, registration data if any, user sessions and
queries, cookies, etc (Cooley, 2003; Kosala & Blockeel,
2000).
The goal of mining the primary Web data is to
effectively interpret the searched Web documents. Web
search engines discover resources on the Web but have
many problems such as (1) the abundance problem, where
hundreds of irrelevant data are returned in response to a
search query, (2) limited coverage problem, where only a
few sites are searched for the query instead of searching
the entire Web, (3) limited query interface, where user can
only interact by providing few keywords, (4) limited
customization to individual users, etc (Garofalakis,
Rastogi, Seshadri, & Hyuseok, 1999). Mining of Web
contents can assist e-businesses in improving the orga-
Data Model Learning Information User
& Retrieval Interface
Best Model Selection
Information Information Analysis &
extraction Knowledge Assimilation
TEAM LinG

nization of retrieved result and increasing the precision of
information retrieval. Some of the data mining applica-
tions appropriate for such type of data are:
Trend prediction within the retrieved information to
indicate future values. For example, an e-auction
company provides information about items to auc-
tion, previous auction details, etc. Predictive mod-
elling can analyse the existing information, and as a
result estimate the values for auctioneer items or
number of people participating in future auctions.
Text clustering within the retrieved information. For
example structured relations can be extracted from
unstructured text collections by finding the struc-
ture of Web documents, and present a hierarchical
structure to represent the relation among text data
in Web documents (Wong & Fu, 2000).
Monitoring a competitors Web site to find unex-
pected information e.g. offering unexpected ser-
vices and products. Because of the large number of
competitors Web sites and huge information in
them, automatic discovery is required. For instance,
association rule mining can discover frequent word
combination in a page that will lead a company to
learn about competitors (Liu, Ma, & Yu, 2001).
Categorization of Web pages by discovering simi-
larity and relationships among various Web sites
using clustering or classification techniques. This
will lead into effectively searching the Web for the
requested Web documents within the categories
rather than the entire Web. Cluster hierarchies of
hypertext documents can be created by analysing
semantic information embedded in link structures
and document contents (Kosala & Blockeel, 2000).
Documents can also be given classification codes
according to keywords present in them.
Providing a higher level of organization for semi-
structured or unstructured data available on the
Web. Users do not scan the entire Web site to find
the required information, instead they use Web
query languages to search within the document or
to obtain structural information about Web docu-
ments. A Web query language restructures extracted
information from Web information sources that are
heterogenous and semi-structured (Abiteboul,
Buneman, & Suciu, 2000). An agent based approach
involving artificial intelligent systems can also orga-
nize Web based information (Dignum & Cortes, 2001).
The goal of mining the secondary Web data is to
capture the buying and traversing habits of customers in
an e-business environment. Secondary Web data in-
cludes Web transaction data extracted from Web logs.
786
Mining for Web-Enabled E-Business Applications
Some of the data mining applications appropriate for such
type of data are:
Promoting cross-marketing strategies across prod-
ucts. Data mining techniques can analyse logs of
different sales indicating customers buying pat-
terns (Cooley, 2003). Classification and clustering
of Web access log can help a company to target their
marketing (advertising) strategies to a certain group
of customers. For example, classification rule min-
ing is able to discover that a certain age group of
people from a certain locality are likely to buy a
certain group of products. Web enabled e-business
can also be benefited with link analysis for repeat
buying recommendations. Schulz, Hahsler, & Jahn
(1999) applied link analysis in traditional retail chains,
and found that 70% cross-selling potential exists.
Associative rule mining can find frequent products
bought together. For example, association rule min-
ing can discover rules such as 75% customers who
place an order for product1 from the /company/
product1/ page also place the order for product2
from the /company/product2/ page.
Maintaining or restructuring Web sites to better
serve the needs of customers. Data mining tech-
niques can assist in Web navigation by discovering
authority sites of a users interest, and overview
sites for those authority sites. For instance, asso-
ciation rule mining can discover correlation be-
tween documents in a Web site and thus estimate
the probability of documents being requested to-
gether (Lan, Bressan, & Ooi, 1999). An example
association rule resulting from the analysis of a
travelling e-business company Web data is: 79%
of visitors who browsed pages about Hotel also
browsed pages on visitor information: places to
visit. This rule can be used in redesigning the Web
site by directly linking the authority and overview
Web sites.
Personalization of Web sites according to each
individuals taste. Data mining techniques can as-
sist in facilitating the development and execution of
marketing strategies such as dynamically changing
a particular Web site for a visitor (Mobasher, Cooley,
& Srivastave, 1999). This is achieved by building a
model representing correlation of Web pages and
users. The goal is to find groups of users performing
similar activities. The built model is capable of
categorizing Web pages and users, and matching
between and across Web pages and/or users
(Mobasher, et al, 1999). According to the clusters of
user profiles, recommendations can be made to a
visitor on return visit or to new visitors (Spiliopoulou,
TEAM LinG
Mining for Web-Enabled E-Business Applications
Pohle, & Faulstich, 1999). For example, people ac-
cessing educational products in a company Web site
between 6-8 p.m. on Friday can be considered as
academics and can be focused accordingly.
DIFFICULTIES IN APPLYING DATA
MINING
The idea of discovering knowledge in large amounts of
data with rich description is both appealing and intuitive,
but technically it is challenging. There should be strate-
gies implemented for better analysis of data collected from
Web-enabled e-business sources.
Data Format: Data collected from Web-enabled e-
business sources is semi-structured and hierarchi-
cal. Data has no absolute schema fixed in advance
and the extracted structure may be irregular or incom-
plete.
This type of data requires an additional processing
before applying to traditional mining algorithms
whose source is mostly confined to structured data.
This pre-processing includes transforming unstruc-
tured data to a format suitable for traditional mining
methods. Web query languages can be used to
obtain structural information from semi-structured
data. Based on this structural information, data ap-
propriate to mining techniques are generated. Web
query languages that combine path expressions with
an SQL-style syntax such as Lorel or UnQL
(Abiteboul, et al, 2000) are a good choice for extract-
ing structural information.
Data Volume: Collected e-business data sets are
large in volume. The mining techniques should be
able to handle such large data sets.
Enumeration of all patterns may be expensive and
unnecessary. In spite, selection of representative
patterns that capture the essence of the entire data
set and their use for mining may prove a more effec-
tive approach. But then selection of such data set
becomes a problem. A more efficient approach would
be to use an iterative and interactive technique that
takes account into real time responses and feedback
into calculation. An interactive process involves
human analyst in the process, so an instant feedback
can be included in the process. An iterative process
first considers a selected number of attributes cho-
sen by the user for analysis, and then keeps adding
other attributes for analysis until the user is satis-
fied. This iterative method reduces the search space
significantly.
Data Quality: Web server logs may not contain all the
data needed. Also, noisy and corrupt data can hide
patterns and make predictions harder (Kohavi, 2001).
Nevertheless, the quality of data is increased with M
the use of electronic interchange. There is less
noise present in the data due to electronic storage
and processing in comparison to manual process-
ing of data.
Data warehousing provides a capability for the
good quality data storage. A warehouse integrates
data from operational systems, e-business applica-
tions, and demographic data providers, and handles
issues such as data inconsistency, missing values,
etc. A Web warehouse may be used as data source.
There has been some initiative to warehouse the
Web data generated from e-business applications,
but still long way to go in terms of data mining
(Bhowmick, Madria, & Ng, 2003).
Another solution of collecting the good quality
Web data is the use of (1) a dedicated server
recording all activities of each user individually, or
(2) cookies or scripts in the absence of such server
(Chan, 1999; Kohavi, 2001). The agent based ap-
proaches that involve artificial intelligence sys-
tems can also be used to discover such Web based
information.
Data Adaptability: Data on the Web is ever chang-
ing. Data mining models and algorithms should be
adapted to deal with real-time data such that the
new data is incorporated for analysis. The con-
structed data model should be updated as the new
data approaches.
User-interface agents can be used to maximize the
productivity of current users interactions with the
system by adapting behaviours. Another solution
can be to dynamically modifying mined informa-
tion as the database changes (Cheung & Lee, 2000)
or to incorporate user feedback to modify the ac-
tions performed by the system.
XML Data: It is assumed that in few years XML will
be the most highly used language of Internet in
exchanging information.
Assuming the metadata stored in XML, the inte-
gration of the two disparate data sources becomes
much more transparent, field names are matched
more easily and semantic conflicts are described
explicitly (Abiteboul et al., 2000). As a result, the
types of data input to and output from the learned
models and the detailed form of the models are
determined. XML documents may not completely
be in the same format thus resulting in missing
values when integrated. Various techniques e.g.,
tag recognition can be used to fill missing informa-
tion created from the mismatch in attributes or tags
(Abiteboul et al., 2000). Moreover, many query
languages such as XML-QL, XSL and XML-GL
787
TEAM LinG

(Abiteboul et al., 2000) are designed specifically for
querying XML and getting structured information
from these documents.
Privacy Issues: There are always some concerns of
proper balancing between companys desire to use
personal information versus individuals desire to
protect it (Piastesky-Shapiro, 2000).
The possible solution is to (1) ensure users of
secure and reliable data transfer by using high
speed high-valued data encryption procedures, and/
or (2) give a choice to user to reveal the information
that he wants to and give some benefit in exchange
of revealing their information such as discount on
certain shopping product etc.
FUTURE TRENDS
Earlier data mining tools such as C5 (http://
www.rulequest.com) and several neural network softwares
(QuickLearn, Sompack, etc) were limited to some indi-
vidual researchers. These individual algorithms are ca-
pable of solving a single data mining task. But now the
second generation data mining system produced by
commercial companies such as clementine (http://
www.spss.com/clementine/), AnswerTree (http://
www.spss.com/answertree/), SAS (http://www.sas.com/),
IBM Intelligent.
Miner (http://www.ibm.com/software/data/iminer/)
and DBMiner (http://db.cs.sfu.ca/DBMiner) incorporate
multiple discoveries (classification, clustering, etc), pre-
processing (data cleaning, transformation, etc) and post-
processing (visualization) tasks, and becoming known to
public and successful. Moreover, tools that combine ad
hoc query or OLAP (Online analytical processing) with
data mining are also developed (Wu, 2000). Faster CPU,
bigger disks and easy net connectivity make these tools
liable to analyse large volume of data.
CONCLUSION
It is easy to collect data from Web-enabled e-business
sources as visitors to a Web site leave the trail which
automatically is stored in log files by Web server. The data
mining tools can process and analyse such Web server
log files or Web contents to discover meaningful informa-
tion. This analysis uncovers the previously unknown
buying habits of their online customers to the companies.
More importantly, the fast feedback the companies ob-
tained using data mining is very helpful in increasing the
companys benefit.
788
Mining for Web-Enabled E-Business Applications
REFERENCES
Abiteboul, S., Buneman, P., & Suciu, D. (2000). Data on
the Web: From relations to substructured data and XML.
California: Morgan Kaumann.
Bhowmick, S.S., Madria, S.K., & Ng, W.K. (2003) Web data
management: A warehouse approach. Springer Comput-
ing & Information Science.
Chan, P.K. (1999). A non-invasive learning approach to
building Web user profile. In Masand & Spiliopoulou
(Eds.), WEBKDD99.
Cheung, D.W., & Lee, S.D. (2000). Maintenance of discovered
association rules. In Knowledge Discovery for Business
Information Systems (The Kluwer International Series in
Engineering and Computer Science, 600). Boston: Kluwer
Academic Publishers.
Cooley, R. (2003, May). The use of Web structure and
content to identify subjectively interesting Web usage
patterns. In ACM Transactions on Internet Technology, 3(2).
Dignum, F., & Cortes, U. (Eds.). (2001). Agent-mediated
electronic commerce III: Current issues in agent-based
electronic commerce systems. Lecture Notes in Artificial
Intelligence, Springer Verlag.
Garofalakis, M.N., Rastogi, R., Seshadri, S., & Hyuseok, S.
(1999). Data mining and the Web: Past, present and future.
In Proceedings of the second International Workshop on
Web Information and Data Management (pp. 43-47).
Kohavi, R. (2001). Mining e-commerce data: The good, the
bad and the ugly. In Proceedings of the seventh ACM
SIGKDD International Conference on Knowledge Dis-
covery and Data Mining (KDD 2001).
Kosala, R., & Blockeel, H. (2000, July). Web mining re-
search: A survey. SIGKDD Explorations, 2(1), 1-15.
Lan, B., Bressan, S., & Ooi, B.C. (1999). Making Web
servers pushier. In Masand & Spiliopoulou (Eds.),
WEBKDD99.
Liu, B., Ma, Y., & Yu, P.H. (2001, August). Discovering
unexpected information from your competitors Web sites.
In Proceedings of the seventh ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and Data
Mining (KDD 2001), SanFrancisco, USA.
Masand, B., & Spiliopoulou, M. (1999, August). KDD99
workshop on Web usage analysis and user profiling
(WEBKDD99), San Diego, CA. ACM.
TEAM LinG
Mining for Web-Enabled E-Business Applications
Mobasher, B., Cooley, R., & Srivastave, J. (1999). Auto-
matic personalization based on Web usage mining. In
Masand & Spiliopoulou (Eds.), WEBKDD99.
Piastesky-Shapiro, G. (2000, January). Knowledge dis-
covery in databases: 10 years after. SIGKDD Explora-
tions, 1(2), 59-61, ACM SIGKDD.
Schulz, A.G., Hahsler, M., & Jahn, M. (1999). A customer
purchase incidence model applied to recommendation
service. In Masand & Spiliopoulou (Eds.), WEBKDD99.
Spiliopoulou, M., Pohle, C., & Faulstich, L.C. (1999).
Improving the effectiveness of a Web site with Web
usage mining. In Masand & Spiliopoulou (Eds.),
WEBKDD99.
Wong, W.C., & Fu, A.W. (2000, July). Finding structure
and characteristic of Web documents for classification. In
Proceedings of the ACM SIGMOD Workshop on Research
issues in Data Mining and Knowledge discovery, ACM.
Wu, J. (2000, August). Business intelligence: What is data
mining? In Data Mining Review Online.
KEY TERMS
Clustering Data Mining Task: To identify items with
similar characteristics, and thus creating a hierarchy of
classes from the existing set of events. A data set is
partitioned into segments of elements (homogeneous)
that share a number of properties.
Data Mining (DM) or Knowledge Discovery in Data-
bases: The extraction of interesting, meaningful, implicit,
previously unknown, valid and actionable information
from a pool of data sources.
Link Analysis Data Mining Task: Establishes inter-
nal relationship to reveal hidden affinity among items in M
a given data set. Link analysis exposes samples and
trends by predicting correlation of items that are other-
wise not obvious.
Mining of Primary Web Data: Assists to effectively
interpret the searched Web documents. Output of this
mining process can help e-business customers to improve
the organization of retrieved result and to increase the
precision of information retrieval.
Mining of Secondary Web Data: Assists to capture
the buying and traversing habits of customers in an e-
business environment. Output of this mining process can
help e-business to predicting customer behaviour in fu-
ture, to personalization of Web sites, to promoting cam-
paign by cross-marketing strategies across products.
Predictive Modelling Data Mining Task: Makes pre-
dictions based on essential characteristics about the
data. The classification task of data mining builds a model
to map (or classify) a data item into one of several pre-
defined classes. The regression task of data mining builds
a model to map a data item to a real-valued prediction
variable.
Primary Web Data: Includes actual Web contents.
Secondary Web Data: Includes Web transaction data
extracted from Web logs Examples are Web server access
logs, proxy server logs, browser logs, registration data if
any, user sessions, user queries, cookies, product corre-
lation and feedback from the customer companies.
Web-Enabled E-Business: A business transaction or
interaction in which participants operate or transact busi-
ness or conduct their trade electronically on the Web.
789
TEAM LinG
790
Mining Frequent Patterns via Pattern
Decomposition
Qinghua Zou
University of California - Los Angeles, USA
Wesley Chu
University of California - Los Angeles, USA
INTRODUCTION
Pattern decomposition is a data-mining technology that
uses known frequent or infrequent patterns to decom-
pose a long itemset into many short ones. It finds frequent
patterns in a dataset in a bottom-up fashion and reduces
the size of the dataset in each step. The algorithm avoids
the process of candidate set generation and decreases the
time for counting supports due to the reduced dataset.
BACKGROUND
A fundamental problem in data mining is the process of
finding frequent itemsets (FI) in a large dataset that enable
essential data-mining tasks, such as discovering associa-
tion rules, mining data correlations, and mining sequential
patterns. Three main classes of algorithms have been
proposed:
Candidates Generation and Test (Agrawal
&Srikant, 1994; Heikki, Toivonen &Verkamo,
1994; Zaki et al., 1997): Starting at k=0, it first
generates candidate k+1 itemsets from known fre-
quent k itemsets and then counts the supports of
the candidates to determine frequent k+1 itemsets
that meet a minimum support requirement.
Sampling Technique (Toivonen, 1996): Uses a sam-
pling method to select a random subset of a dataset
for generating candidate itemsets and then tests
these candidates to identify frequent patterns. In
general, the accuracy of this approach is highly
dependent on the characteristics of the dataset and
the sampling technique that has been used.
Data Transformation: Transforms an original
dataset to a new one that contains a smaller search
space than the original dataset. FP-tree-based (Han,
Pei & Yin, 2000) mining first builds a compressed
data representation from a dataset, and then, mining
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
tasks are performed on the FP-tree rather than on the
dataset. It has performance improvements over
Apriori (Agrawal &Srikant, 1994), since infrequent
items do not appear on the FP-tree, and, thus, the FP-
tree has a smaller search space than the original
dataset. However, FP-tree cannot reduce the search
space further by using infrequent 2-item or longer
itemsets.
What distinguishes pattern decomposition (Zou et
al., 2002) from most previous works is that it reduces the
search space of a dataset in each step of its mining
process.
MAIN THRUST
Both the technology and application will be discussed to
help clarify the meaning of pattern decomposition.
Search Space Definition
Let N=X:Y be a transaction where X, called the head of N,
is the set of required items, and Y, called the tail of N, is the
set of optional items. The set of possible subsets of Y is
called the power set of Y, denoted by P(Y).
Definition 1
For N=X:Y, the set of all the itemsets obtained by
concatenating X with the itemsets in P(Y) is called the
search space of N, denoted as {X:Y}. That is,
{ X : Y } = { X V | V P(Y )}.
For example, the search space {b:cd} includes four
itemsets b, bc, bd, and bcd. The search space {:abcde}
includes all subsets of abcde.
TEAM LinG
Mining Frequent Patterns via Pattern Decomposition
By definition 1, we have {X:Y}={X:Z}, where Z=Y-X
refer to the set of items contained in Y but not in X. Thus,
we will assume that Y does not contain any item in X, when
{X:Y} is mentioned in this article.
Definition 2
Let S, S1, and S2 be search spaces. The set {S1, S2} is a
partition of S if and only if S= S 1 S2 and S1 S2= . The
relationship is denoted by S=S1+S 2 or S1= S-S2 or S 2= S-S1.
We say S is partitioned into S1 and S2. Similarly, a set {S1,
S2, , Sk} is a partition of S if and only if S= S1 S2 Sk
and Si Sj= for i,j [1..k] and i j. We denote it as
S=S1+S2++Sk.
Let a be an item where aX is an itemset by concatenat-
ing a with X.
Theorem 1
For a X,Y, the search space {X:aY} can be partitioned
into {Xa:Y} and {X:Y} by item a (i.e.,
{X:aY}={Xa:Y}+{X:Y}).
Proof
It follows from the fact that each itemset of {X:aY} either
contains a (i.e., {Xa:Y}) or does not contain a (i.e., {X:Y}).
For example, we have {b:cd}={bc:d}+{b:d}.
Theorem 2
Partition Search Space: Let a1, a2, , ak be distinct
items and a1a2akY be an itemset; the search space
of {X: a1a2akY} can be partitioned into
k For example, if we know that an itemset abc is frequent,
{Xa
i =1
i : ai +1 K a k Y } + { X : Y }, where ai X ,Y .
Proof
It follows by partitioning the search space via items
a1,a2,,ak sequentially as in theorem 1.
For example, we have {b:cd}={bc:d}+{bd:}+{b:} and
{a:bcde}= {ab:cde} +{ac:de}+{a:de}.
Let {X:Y} be a search space and Z be a known frequent
itemset. Since Z is frequent, all subsets of Z will be frequent
(i.e., every itemset of {:Z} is frequent). Theorem 3 shows
how to prune the space {X:Y} by Z.
Theorem 3
M
Pruning Search Space: If Z does not contain the
head X, the space {X:Y} cannot be pruned by Z (i.e.,
{X:Y}-{:Z}={X:Y}). Otherwise, the space can be
pruned as
k
{X:Y}-{:Z} = { Xai : ai +1 ...ak (Y Z )} , a1a2ak=Y-Z.
i =1
Proof
If Z does not contain X, no itemset in {X:Y} is subsumed
by Z. Therefore, knowing that Z is frequent, we cannot
prune any part of the search space {X:Y}.
Otherwise, when X is a subset of Z, we have
k
{X:Y}= { Xai : ai +1 ...a kV } + X : V , where V=Y Z. The
i =1
head in the first part is Xai where ai is a member of Y-Z.
Since Z does not contain ai, the first part cannot be pruned
by Z. For the second part, we have {X:V}-{:Z}={X:V}-
{X:(Z-X)}. Since X Y= , we have V Z-X. Therefore,
{X:V} can be pruned away entirely.
For example, we have {:bcde}-{:abcd} = {:bcde}-{:bcd}
= {e:bcd}. Here, a is irrelevant and is removed in the first
step. Another example is {e:bcd}-{:abe} = {e:bcd}-{:be}
= {e:bcd}-{e:b} = {ec:bd}+{ed:b}.
Pattern Decomposition
Given a known frequent itemset Z, we are able to decom-
pose the search space of a transaction N=X:Y to N=Z:Y,
if X is a subset of Z, where Y is the set of items that appears
in Y but not in Z, denoted by PD(N=X:Y|Z)= Z:Y.
we can decompose a transaction N=a:bcd into N=abc:d;
that is, PD(a:bcd|abc)=abc:d.
Given a known infrequent itemset Z, we also can
decompose the search space of a transaction N=X:Y. For
simplicity, we use three examples to show the decompo-
sition by known infrequent itemsets and leave out its
formal mathematic formula in general cases. Interested
readers can refer to Zou, Chu, and Lu (2002) for details. For
example, if N=d:abcef and a known infrequent itemsets,
then we have:
For infrequent 1-itemset ~a, PD(d:abcef|~a) = d:bcef
by dropping a from its tail.
For infrequent 2-itemset ~ab, PD(d:abcef|~ab) =
d:bcef+da:cef by excluding ab.
791
TEAM LinG

For infrequent 3-itemset ~abc, PD(d:abcef|~abc) =
d:bcef+da:cef+dab:ef by excluding abc.
By decomposing a transaction t, we reduce the number
of items in its tails and thus reduce its search space. For
example, the search space of a:bcd contains the following
eight itemsets {a, ab, ac, ad, abc, abd, acd, abcd}. Its
decomposition result, abc:d, contains only two itemsets
{abc, abcd}, which is only 25% of its original search space.
When using pattern decomposition, we find frequent
patterns in a step-wise fashion starting at step 1 for 1-item
itemsets. At step k, it first counts the support for every
possible k-item itemsets contained in the dataset Dk to find
frequent k-item itemsets Lk and infrequent k-item itemsets
~L k. Then, using the Lk and ~Lk, Dk, they can be decom-
posed into Dk+1, which has a smaller search space than Dk.
These steps continue until the search space Dk becomes
empty.
An Application
The motivation of our work originates from the problem of
finding multi-word combinations in a group of medical
report documents, where sentences can be viewed as
transactions and words can be viewed as items. The
problem is to find all multi-word combinations that occur
at least in two sentences of a document.
As a simple example, for the following text:
Aspirin greatly underused in people with heart disease.
DALLAS (AP) Too few heart patients are taking aspirin,
despite its widely known ability to prevent heart attacks,
according to a study released Monday.
The study, published in the American Heart Associations
journal Circulation, found that only 26% of patients who
had heart disease and could have benefited from aspirin
took the pain reliever.
This suggests that theres a substantial number of
patients who are at higher risk of more problems because
theyre not taking aspirin, said Dr. Randall Stafford, an
internist at Harvards Massachusetts General Hospital,
who led the study. As we all know, this is a very
inexpensive medication very affordable.
The regular use of aspirin has been shown to reduce the
risk of blood clots that can block an artery and trigger a
heart attack. Experts say aspirin also can reduce the risk
of a stroke and angina, or severe chest pain.
Because regular aspirin use can cause some side effects,
such as stomach ulcers, internal bleeding, and allergic
reactions, doctors too often are reluctant to prescribe it
for heart patients, Stafford said.
Theres a bias in medicine toward treatment, and within
that bias, we tend to underutilize preventative services,
even if theyve been clearly proven, said Marty Sullivan,
792
Mining Frequent Patterns via Pattern Decomposition
a professor of cardiology at Duke University in Durham,
North Carolina.
Staffords findings were based on 1996 data from 10,942
doctor visits by people with heart disease. The study
may underestimate aspirin use; some doctors may not
have reported instances in which they recommended
patients take over-the-counter medications, he said.
He called the data a wake-up call to doctors who
focus too much on acute medical problems and ignore
general prevention.
We can find frequent one-word, two-word, three-
word, four-word, and five-word combinations. For in-
stance, we found 14 four-word combinations.
heart aspirin use regul, aspirin they take not, aspirin
patient take not, patient doct use some, aspirin patient
study take, patient they take not, aspirin patient use
some, aspirin doct use some, aspirin patient they not,
aspirin patient they take, aspirin patient doct some,
heart aspirin patient too, aspirin patient doct use, heart
aspirin patient study.
Multi-word combinations are effective for document
indexing and summarization. The work in Johnson, et al.
(2002) shows that multi-word combinations can index
documents more accurately than single-word indexing
terms. Multi-word combinations can delineate the con-
cepts or content of a domain-specific document collec-
tion more precisely than single word. For example, from
the frequent one-word table, we may infer that heart,
aspirin, and patient are the most important concepts in
the text, since they occur more often than others. For the
frequent two-word table, we see a large number of two-
word combinations with aspirin (i.e., aspirin patient,
heart aspirin, aspirin use, aspirin take, etc.). This infers
that the document emphasizes aspirin and aspirin-re-
lated topics more than any other words.
FUTURE TRENDS
There is a growing need for mining frequent sequence
patterns from human genome datasets. There are 23 pairs
of human chromosomes, approximately 30,000 genes, and
more than 1,000,000 proteins. The previously discussed
pattern decomposition method can be used to capture
sequential patterns with some small modifications.
When the frequent patterns are long, mining frequent
itemsets (FI) are infeasible because of the exponential
number of frequent itemsets. Thus, algorithms mining
frequent closed itemsets (FCI) (Pasquier, Bastide, Taouil
& Lakhal, 1999; Pei, Han & Mao, 2000; Zaki & Hsiao, 1999)
are proposed, since FCI is enough to generate associa-
TEAM LinG
Mining Frequent Patterns via Pattern Decomposition
tion rules. However, FCI also could be as exponentially
large as the FI. As a result, many algorithms for mining
maximal frequent itemsets (MFI) are proposed, such as
Mafia (Burdick, Calimlim & Gehrke, 2001), GenMax (Gouda
& Zaki, 2001), and SmartMiner (Zou, Chu & Lu, 2002).
The main idea of pattern decomposition also is used
in SmartMiner, except that SmartMiner uses tail informa-
tion (frequent itemsets) to decompose the search space of
a dataset rather than the dataset itself. While pattern
decomposition avoids candidate set generation,
SmartMiner avoids superset checking, which is a time-
consuming process.
CONCLUSION
We propose to use pattern decomposition to find fre-
quent patterns in large datasets. The PD algorithm shrinks
the dataset in each pass so that the search space of the
dataset is reduced. Pattern decomposition avoids the
costly candidate set generation procedure, and using
reduced datasets greatly decreases the time for support
counting.
ACKNOWLEDGMENT
This research is supported by NSF IIS ITR Grant # 6300555.
REFERENCES
Agrawal, R. & Srikant, R. (1994). Fast algorithms for
mining association rules. Proceedings of the 1994 Inter-
national Conference on Very Large Data Bases.
Burdick, D., Calimlim, M., & Gehrke, J. (2001). MAFIA: A
maximal frequent itemset algorithm for transactional data-
bases. Proceedings of the International Conference on
Data Engineering.
Gouda, K., & Zaki, M.J. (2001). Efficiently mining maximal
frequent itemsets. Proceedings of the IEEE International
Conference on Data Mining, San Jose, California.
Han, J., Pei, J., & Yin, Y. (2000). Mining frequent patterns
without candidate generation. Proceedings of the 2000
ACM International Conference on Management of Data,
Dallas, Texas.
Heikki, M., Toivonen, H., & Verkamo, A.I. (1994). Efficient
algorithms for discovering association rules. Proceed-
ings of the AAAI Workshop on Knowledge Discovery in
Databases, Seattle, Washington.
Johnson, D., Zou, Q., Dionisio, J.D., Liu, Z., Chu, W.W.
(2002). Modeling medical content for automated summa- M
rization. Annals of the New York Academy of Sciences.
Pasquier, N., Bastide, Y., Taouil, R., & Lakhal, L. (1999).
Discovering frequent closed itemsets for association
rules. Proceedings of the 7th International Conference
on Database Theory.
Pei, J., Han, J., & Mao, R. (2000). Closet: An efficient
algorithm for mining frequent closed itemsets. Proceed-
ings of the SIGMOD International Workshop on Data
Mining and Knowledge Discovery.
Toivonen, H. (1996). Sampling large databases for asso-
ciation rules. Proceedings of the 22nd International
Conference on Very Large Data Bases, Bombay, India.
Zaki, M.J., & Hsiao, C. (1999). Charm: An efficient algo-
rithm for closed association rule mining. Technical Re-
port 99-10. Rensselaer Polytechnic Institute.
Zaki, M.J., Parthasarathy, S., Ogihara, M., & Li, W. (1997).
New algorithms for fast discovery of association rules.
Proceedings of the Third International Conference on
Knowledge Discovery in Databases and Data Mining.
Zou, Q., Chu, W., Johnson, D., & Chiu, H. (2002). Using
pattern decomposition (PD) methods for finding all fre-
quent patterns in large datasets. Journal Knowledge and
Information Systems (KAIS).
Zou, Q., Chu, W., & Lu, B. (2002). SmartMiner: A depth
first algorithm guided by tail information for mining maxi-
mal frequent itemsets. Proceedings of the IEEE Interna-
tional Conference on Data Mining, Japan.
KEY TERMS
Frequent Itemset (FI): An itemset whose support is
greater than or equal to the minimal support.
Infrequent Pattern: An itemset that is not a frequent
pattern.
Minimal Support (minSup): A user-given number that
specifies the minimal number of transactions in which an
interested pattern should be contained.
Pattern Decomposition: A technique that uses known
frequent or infrequent patterns to reduce the search space
of a dataset.
Search Space: The union of the search space of every
transaction in a dataset.
793
TEAM LinG

Search Space of a Transaction N=X:Y: The set of
unknown frequent itemsets contained by N. Its size is
decided by the number of items in the tail of N, i.e. Y.
Support of an Itemset x: The number of transactions
that contains x.
794
Mining Frequent Patterns via Pattern Decomposition
Transaction: An instance that usually contains a set
of items. In this article, we extend a transaction to a
composition of a head and a tail (i.e., N=X:Y), where the
head represents a known frequent itemset, and the tail is
the set of items for extending the head for new frequent
patterns.
TEAM LinG

Mining Group Differences
Shane M. Butler
Monash University, Australia
Geoffrey I. Webb
Monash University, Australia
INTRODUCTION
Finding differences among two or more groups is an
important data-mining task. For example, a retailer might
want to know what the different is in customer purchas-
ing behaviors during a sale compared to a normal trading
day. With this information, the retailer may gain insight
into the effects of holding a sale and may factor that into
future campaigns. Another possibility would be to in-
vestigate what is different about customers who have a
loyalty card compared to those who dont. This could
allow the retailer to better understand loyalty
cardholders, to increase loyalty revenue, or to attempt
to make the loyalty program more appealing to non-
cardholders.
This article gives an overview of such group mining
techniques. First, we discuss two data-mining methods
designed specifically for this purposeEmerging Pat-
terns and Contrast Sets. We will discuss how these two
methods relate and how other methods, such as explor-
atory rule discovery, can also be applied to this task.
Exploratory data-mining techniques, such as the tech-
niques used to find group differences, potentially can
result in a large number of models being presented to the
user. As a result, filter mechanisms can be a useful way to
automatically remove models that are unlikely to be of
interest to the user. In this article, we will examine a
number of such filter mechanisms that can be used to
reduce the number of models with which the user is
confronted.
BACKGROUND
There have been two main approaches to the group
discovery problem from two different schools of
thought. The first, Emerging Patterns, evolved as a clas-
sification method, while the second, Contrast Sets,
grew as an exploratory method. The algorithms of both
approaches are based on the Max-Miner rule discovery
system (Bayardo Jr., 1998). Therefore, we will briefly
describe rule discovery.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
795
M
Rule discovery is the process of finding rules that best
describe a dataset. A dataset is a collection of records in
which each record contains one or more discrete attribute-
value pairs (or items). A rule is simply a combination of
conditions that, if true, can be used to predict an outcome.
A hypothetical rule about consumer purchasing behav-
iors, for example, might be IF buys_milk AND
buys_cookies THEN buys_cream.
Association rule discovery (Agrawal, Imielinski &
Swami, 1993; Agrawal & Srikant, 1994) is a popular
rule-discovery approach. In association rule mining,
rules are sought specifically in the form of where the
antecedent group of items (or itemset), A, implies the
consequent itemset, C. An association rule is written as
A C . Of particular interest are the rules where the
probability of C is increased when the items in A also
occur. Often association rule-mining systems restrict
the consequent itemset to hold only one item as it
reduces the complexity of finding the rules.
In association rule mining, we often are searching
for rules that fulfill the requirement of a minimum
support criteria, minsup, and a minimum confidence
criteria, minconf. Where support is defined as the fre-
quency with which A and C co-occur:
support( A C ) = frequency(A C )
and confidence is defined as the frequency with which A
and C co-occur, divided by the frequency with which A
occurs throughout all the data:
support ( A C )
confidence(A C ) =
frequency( A)
The association rules discovered through this pro-
cess then are sorted according to some user-specified
interestingness measure before they are displayed to
the user.
Another type of rule discovery is k-most interesting
rule discovery (Webb, 2000). In contrast to the support-
confidence framework, there is no minimum support or
TEAM LinG

confidence requirement. Instead, k-most interesting rule
discovery focuses on the discovery of up to k rules that
maximize some user-specified interestingness measure.
MAIN THRUST
Emerging Patterns
Emerging Pattern analysis is applied to two or more
datasets, where each dataset contains data relating to a
different group. An Emerging Pattern is defined as an
itemset whose support increases significantly from one
group to another (Dong & Li, 1999). This support in-
crease is represented by the growth ratethe ratio of
support of an itemset in group 1 over that of group 2. The
support of a group G is given by:
count G ( X )
supp G (X ) =
G support (cset, Gi ) support (cset , G j ) mindev
The GrowthRate( X ) is defined as 0 if supp1 (X ) = 0 and
supp 2 (X ) = 0 ; if supp1 (X ) = 0 and supp 2 ( X ) 0 ; or
else supp 2 ( X ) supp1 (X ) . The special case where
GrowthRate(X ) = is called a Jumping Emerging Pattern,
as it is said to have jumped from not occurring in one group
to occurring in another group. This also can be thought
of as an association rule having a confidence equaling 1.0.
Emerging Patterns are not presented to the user, as
models are in the exploratory discovery framework. Rather,
the Emerging Pattern discovery research has focused on
using the mined Emerging Patterns for classification,
similar to the goals of Liu et al. (1998, 2001). Emerging
Pattern mining-based classification systems include CAEP
(Dong, Zhang, Wong & Li, 1999), JEP-C (Li, Dong &
Ramamohanarao, 2001), BCEP (Fan & Ramamohanarao,
2003), and DeEP (Li, Dong, Ramamohanarao & Wong,
2004). Since the Emerging Patterns are classification based,
the focus is on classification accuracy. This means no
filtering method is used, other than the infinite growth rate
constraint used during discovery by some the classifiers
(e.g., JEP-C and DeEP). This constraint discards any
Emerging Pattern X for which GrowthRate(X ) .
Contrast Sets
Contrast Sets (Bay & Pazzani, 1999, 2001) are similar to
Emerging Patterns, in that they are also itemsets whose
support differs significantly across datasets. However,
the focus of Contrast Set research has been to develop an
796
Mining Group Differences
exploratory method for finding differences between one
group and another that the user can utilize, rather than as
a classification system focusing on prediction accuracy.
To this end, they present filtering and pruning methods
to ensure only the most interesting and optimal number
rules are shown to the user, from what is potentially a large
space of possible rules.
Contrast Sets are discovered using STUCCO, an algo-
rithm that is based on the Max-Miner search algorithm
(Bayardo Jr., 1998). Initially, only Contrast Sets are sought
that have supports that are both significant and the
difference large (i.e., the difference is greater than a user-
defined parameter, mindev). Significant Contrast Sets
(cset), therefore, are defined as those that meet the criteria:
P(cset | Gi ) P (cset | G j )
Large Contrast Sets are those for which:
As Bay and Pazzani have noted, the user is likely to be
overwhelmed by the number of results. Therefore, a filter
method is applied to reduce the number of Contrast Sets
presented to the user and to control the risk of type-1 error
(i.e., the risk of reporting a Contrast Set when no differ-
ence exists). The filter method employed involves a chi-
square test of statistical significance between the itemset
on one group to that Contrast Set on the other group(s).
A correction for multiple comparisons is applied that
lowers the value of as the size of the Contrast Set
(number of attribute value pairs) increases.
Further pruning mechanisms also are used to filter
Contrast Sets that are purely specializations of other more
general Contrast Sets. This is done using another chi-
square test of significance to test the difference between
the parent Contrast Set and its specialization Contrast Set.
Mining Group Differences Using Rule
Discovery
Webb, Butler, and Newlands (2003) studied how Con-
trast Sets relate to generic rule discovery approaches.
They used the OPUS_AR algorithm-based Magnum Opus
software to discover rules and to compare them to those
discovered by the STUCCO algorithm.
OPUS_AR (Webb, 2000) is a rule-discovery algo-
rithm based on the OPUS (Webb, 1995) efficient search
technique, to which the Max-Miner algorithm is closely
related. By limiting the consequent to a group variable,
this rule discovery framework is able to be adapted for
group discovery.
TEAM LinG
Mining Group Differences
While STUCCO and Magnum Opus specify different
support conditions in the discovery phase, their condi-
tions were proven to be equivalent (Webb et al., 2003).
Further investigation found that the key difference be-
tween the two techniques was the filtering technique.
Magnum Opus uses a binomial sign test to filter spurious
rules, while STUCCO uses a chi-square test. STUCCO
attempts to control the risk of type-1 error by applying a
correction for multiple comparisons. However, such a
correction, when given a large number of tests, will reduce
the value to an extremely low number, meaning that the
risk of type-2 error (i.e., the risk of not accepting a non-
spurious rule) is substantially increased. Magnum Opus
does not apply such corrections so as not to increase the
risk of type-2 error.
While a chi-square approach is likely to be better suited
to Contrast Set discovery, the correction for multiple com-
parisons, combined with STUCCOs minimum difference, is
a much stricter filter than that employed by Magnum Opus.
As a result of Magnum Opus much more lenient filter
mechanisms, many more rules are being presented to the end
user. After finding that the main difference between the
systems was their control of type-1 and type-2 errors via
differing statistical test methods, Webb, et al. (2003) con-
cluded that Contrast Set mining is, in fact, a special case of
the rule discovery task.
Experience has shown that filters are important for
removing spurious rules, but it is not obvious which of the
filtering methods used by systems like Magnum Opus and
STUCCO is better suited to the group discovery task.
Given the apparent tradeoff between type-1 and type-2
error in these data-mining systems, recent developments
(Webb, 2003) have focused on a new filter method to avoid
introducing type-1 and type-2 errors. This approach di-
vides the dataset into exploratory and holdout sets. Like
the training and test set method of statistically evaluating
a model within the classification framework, one set is used
for learning (the exploratory set) and the other is used for
evaluating the models (the holdout set). A statistical test
then is used for the filtering of spurious rules, and it is
statistically sound, since the statistical tests are applied
using a different set. A key difference between the tradi-
tional training and test set methodology of the classifica-
tion framework and the new holdout technique is that many
models are being evaluated in the exploratory framework
rather than only one model in the classification framework.
We envisage the holdout technique will be one area of
future research, as it is adapted by exploratory data-mining
techniques as a statistically sound filter method.
Case Study
In order to evaluate STUCCO and the more lenient Magnum
Opus filter mechanisms, Webb, Butler, and Newlands (2003)
conducted a study with a retailer to find interesting pat-
terns between transactions from two different days. This M
data was traditional market-basket transactional data, con-
taining the purchasing behaviors of customers across the
many departments. Magnum Opus was used with the
group, encoded as a variable and the consequent re-
stricted to that variable only.
In this experiment, Magnum Opus discovered all of
the Contrast Sets that STUCCO found, and more. This is
indicative of the more lenient filtering method of Mag-
num Opus. It was also interesting that, while all of the
Contrast Sets discovered by STUCCO were only of size
1, Magnum Opus discovered conjunctions of sizes up to
three department codes.
This information was presented to the retail market-
ing manager in the form of a survey. For each rule, the
manager was asked if the rule was surprising and if it was
potentially useful to the organization. For ease of under-
standing, the information was transformed into a plain
text statement.
The domain expert judged a greater percentage of the
Magnum Opus rules of surprise than the STUCCO con-
trasts; however, the result was not statistically signifi-
cant. The percentage of rules found that potentially were
useful were similar for both systems. In this case, Mag-
num Opus probably found some rules that were spurious,
and STUCCO probably failed to discover some rules that
were potentially interesting.
FUTURE TRENDS
Mining differences among groups will continue to grow
as an important research area. One area likely to be of
future interest is improving filter mechanisms. Experi-
ence has shown that the use of filter is important, as it
reduces the number of rules, thus avoiding overwhelm-
ing the user. There is a need to develop alternative filters
as well as to determine which filters are best suited to
different types of problems.
An interestingness measure is a user-generated speci-
fication of what makes a rule potentially interesting.
Interestingness measures are another important issue,
because they attempt to reflect the users interest in a
model during the discovery phase. Therefore, the devel-
opment of new interestingness measures and determina-
tion of their appropriateness for different tasks are both
expected to be areas of future study.
Finally, while the methods discussed in this article
focus on discrete attribute-value data, it is likely that
there will be future research on how group mining can
utilize quantitative, structural, and sequence data. For
example, group mining of sequence data could be used
to investigate what is different about the sequence of
797
TEAM LinG

events between fraudulent and non-fraudulent credit
card transactions.
CONCLUSION
We have presented an overview of techniques for mining
differences among groups, discussing Emerging Pattern
discovery, Contrast Set discovery, and association rule
discovery approaches. Emerging Patterns are useful in a
classification system where prediction accuracy is the
focus but are not designed for presenting the group
differences to the user and thus dont have any filters.
Exploratory data mining can result in a large number of
rules. Contrast Set discovery is an exploratory technique
that includes mechanisms to filter spurious rules, thus
reducing the number of rules presented to the user. By
forcing the consequent to be the group variable during
rule discovery, generic rule discovery software like Mag-
num Opus can be used to discover group differences. The
number of differences reported to the user by STUCCO
and Magnum Opus are related to the different filter mecha-
nisms for controlling the output of potentially spurious
rules. Magnum Opus uses a more lenient filter than
STUCCO and thus presents more rules to the user. A new
method, the holdout technique, will be an improvement
over other filter methods, since the technique is statisti-
cally sound.
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
association rules between sets of items in large data-
bases. Proceedings of the 1993 ACM SIGMOD Interna-
tional Conference on Management of Data, Washing-
ton, D.C., USA.
Agrawal, R., & Srikant, R. (1994). Fast algorithms for
mining association rules. Proceedings of the 20th Inter-
national Conference on Very Large Data Bases, Santiago,
Chile.
Bay, S.D., & Pazzani, M.J. (1999). Detecting change in
categorical data: Mining contrast sets. Proceedings of
the Fifth ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, San Diego,
USA.
Bay, S.D., & Pazzani, M.J. (2001). Detecting group differ-
ences: Mining contrast sets. Data Mining and Knowl-
edge Discovery, 5(3), 213-246.
798
Mining Group Differences
Bayardo, Jr., R.J. (1998). Efficiently mining long patterns
from databases. Proceedings of the 1998 ACM SIGMOD
International Conference on Management of Data, 85-93,
Seattle, Washington, USA.
Dong, G., & Li, J. (1999). Efficient mining of emerging
patterns: Discovering trends and differences. Proceed-
ings of the Fifth International Conference on Knowledge
Discovery and Data Mining, San Diego, California, USA.
Dong, G., Zhang, X., Wong, L., & Li, J. (1999). CAEP:
Classification by aggregating emerging patterns. Pro-
ceedings of the Second International Conference on
Discovery Science, Tokyo, Japan.
Fan, H., & Ramamohanarao, K. (2003). A Bayesian
approach to use emerging patterns for classification.
Proceedings of the 14th Australasian Database Confer-
ence, Adelaide, Australia.
Li, J., Dong, G., & Ramamohanarao, K. (2001). Making
use of the most expressive jumping emerging patterns
for classification. Knowledge and Information Sys-
tems, 3(2), 131-145.
Li, J., Dong, G., Ramamohanarao, K., & Wong, L. (2004).
DeEPs: A new instance-based lazy discovery and classi-
fication system. Machine Learning, 54(2), 99-124.
Liu, B., Hsu, W., & Ma, Y. (1998). Integrating classifi-
cation and association rule mining. Proceedings of the
Fourth International Conference on Knowledge Dis-
covery and Data Mining, New York, New York.
Liu, B., Ma, Y., & Wong, C.K. (2001). Classification
using association rules: Weaknesses and enhancements.
In V. Kumar et al. (Eds.), Data mining for scientific and
engineering applications (pp. 506-605). Boston: Kluwer
Academic Publishing.
Webb, G.I. (1995). An efficient admissible algorithm for
unordered search. Journal of Artificial Intelligence Re-
search, 3, 431-465.
Webb, G.I. (2000). Efficient search for association
rules. Proceedings of the Sixth ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and Data
Mining, Boston, Massachesetts, USA.
Webb, G.I. (2003). Preliminary investigations into sta-
tistically valid exploratory rule discovery. Proceedings
of the Australasian Data Mining Workshop, Canberra,
Australia.
Webb, G.I., Butler, S.M., & Newlands, D. (2003). On
detecting differences between groups. Proceedings of
the Ninth ACM SIGKDD International Conference on
TEAM LinG
Mining Group Differences
Knowledge Discovery and Data Mining, Washington,
D.C., USA.
KEY TERMS
Association Rule: A rule relating two itemsets
the antecedent and the consequent. The rule indicates
that the presence of the antecedent implies that the
consequent is more probable in the data. Written as
AC.
Contrast Set: Similar to an Emerging Pattern, it is also
an itemset whose support differs across groups. The main
difference is the methods application as an exploratory
technique rather than as a classification one.
Emerging Pattern: An itemset that occurs signifi-
cantly more frequently in one group than another. Uti-
lized as a classification method by several algorithms.
Filter Technique: Any technique for reducing the
number of models with the aim of avoiding overwhelm-
ing the user.
Growth Rate: The ratio of the proportion of data
covered by the Emerging Pattern in one group over the M
proportion of the data it covers in another group.
Holdout Technique: A filter technique that splits the
data into exploratory and holdout sets. Rules discovered
from the exploratory set then can be evaluated against the
holdout set using statistical tests.
Itemset: A conjunction of items (attribute-value pairs)
(e.g., age = teen hair = brown ).
k-Most Interesting Rule Discovery: The process of
finding k rules that optimize some interestingness mea-
sure. Minimum support and/or confidence constraints are
not used.
Market Basket: An itemset; this term is sometimes
used in the retail data-mining context, where the itemsets
are collections of products that are purchased in a single
transaction.
Rule Discovery: The process of finding rules that
then can be used to predict some outcome (e.g., IF 13
<= age <= 19 THEN teenager).
799
TEAM LinG
800
Mining Historical XML
Qiankun Zhao
Nanyang Technological University, Singapore
Sourav Saha Bhowmick
Nanyang Technological University, Singapore
INTRODUCTION
Nowadays the Web poses itself as the largest data re-
pository ever available in the history of humankind
(Reis et al., 2004). However, the availability of huge
amount of Web data does not imply that users can get
whatever they want more easily. On the contrary, the
massive amount of data on the Web has overwhelmed
their abilities to find the desired information. It has
been claimed that 99% of the data reachable on the Web
is useless to 99% of the users (Han & Kamber, 2000, pp.
436). That is, an individual may be interested in only a
tiny fragment of the Web data. However, the huge and
diverse properties of Web data do imply that Web data
provides a rich and unprecedented data mining source.
Web mining was introduced to discover hidden knowl-
edge from Web data and services automatically (Etzioni,
1996). According to the type of Web data, Web mining
can be classified into three categories: Web content
mining, Web structure mining, and Web usage mining
(Madria et al., 1999). Web content mining is to extract
patterns from online information such as HTML files,
e-mails, or images (Dumais & Chen, 2000; Ester et al.,
2002). Web structure mining is to analysis the link
structures of Web data, which can be inter-links among
different Web documents (Kleinberg 1998) or intra-
links within individual Web document (Arasu & Hector,
2003; Lerman et al., 2004). Web usage mining is defined
as to discover interesting usage patterns from the sec-
ondary data derived from the interaction of users while
surfing the Web (Srivastava et al., 2000; Cooley, 2003).
Recently, XML is widely used as a standard for data
exchanging in the Internet. Existing work on XML data
mining includes frequent substructure mining (Inokuchi
et al., 2000; Kuramochi & Karypis, 2001; Zaki, 2002,
Yan & Han, 2003; Huan et al., 2003), classification
(Zaki & Aggarwal, 2003; Huan et al., 2004), and associa-
tion rule mining (Braga et al., 2002). As data in different
domains can be represented as XML documents, XML
data mining can be useful in many applications such as
bioinformatics, chemistry, network analysis (Deshpande
et al., 2003; Huan et al., 2004) and etcetera.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
The historical XML mining research is largely inspired
by two research communities: XML data mining and
XML data change detection. The XML data mining
community has looked at developing novel algorithms
to mine snapshots of XML data. The database commu-
nity has focused on detecting, representing, and query-
ing changes to XML data.
Some of the initial work for XML data mining is
based on the use of the XPath language as the main
component to query XML documents (Braga et al.,
2002; Braga et al., 2003). In Braga et al. (2002, 2003),
the authors presented the XMINE operator, which is a
tool developed to extract XML association rules for
XML documents. The operator is based on XPath and
inspired by the syntax of XQuery. It allows us to express
complex mining tasks, compactly and intuitively. XMINE
can be used to specify indifferently (and simultaneously)
mining tasks both on the content and on the structure of
the data, since the distinction in XML is slight.
Other works for XML data mining focus on extract-
ing the frequent tree patterns from the structure of XML
data such as TreeFinder (Termier et al., 2002) and
TreeMiner (Zaki, 2002). TreeFinder uses an Inductive
Logic Programming approach. Notice that TreeFinder
cannot produce complete results. It may miss many
frequent subtrees, especially when the support thresh-
old is small or trees in the database have common node
labels. TreeMiner can produce the complete results by
using a novel vertical representation for fast subtree
support counting.
Different from the above techniques, which focus on
designing ad-hoc algorithms to extract structures that
occur frequently in the snapshot data collections, his-
torical XML mining focus on the sequence of changes
among XML versions.
Considering the dynamic nature of XML data, many
efforts have been directed into the research of change
detection for XML data. XML TreeDiff (Curbera &
Epstein, 1999) computes the difference between two
XML documents using hash values and simple tree
TEAM LinG
Mining Historical XML
comparison algorithm. XyDiff (Cobena et al., 2002) is
proposed to detect changes of ordered XML docu-
ments. Besides insertion, deletion, and updating,
XyDiff also support move operation. X-Diff (Wang et
al., 2003) is designed to detect changes of unordered
XML documents. In our historical XML mining, we
extend the XML change detection techniques to dis-
cover hidden knowledge from the history of changes to
XML data with data mining techniques.
MAIN THRUST
Overview
Consider the dynamic property of XML data and exist-
ing XML data mining research; it can be observed that
the dynamic nature of XML leads to two challenging
problems in XML data mining. First is the maintenance
of the mining results for existing mining techniques. As
the data source changes, new knowledge may be found
and some old ones may not be valid. Second, is the
discovery of novel knowledge hidden behind the histori-
cal changes, some of them are difficult or impossible to
be discovered from snapshot data. In this paper, we
focus on the second issue. That is, to discover novel
hidden knowledge from the historical changes to XML
data. Suppose there is a sequence of XML documents,
which are different versions of the same documents.
Then, following novel knowledge can be discovered.
Note that, by no means we claim that the list is exhaus-
tive. We use them as representatives for the various
types of knowledge behind the history of changes.
Frequently changing/Frozen structures/con-
tents: Some parts of the structure or content
change more frequently and significantly com-
pared to other structures. Such structures and con-
tents reflect the relatively more dynamic parts of
the XML document. Frozen structures/contents
represent the most stable part of the XML docu-
ment. Identifying such structure is useful for vari-
ous applications such as trend monitoring and
change detection of very large XML documents.
Association rules: Some structures/contents are
associated in terms of their changes. The associa-
tion rules imply the concurrence of changes among
different parts of the XML document. Such knowl-
edge can be used for XML change prediction,
XML index maintenance, and XML based multi-
media annotation.
Change patterns: From the historical changes,
one may observe that more and more nodes are
inserted under certain substructures, while nodes
are inserted and deleted frequently under others.
Such change patterns can be critical for monitoring M
and predicting trends in e-commerce Web sites.
They may imply certain underlying semantic mean-
ings; and can be exploited by strategy makers.
Applications
Such novel knowledge can be useful in different applica-
tions, such as intelligent change detection for very large
XML documents, Web usage mining, dynamic XML
indexing, association rule mining, evolutionary pattern
based classification, and etcetera. We only elaborate on
the first two applications due to the limitation of space.
Suppose one can discover substructures that change
frequently and those that do not (frozen structures),
then he/she can use this knowledge to detect changes to
relevant portions of the documents at different fre-
quency based on their change patterns. That is, one can
detect changes to frequently changing content and struc-
ture at a different frequency compared to structures that
do not change frequently. Moreover, one may ignore
frozen substructures during change detection, as most
likely they are not going to change. As one of the major
limitations of existing XML change detection systems
(Cobena et al., 2002; Wang et al., 2003) is that they are
not scalable for very large XML documents. Knowledge
extracted from historical changes can be used to im-
prove the scalability of XML change detection systems.
Recently, a lot of work has been done in Web usage
mining. However, most of the existing works focus on
snapshot Web usage data, while usage data is dynamic in
real life. Knowledge hidden behind historical changes
of Web usage data, which reflects how Web access
patterns (WAP) change, is critical to adaptive Web,
Web site maintenance, business intelligence, and etcet-
era. The Web usage data can be considered as a set of
trees, which have the similar structures as XML docu-
ments. By partitioning Web usage data according to the
user-defined calendar pattern, we can obtain a sequence
of changes from the historical Web access patterns.
From the changes, useful knowledge, such as how cer-
tain Web access patterns changed, which parts changes
more frequently and which parts do not, can be ex-
tracted. Some preliminary results of mining the changes
to historical Web access patterns have been shown in
Zhao and Bhowmick (2004).
Research Issues
To the best of our knowledge, existing state-of-the-art
XML (structure related) data mining techniques (Yan &
Han, 2002; Yan & Han, 2003) cannot extract such novel
knowledge. Even if we apply such techniques repeatedly
801
TEAM LinG

to a sequence of snapshots of XML structure data, they
cannot discover such knowledge efficiently and com-
pletely. It is because that the historical XML mining
research focuses on the structural changes between ver-
sions of XML documents, which is generated by XML
data change detection tools (Zhao & Bhowmick, 2004).
Given a sequence of XML documents (which are
versions of the same XML document), the objective of
historical XML data mining is to extract hidden and
useful knowledge from the sequence of changes to the
XML versions. In this article, we focus on the structural
changes of XML documents. We proposed three major
issues for historical XML mining. They are identifying
interesting structures, mining XML delta association
rules, and classification/clustering evolutionary XML
data. Based on the historical change behaviors of the
XML data, the first issue is to discover the interesting
structures. We elaborate on some of the representatives.
Frozen/Frequently changing structure: Frozen
structure refers to structure that does not change
frequently and significantly in the history. There
are different possible reasons that a structure does
not change in the sequence of XML document
versions. First, the structure is so well designed
that it does not change even if the content has
changed. Second, some parts of the XML docu-
ment may be ignored or redundant so that they
never change. Also, some data in the XML docu-
ment may never change by nature.
Frequently changing structure: Refers to sub-
structures in the XML document may change more
frequently and significantly compared to others.
They may reflect the relatively more dynamic parts
of the XML document.
Periodic dynamic structure: Among the history
of changes for some structures, there may be cer-
tain fixed patterns. Such change patterns may occur
repeatedly in the history. Those structures, where
there exist certain patterns in their change histo-
ries, are called periodic dynamic structure.
Increasing/Decreasing dynamic structure:
Among those frequently changing structures, some
of them change according to certain patterns. For
instance, the changes of some structures may be-
come more significant and more frequent. Such
structures will be defined as increasing dynamic
structures. Similarly, decreasing dynamic struc-
tures denote for structures whose frequency and
significance of changes are decreasing.
Outlier structures: From the historical change
behavior of the structures, one may observe that
some of them may not comply with their change
802
Mining Historical XML
patterns as predicted based on the history. For
instance, some of the frozen structures that are
not supposed to change may change while some
of the frequently changing structures that are
supposed to change may not change. Such changes
may happen for various reasons. For example,
some of the structures may be modified by mis-
takes; some of them may be the result of some
intrusion or fraud actions; and others may be
caused by intentional or highly occurrence of
underling events.
Besides the interesting structures, association rules
between structures can be extracted as well. There are
two types of association rules. One is structural delta
association rule and another is semantic delta associa-
tion rule.
Structural delta association rule: Among those
interesting structures, we may observe that some
of the structures change together frequently with
certain confidence. For example, whenever struc-
ture A changes, structure B also changes with a
probability of 80%. Structural delta association
rule is used to represent such kind of knowledge.
It can be used for different applications such as
version and concurrency control systems.
Semantic delta association rule: By incorpo-
rating some metadata, such as types of changes,
ontology, and content summaries of leaf nodes,
semantic delta association rule can be extracted.
For example, with the history of changes in an e-
commerce Web site, we may discover that when
product A becomes more popular, product B will
become less popular. Such semantic delta asso-
ciation rules can be useful for competitor moni-
toring and strategy analysis in e-commerce.
FUTURE TRENDS
Considering the different types of mining we proposed
and the types of data we are going to mine, there are
many challenges ahead. We elaborate on some of the
representatives here.
Real Data Collection
To collect real data of historical XML structural ver-
sions is a challenge. To get the structural data from
XML documents, a parser is needed. Currently, there
are many XML parsers available. However, it has been
verified that the parsing process is the most expensive
TEAM LinG
Mining Historical XML
process of XML data management (Nicola & John, 2003).
Moreover, to get the historical structural information,
every time the XML document changes the entire docu-
ment has to be parsed again. Consequently, to extract
historical structural information is a very expensive task.
Especially for very large XML documents, usually only
some parts of the document change frequently. In order
to get knowledge that is more useful, a longer sequence
of historical structural data and larger datasets are desir-
able. However, to get a longer and larger historical XML
structural datasets, a longer time is needed to collect the
corresponding XML documents.
Frequency of Data Gathering
With the dynamic nature of XML data, the process of
gathering all versions of the historical XML structural
data becomes a challenging task. The most naive method
is to keep checking corresponding XML documents
continuously, but this approach is very expensive and
may overwhelm the network bandwidth. An alternative
method is to determine the frequency of checking by
analyzing the historical changes of similar XML docu-
ments. However, this method does not guarantee that all
versions of the historical data can be detected. In our
research, we propose to extract appropriate frequency for
data gathering based on the changes of the historical data.
Incremental Mining
The high cost of some data mining processes and the
dynamic nature of XML data make it desirable to mine
the data incrementally based on the part of data that
changed rather than mining the entire data again from
scratch. Such algorithms are based on the previous
mining results. Most recently, there is a survey of
incremental mining of sequential patterns in large data-
base (Parthasarathy et al., 1999). Integrated with a change
detection system, our incremental mining of historical
XML structural data can keep the discovered knowledge
up to date and valid. However, the challenge is that
existing incremental mining techniques are for rela-
tional and transactional data, while XML structural is
semistructure. How to modify those approaches so that
they can be used for incremental mining historical XML
will be one of our research issues.
CONCLUSION
In this article, we present a novel research direction:
mining historical XML documents. Different from ex-
isting research of XML data mining and XML change
detection, the temporal and dynamic property of XML
data is incorporated with the semistructure property in M
historical XML mining. Examples shown that the re-
sults of historical XML mining can be useful in a variety
of applications, such as intelligent change detection for
very large XML documents, Web usage mining, dy-
namic XML indexing, association rule mining, evolu-
tionary pattern based classification, and etcetera. By
exploring the characteristics of XML changes, we present
a framework of historical XML mining with a list of
major research issues and challenges.
REFERENCES
Arasu, A., & Hector, G.-M. (2003). Extracting struc-
tured data from Web pages. In Proceedings of ACM
SIGMOD (pp. 337-348).
Braga, D., Campi, A., Ceri, S., Klemettinen, M., & Lanzi,
P.L. (2002). A tool for extracting XML association
rules. In Proceedings of IEEE ICTAI (pp. 57-65).
Braga, D., Campi, A., Ceri, S., Klemettinen, M., & Lanzi,
P.L. (2003). Discovering interesting information in
XML data with association rules. In Proceeding of ACM
SAC (pp. 450-454).
Cobena, G., Abiteboul, S., & Marian, A. (2002). Detect-
ing changes in XML documents. In Proceedings of
IEEE ICDE (pp. 41-52).
Cooley, R. (2003). The use of Web structure and con-
tent to identify subjectively interesting Web usage pat-
terns. In ACM Transactions on Internet Technology
(pp. 93-116).
Curbera, & Epstein, D.A. (1999). Fast dierence and
update of XML documents. In Proceeding of XTech.
Deshpande, M., Kuramochi, M., & Karypis, G. (2003).
Frequent sub-structure-based approaches for classify-
ing chemical compounds. In Proceedings of IEEE ICDM
(pp. 35-42).
Dumais, S.T., & Chen, H. (2000). Hierarchical
classication of Web content. In Proceedings of An-
nual International ACM SIGIR (pp. 256-263).
Ester, M., Kriegel, H.-P., & Schubert, M. (2002). Web
site mining: A new way to spot competitors, customers
and suppliers in the World Wide Web. In Proceedings
of the eighth ACM SIGKDD (pp. 249-258).
Etzioni, O. (1996). The World-Wide Web: Quagmire or
gold mine? Communications of the ACM, 39(11), 65-68.
803
TEAM LinG

Han, J., & Kamber, M. (2000). Data mining concepts and
techniques. Morgan Kaufmann.
Huan, J., Wang, W., & Prins, J. (2003). Efficient mining
of frequent subgraph in the presence of isomorphism. In
Proceedings of IEEE ICDM (pp. 549-552).
Huan, J., Wang, W., Washington, A., Prins, J., Shah, R.,
& Tropshas, A. (2004). Accurate classification of protein
structural families using coherent subgraph analysis. In
Proceedings of PSB (pp. 411-422).
Kleinberg, J.M. (1998). Authoritative sources in a
hyperlinked environment. Journal of the ACM, 46(5),
604-632.
Lerman, K., Getoor, L., Minton, S., & Knoblock, C.
(2004). Using the structure of Web sites for automatic
segmentation of tables. In Proceedings of ACM SIGMOD
(pp. 119-130).
Madria, S.K., Bhowmick, S.S., Ng, W.K., & Lim, E.-P.
(1999). Research issues in Web data mining. In DMKD
(pp. 303-312).
Matthias, N., & Jasmi, J. (2003). XML parsing: A threat to
database performance. In Proceedings of CIKM (pp. 175-
178).
McHugh, J., Abiteboul, S., Goldman, R., Quass, D., &
Widom, J. (1997). Lore: A database management sys-
tem for semistructured data. ACM SIGMOD Record,
26(3), 54-66.
Reis, D. de C., Golgher, P.B, da Silva Altigran, S., &
Laender, A.H.F. (2004). Automatic Web news extraction
using tree edit distance. In Proceedings of WWW (pp. 502-
511).
Srinivasan, P., Zaki, M.J., Ogihara, M. & Dwarkadas, S..
(1999). Incremental and interactive sequence mining. In
Proceedings of CIKM (pp. 251-258).
Shearer, K., Dorai, C., & Venkatesh, S. (2000). Incorpo-
rating domain knowledge with video and voice data
analysis in news broadcasts. In Proceedings of ACM
SIGKDD (pp. 46-53).
Srivastava, J., Cooley, R., Deshpande, M., & Tan, P.-N.
(2000). Web usage mining: Discovery and applications
of usage patterns from Web data. ACM SIGKDD Explo-
rations, 1(2), 12-23.
804
Mining Historical XML
Wang, Y., DeWitt, D.J., & Cai, J.-Y. (2003). X-difi: An
effective change detection algorithm for XML documents.
In Proceedings of IEEE ICDE (pp. 519-530).
Yang, L.H., Lee, M.L., & Hsu, W. (2003). Efficient mining
of XML query patterns for caching. In Proceedings of
VLDB (pp. 6980).
Zaki, M.J. (2002). Efficiently mining frequent trees in a
forest. In Proceedings of ACM SIGKDD (pp. 71-80).
Zaki, M.J., & Aggarwal, C.C. (2003). XRules: An effective
structural classifier for XML data. In Proceedings of ACM
SIGKDD (pp. 316-325).
Zhao, Q., & Bhowmick, S.S. (2004). Mining history of
changes to Web access patterns. In Proceeding of PKDD.
KEY TERMS
Changes to XML: Given two XML document, the set
of edit operations that transform one document to another
is called changes to XML.
Historical XML: A sequence of XML documents,
which are different versions of the same XML document.
It records the change history of the XML document.
Mining Historical XML: It is the process of knowl-
edge discovery from the historical changes to versions
of XML documents. It is the integration of XML change
detection systems and XML data mining techniques.
Semi-Structured Data Mining: Semi-structured data
mining is a sub-field of data mining, where the data
collections are semi-structured, such as Web data, chemi-
cal data, biological data, network data, and etcetera.
Web Mining: Web data mining is to use data mining
techniques to automatically discovery and extract infor-
mation from Web data and services.
XML: Extensible Markup Language (XML) is a
simple, very flexible text format derived from SGML.
XML Structural Changes: Among the changes to
XML, not all the changes may cause the structure changes
of the XML when it is represented as a tree structure. In
this case, only insertion and deletion are called XML
structural changes.
TEAM LinG

Mining Images for Structure
Terry Caelli
Australian National University, Australia
INTRODUCTION
Most data warehousing and mining involves storing and
retrieving data either in numerical or symbolic form,
varying from tables of numbers to text. However, when
it comes to everyday images, sounds, and music, the
problem turns out to be far more complex. The major
problem with image data mining is not so much image
storage, per se, but rather how to automatically index,
extract, and retrieve image content (content-based re-
trieval [CBR]). Most current image data-mining tech-
nologies encode image content by means of image
feature statistics such as color histograms, edge, tex-
ture, or shape densities. Two well- known examples of
CBR are IBMs QBIC system used in the State Heritage
Museum and PICASSO (Corridoni, Del Bimbo & Pala,
1999) used for the retrieval of paintings. More recently,
there have been some developments in indexing and
retrieving images based on the semantics, particularly
in the context of multimedia, where, typically, there is
a need to index voice and video (semantic-based re-
trieval [SBR]). Recent examples include the study by
Lay and Guan (2004) on artistry-based retrieval of
artworks and that of Benitez and Chang (2002) on com-
bining semantic and perceptual information in multime-
dia retrieval for sporting events.
However, this type of concept or semantics-based image
indexing and retrieval requires new methods for encoding
and matching images, based on how content is structured,
and here we briefly review two approaches to this.
BACKGROUND
Generally speaking, image structure is defined in terms
of image features and their relations. For SBR, such
features and relations reference scene information.
These features typically are multi-scaled, varying from
pixel attributes derived from localized image windows
to edges, regions, and even larger image area properties.
MAIN THRUST
In recent years there has been an increasing interest in
SBR. However, this requires the development of meth-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
805
M
ods for binding image content with semantics. In turn,
this reduces to the need for models and algorithms that
are capable of efficiently encoding and matching rela-
tional properties of images and associating these rela-
tional properties with semantic descriptions of what is
being sensed. To illustrate this approach, we briefly
discuss two representative examples of such methods:
(1) Bayesian Networks (Bayesian Nets) for SBR, based
first on multi-scaled image and then on image feature
models; (2) principal components analysis (also termed
latent semantic indexing or spectral methods).
Bayesian Network Approaches
Bayesian Nets have recently proved to be a powerful
method for SBR, since semantics are defined in terms
of the dependencies between image features (nodes),
their labels, and known states of what is being sensed.
Inference is performed by propagation probabilities
through the network. For example, Benitez et al. (2003)
have developed MediaNet, a knowledge representation
network and inference model for the retrieval of con-
ceptually defined scene properties integrated with natu-
ral language processing. In a similar way, Hidden Markov
Random Fields (HMRFs) have become a common class
of image models for binding images with symbolic
descriptions. In particular, Hierarchical Hidden Markov
Random Fields (HHMRF) provide a powerful SBR rep-
resentation. HHMRFs are defined over multi-scaled
image pixel or features defined by Gaussian or Laplacian
pyramids (Bouman & Shapiro, 1994). Each feature, or
pixel, x , at a given scale is measured (observed) to
evidence scene properties, states, s, corresponding to
semantic entities such as ground, buildings, and so
forth, as schematically illustrated in Figure 1. The rela-
tionships between states serves to define the grammar.
The link between observations and states defines, in this
approach, the image semantics. Accordingly, at each
scale, l, we have a set of observations and states, where
p(ol (x) /sl (x)) defines the dependency of the observation
at scale, l, on the state of the world (scene).
Specifically, the HHMRF assumes that the state at
a pixel, x , is dependent on the states of its neighboring
pixels at the same or neighboring levels of the pyramid.
A simple example of the expressiveness of this model is
a forestry scene. This could be an image region (label:
TEAM LinG

Figure 1. The hierarchical hidden Markov random
field (HHMRF) model for image understanding. Here,
the hidden state variables, X, at each scale are
evidenced by observations, Y, at the same scale and
the state dependencies within and between levels of
the hierarchy. The HHMRF is defined over pixels and/
or feature graphs.
forest) dependent on a set of regions labeled tree at the
next scale, which, in turn, are dependent on trunk,
branches, and leaves labels at the next scale, and so
forth. These labels have specific positional relations
over scales, and the observations for each label are
supported by their compatibilities and observations.
Consequently, the SBR query for finding forestry
images is translated into finding the posterior maximum
likelihood (MAP) of labeling the image regions, given
the forestry model. Using Bayes rule, this reduces to
the following optimization problem:
s*l (x)) argmax{p(sl (x) /ol (x)) p(sl v (x u))}
S u,v
where l v corresponds to the states above and below
level l of the hierarchy. In other words, the derived
labeling MAP probability is equivalent to a probabilistic
answer to the query if this image is a forestry scene.
There are many approaches to approximate solutions to
this problem, including relaxation labeling, expectation
maximization (EM), loopy belief propagation and the
junction tree algorithm (see definitions in Terms and
Definitions). All of these methods are concerned with
optimal propagation of evidence over different layers of
the graphical representation of the image, given the
model and the observations, and all have their limita-
tions. When the HHMRF model is approximated by a
triangulated image state model, the junction tree algo-
rithm is optimal. However, triangulating such hierarchi-
cal meshes is computationally expensive. On the other
hand, the other approaches mentioned previously are
not optimal, converging to local minima (Caetano &
Caelli, 2004).
When the structural information in the query and
image is defined in terms of features (i.e., regions or
edge segments) and their relational attributes, again,
HMRFs can be applied to image feature matching. In this
806
Mining Images for Structure
case, the HMRF is defined over graphs that depict fea-
tures and their relations. That is, consider two attributed
graphs, Gs and G x , representing the image and the query,
respectively. We want to determine just how, if at all, the
query (graph) structure is embedded somewhere in the
image (graph). We define HMRF over the query graph,
G x . A single node in G x is defined by xi , and in the graph
Gs , by s . Each node in each graph has vertex and edge
attributes, and the query corresponds to solving a sub-
graph isomorphism problem that involves the assignment
of each xi a unique s , assuming that there is only one
instance of the query structure embedded in the image,
although this can be generalized. In this formulation, the
HMRF model considers each node xi in G x as a random
variable that can assume any of S possible values corre-
sponding to the nodes of Gs .
The Observation Component: Using HMRF for-
malities, the similarity (distance: dist) between
vertex attributes of both graphs is consequently
defined as the observation matrix model
Bi = p ( y xi / xi = s ) = dist ( y xi , y s ) .
The Markov Component: Here, we use the bi-
nary (relational) attributes to construct the com-
patibility functions between states of neighboring
nodes. Assume that xi and x j are neighbors in the
HMRF (being connected in G x ). Similar to the
previously described unary attributes, we have
A ji; = p( x j = s / xi = s ) = dist ( y xij , y s ) .
Optimization Problem and Solutions: Given
this general HMRF formulation for graph match-
ing, the optimal solution reduces to that of deriv-
ing a state vector s * = ( s 1 ,.., s T ) where s i G s for
each vertex x i G x such that the MAP criterion is
satisfied, given the model = ( A, B) and data
s * = arg max{ p ( x1 = s ,...., xT = s / )} .
s .. s
Specifically, we introduce two sets of HMRF mod-
els for solving this problem.
TEAM LinG
Mining Images for Structure

First, probabilistic relaxation labeling (PRL) is a parallel adjacency matrix where relations are defined in terms of
iterative update method for deriving the most consistent the matrix off-diagonal elements. These matrices are de- M
labels, having the form composed into their eigenvalues and eigenvectors, as

p t 1 (d i ) p( yi / d i ) p ( d i / d i ) p t (d i ) G = P P ' , G = T
1,..i 1,i 1,..,n i i

where T corresponds to the covariance matrix of

where pt (d i ) is typically factored into pair-wise condi- the vertex and edge attributes. In the case where only
tionals. This algorithm is known to converge to local adjacencies (1,0) are used, then G is simply the adja-
minima and is particularly sensitive to initial values. Al- cency matrix. The core idea is that vertices are then
beit, this is an example of an inexact (suboptimal) model projected into a selected eigen-subspace, and corre-
applied to the complete data model. The following models spondences are determined by proximities in such
are the oppositeexact inference models over approxi- spaces. The method, in different variations, has been
mate data modelsand so their optimality is critically explored by Scott and Longuet-Higgins (1991), Shapiro
dependent on how well the data model approximation is. and Brady (1992), and Caelli and Kosinov (2004).
Single Path Dynamic Programming (SPDP) con- Normalization is required when graphs are not of the
sists of approximating the fully connected graph into a same size. Sometimes, only the eigenspectra (i.e., list
single chain, which traverses each vertex exactly once. of eigenvalues) are used to compare graphs (Siddiqi et
This model allows us to use dynamic programming in al., 1999; Luo & Hancock, 2001).
order to find the optimal match between the graphs. In the Such spectral methods offer insightful ways to
second type of model, we improve over this simple chain define image structures, which often can correspond to
model by proposing HMRFs with maximal cliques of semantically corresponding properties between the
size 3 and 4, which still retain the model feasibility for query and the image. For example, comparing two
accomplishing optimal inference via junction tree (JT) different views of the same animal often can result in
methods (Lauritzen, 1996). These cliques encode more the predicted correspondences having semantically
information than the single path models insofar as verti- correct labels, as shown in Figure 2 and Table 1, even
ces are connected to 2 (JT3) and 3 (JT4) other neighbor- when only basic adjacencies (simply whether two fea-
ing vertices, as compared to only 1 (the previous) in the tures are connected or not, without attributes such as
SPDP case. Such larger cliques, their associated vertex distances, angles, etc.) are used to encode the rela-
labels, then are used to derive the optimal labels for a tional structures.
given vertex. The JT algorithm is similar to dynamic
programming involving a forward and backward pass for
determining the most likely labels, given the evidence Figure 2. Two sample shapesboxer-10 and boxer-
and neighborhood consistencies (Caetano & Caelli, 2004). 24and their shock graph representations (Siddiqi
Another recent approach to encoding and retrieving et al., 1999)
image structures in ways that can index their semantics is
the method of Latent Semantic Indexing (LSI), as used in
many other areas of data mining (Berry et al., 1995).
Here, the basic idea is to use Principle Components
Analysis (PCA) to determine the basic dimensionality of
a relational data structure (i.e., items by attribute ma-
trix), project, and cluster items, and to retrieve examples
via proximity in the projection subspaces. For SBR and
for images, what is needed is a form of LSI that functions
on the relational attributes of image features, and re-
cently, the LSI method has been extended to incorporate
such structures. Here, relationships between features
are encoded as a graph, and the LSI approach offers a
different method for solving graph-matching problems.
In this way, SBR consists of matching the query (a graph)
with observed image data encoded also as a graph.
The LSI approach, when applied to relational fea-
tures, uses standard matrix models for graphs, such as the

807

TEAM LinG

Table 1. Cluster decomposition obtained from
correspondence clustering of the two shock graphs
for shapes BOXER-10 and BOXER-24. Each cluster
enumerates the vertices from the two graphs that are
grouped together and lists in brackets the part of the
body from which the vertex approximately comes. The
percentage of variance explained by the first three
dimensions for shapes Boxer-10 and Boxer-24 was
32% and 33%, respectively (Caelli & Kosinov, 2004).
FUTURE TRENDS
As can be seen from the previous discussion, SBR re-
quires the development of new methods for relational
queries, ideally to pose such querying as an optimization
problem that results in a degree of match that, in some
sense, is derived optimally rather than by using heuristic
methods. Here, we have discussed briefly two such meth-
ods: one based on Bayesian methods and the other based
on matrix methods. What is clearly needed is an integra-
tion of both approaches in order to capitalize on the
benefits of both views. Recent developments in both
deterministic and stochastic kernel methods (Schlkopf
& Smola, 2002) may provide the basis for such an integra-
tion.
CONCLUSION
Image data mining is still an emerging area where ulti-
mate success and use can only come about if we can
solve the difficult problem of querying images and
searching for content as humans do. This must involve
the creation of new SBR algorithms that clearly are based
on the ability to support relational queries in an optimal
808
Mining Images for Structure
and robust way. Machine learning, Bayesian networks,
and even new uses of matrix algebra all open up new
possibilities. Unless computers can store, evaluate and
interpret images the way we do, image data warehousing
and mining will remain a doubtful and certainly labor-
intensive technology with limited use without significant
human intervention.
REFERENCES
Benitez, A., & Chang, S. (2002). Multimedia knowledge
interrelations: Summarization and evaluation. Proceed-
ings of the MDM/KDD-2002, Edmonton, Alberta,
Canada.
Berry, M., Dumais, S., & OBien, G. (1994). SIAM Review,
37(4), 573-595.
Bouman, C., & Shapiro, M. (1994). A multiscale random
field model for Bayesian image segmentation. IEEE:
Transactions On Image Processing, 3(2), 162-177.
Caelli, T., Cheng, L., & Feng, Q. (2003). A Bayesian
approach to image understanding: From images to vir-
tual forests. Proceedings of the 16 th International
Conference on Vision Interface (pp. 1-12), Halifax, Nova
Scotia, Canada.
Caelli, T., & Kosinov, S. (2004). Inexact graph matching
using eigenspace projection clustering. International
Journal of Pattern Recognition and Artificial Intelli-
gence, 18(3), 329-354.
Caetano, T., & Caelli, T. (2004). Graphical models for
graph matching. Proceedings of the IEEE International
Conference on Computer Vision and Pattern Recogni-
tion: CVPR (pp. 466-473).
Corridoni, J., Del Bimbo, A., & Pala, P. (1999). Retrieval of
paintings using effects induced by colour features. IEEE
Multimedia, 6(3), 38-53.
Lauritzen, S. (1996). Graphical models. Oxford: Clarendon
Press.
Lay, J., & Guan, L. (2004). Retrieval for color artistry
concepts. IEEE Transactions on Image Processing,
13(3), 326-339.
Luo, B., & Hancock, E. (2001). Structural graph match-
ing using the EM algorithm and singular value decompo-
sition. IEEE Transaction of Pattern Analysis and Ma-
chine Intelligence, 23(10), 1120-1136.
Schlkopf, B., & Smola, A. (2002). Learning with
kernels. Cambridge, MA: MIT Press.
TEAM LinG
Mining Images for Structure
Scott, G., & Longuet-Higgins, H. (1991). An algorithm for
associating the features of two patterns. Proceedings of
the Royal Society of London.
Shapiro, L., & Brady, J. (1992). Feature-based correspon-
denceAn eigenvector approach. Image and Vision
Computing, 10, 268-281.
Siddiqi, K., Shokoufandeh, P., Dickinson, S., & Zucker, S.
(1999). Shock graphs and shape matching. International
Journal of Computer Vision, 30, 1-24.
KEY TERMS
Attributed Graph: Graphs whose vertices and edges
have attributes typically defined by vectors and matri-
ces, respectively.
Bayesian Networks: A graphical model defining
the dependencies between random variables.
Dynamic Programming: A method for deriving
the optimal path through a mesh. For hidden Markov
models (HMMs), it is also termed the Viterbi algorithm
and involves a method for deriving the optimal state
sequence, given a model and an observation sequence.
Expectation Maximization: Process of updating
model parameters from new data where the new param-
eter values constitute maximum posterior probability
estimates. Typically used for mixture models.
Grammars: A set of formal rules that define how to
perform inference over a dictionary of terms.
Graph: A set of vertices (nodes) connected in various
ways by edges.
Graph Spectra: The plot of the eigenvalues of the
graph adjacency matrix.
Hidden Markov Random Field (HMRF): A Markov
Random Field with additional observation variables at M
each node, whose values are dependent on the node
states. Additional pyramids of MRFs defined over the
HMRF give rise to hierarchical HMRFs (HHMRFs).
Image Features: Discrete properties of images that
can be local or global. Examples of local features include
edges, contours, textures, and regions. Examples of glo-
bal features include color histograms and Fourier compo-
nents.
Image Understanding: The process of interpreting
images in terms of what is being sensed.
Junction Tree Algorithm: A two-pass method for
updating probabilities in Bayesian networks. For trian-
gulated networks the inference procedure is optimal.
Loopy Belief Propagation: A parallel method for
update beliefs or probabilities of states of random vari-
ables in a Bayesian network. It is a second-order exten-
sion of probabilistic relaxation labeling.
Markov Random Field (MRF): A set of random
variables defined over a graph, where dependencies
between variables (nodes) are defined by local cliques.
Photogrammetry: The science or art of obtaining
reliable measurements or information from images.
Relaxation Labeling: A parallel method for updat-
ing beliefs or probabilities of states of random variables
in a Bayesian network. Node probabilities are updated in
terms of their consistencies with neighboring nodes and
the current evidence.
Shape-From-X: The process of inferring surface
depth information from image features such as stereo,
motion, shading, and perspective.
809
TEAM LinG
810
Mining Microarray Data
Nanxiang Ge
Aventis, USA
Li Liu
Aventis, USA
INTRODUCTION
During the last 10 years and in particularly within the
last few years, there has been a data explosion associ-
ated with the completion of the human genome project
(HGP) (IHGMC and Venter et al., 2001) in 2001 and the
many sophisticated genomics technologies. The human
genome (and genome from other species) now provides
an enormous amount of data waiting to be transformed
into useful information and scientific knowledge. The
availability of genome sequence data also sparks the
development of many new technology platforms. Among
the available different technology platforms, microarray
is one of the technologies that is becoming more and
more mature and has been widely used as a tool for
scientific discovery. The major application of microarray
is for simultaneously measuring the expression level of
thousands of genes in the cell. It has been widely used in
drug discovery and starts to impact the drug develop-
ment process.
The mining microarray database is one of the many
challenges facing the field of bioinformatics, computa-
tional biology and biostatistics. We review the issues in
microarray data mining.
BACKGROUND
The quantity of data generated from a microarray ex-
periment is usually large. Besides the actual gene ex-
pression measurement, there are many other data avail-
able such as the corresponding sequence information,
genes chromosome location information, gene ontol-
ogy classification and gene pathway information. Man-
agement of such data is a very challenging issue for
bioinformatics. The following is a list of requirements
for data management.
Data Organization: different microarray platforms
generate data with different formats, different gene
identifiers, etc. Sequence data, gene ontology data,
and gene pathway information all with diverse for-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
mats. Sensible organization of such diverse data
types will ease the data mining process.
Data Standards: the microarray data analysis com-
munity has developed a standard for microarray
data: the Minimum Information About a Microarray
Experiment (MIAME: http://www.mged.org). The
MIAME standard is needed to enable the consis-
tent interpretation of experiment results and po-
tentially to reproduce the experiment.
MAIN THRUST
Mining microarray data is a very challenging task. The
raw data from microarray experiments usually comes in
image format. Images are then quantified using image
analysis software. Such quantified data are then subject
to three steps of analysis:
Pre-processing microarray data
Mining microarray data.
Joint mining of microarray data and sequence
database.
We review the data analysis methods in these three
aspects. We focus our discussion on Affymetrix (http:/
/www.affymetrix.com) technology, but many of the
methods are applicable to data from other platforms.
Pre-Processing Microarray Data
While effectively managing microarray and related data
is an important first step, microarray data have to be pre-
processed so that downstream analysis can be performed.
Array-to-array and sample-to-sample variations are the
main reason for the requirement of pre-processing.
Typically, pre-processing involves the following four
steps:
Image Analysis: image analysis in microarray
experiment involves gridding of image, signal ex-
traction and back ground adjustment. Affymetrixs
TEAM LinG
Mining Microarray Data
microarray analysis suite (MAS) provides the quan-
tification software.
Data Normalization: normalization is a step neces-
sary for remove systematic array to array variations.
Different normalization methods have been pro-
posed. Cyclic loess method (Dudoit et al., 2002),
quantile normalization method and contrast-based
method normalize the probe level data; Scaling
method and non-linear method normalize expres-
sion intensity level data. Bolstad, Irizarry, Astrand,
and Speed (2003) provides a comparison of all these
methods and suggests that simple quantile normal-
ization performs relatively stable.
Estimation of Expression Intensities: different
methods to summarize expression intensity from
probe level data have appeared in literature. Among
them, Li and Wong (2001) proposed a model
based expression intensity (MEBI), Irizarrys
(2003) robust multi-array (RMA) methods and the
method provided in Affymetrix MAS5 software.
Data Transformation: data transformation is a
very important step and it enables the data to fit to
many of the assumptions behind statistical meth-
ods. Log transformation and glog (Durbin, Hardin,
Hawkins, & Rocke, 2002) transformation are the
two commonly used methods.
Mining Microarray Data
In principal, the current data mining activities in
microarray data can be grouped into two types of stud-
ies: unsupervised and supervised. Unsupervised analy-
sis has been used widely for mining microarray experi-
ment. Cluster analysis has been the dominant method
for unsupervised mining.
Examples of unsupervised data mining:
Eisen, Spellman, Brown, and Botstein (1998) stud-
ied the gene expression of budding yeast Saccha-
romyces cerevisiae spotted on cDNA microarrays
during the diauxic shift, the mitotic cell division
cycle, sporulation, and temperature and reducing
shocks. Hierarchical clustering was applied to this
gene expression data, and the result was repre-
sented by a tree whose branch lengths reflected
the degree of similarity between genes, which was
assessed by a pair-wise similarity function. The
computed tree was then used to order genes in the
original data table, and genes with similar expres-
sion pattern were grouped together. The ordered
gene expression table can be displayed graphically
in a colored image, where cells with log ratios of
0s were colored black, cells with positive log
ratios were colored red, cells with negative log
ratios were colored green, and the intensities of M
reds or greens were proportional to the absolute
values of the log ratios. The clustering analysis
efficiently grouped genes with similar functions
together, and the colored image provided an over-
all pattern in the data. Clustering analysis can also
help us understand the novel genes if they were co-
expressed with genes with known functions.
Standard clustering analysis, such as hierarchical
clustering, k-means clustering, self-organizing
maps, are very useful in mining the microarray
data. However, these data tables are often cor-
rupted with extreme values (outliers), missing
values, and non-normal distributions that preclude
standard analysis. Liu, Hawkins, Ghosh, and Young
(2003) proposed a robust analysis method, called
rSVD (Robust Singular Value Decomposition), to
address these problems. The method applies a
combination of mathematical and statistical meth-
ods to progressively take the data set apart so that
different aspects can be examined for both general
patterns and for very specific effects. The benefits
of this robust analysis will be both the understand-
ing of large-scale shifts in gene effects and the
isolation of particular sample-by-gene effects that
might be either unusual interactions or the result
of experimental flaws. The method requires a
single pass, and does not resort to complex clean-
ing or imputation of the data table before analy-
sis. The method rSVD was applied to a micro array
data, revealed different aspects of the data, and
gave some interesting findings.
Examples for supervised data mining:
Golub et al. (1999) studied the gene expression of
two types of acute leukemias, acute myeloid leu-
kemia (AML) and acute lymphoblastic leukemia
(ALL), and demonstrated the feasibility of cancer
prediction based on gene expression data. The data
comes from Affymetrix arrays with 6817 genes,
and consists of 47 cases of ALL and 25 cases of
AML. 38 samples (27 ALL, 11 AML) were used as
training data. A set of 50 genes with the highest
correlations with an idealized expression pat-
tern vector, where the expression level is uni-
formly high for AML and uniformly low for ALL,
were selected. The prediction of a new sample was
based on weighted votes of these 50 genes. The
method made strong prediction for 29 of the 34
test samples, and the accuracy was 100%. Golubs
method is actually a minor variant of the maximum
likelihood linear discriminate analysis for two
811
TEAM LinG

classes. Instead of using variances in computing
weights, Golubs method uses standard deviations.
Dudoit el al. (2002) compared the prediction perfor-
mance of a variety of classification methods, such as
linear/quadratic discriminate analysis, nearest neigh-
bor method, classification trees, aggregated classi-
fiers, bagging and boosting, on three microarray
datasets, lymphoma data (Alizadeh et al.), leukemia
data (Golub et al.) and NCI60 data (Ross et al.).
Based on their comparisons, the rankings of the
classifiers were similar across datasets and the main
conclusion, for the three datasets, is that simple
classifiers such as diagonal linear discriminate analy-
sis and nearest neighbors perform remarkably well
compared to more sophisticated methods such as
aggregated classification trees.
Dimension reduction techniques, such as principal
component analysis (PCA) and partial least squares
(PLS) can be used to reduce the dimension of the
microarray data before certain classifier is used.
For example, Nguyen et al. (2002) proposed an
analysis procedure to predict tumor samples. The
procedure first reduces the dimension of the
microarray data using PLS, and then applies logis-
tic discrimination (LD) or quadratic discriminate
analysis (QDA). See also West et al. (2001) and
Huang et al. (2003).
Joint Data Mining of Microarray, Gene
Ontology and TFBS Data
Combining microarray data and other available data such
as DNA sequence data, gene ontology data has attracted
lots of attention in recent years. From a system biology
view, combining different data types provides added
power than looking microarray data alone. This is still a
very active research area for data mining and we review
some of the current work done in two aspects: combining
with gene ontology database and combining with (TFBS)
database.
Combining microarray data with gene ontology
data is important in interpreting the microarray
data. Expression Analysis Systematic Explorer
(EASE, http://david.niaid.nih.gov/david/ease.htm) is
developed by Hosack, Dennis, Sherman, Lane, and
Lempicki (2003), and is a customizable, standalone
software application that facilitates the biological
interpretation of gene lists derived from the results
of microarray, proteomics, and SAGE experiments.
EASE can generate the annotations of a list of
genes in one shot, automatically link to online
analysis tools, and provide statistical methods for
812
Mining Microarray Data
discovering enriched biological themes within gene
lists. Blalock et al. (2004) studied the gene expres-
sion of Alzheimers disease (AD), and found some
interesting over-represented gene categories
among the regulated genes using EASE. The find-
ings suggest a new model of AD pathogenesis in
which a genomically orchestrated up-regulation of
tumor suppressor-mediated differentiation and in-
volution processes induces the spread of pathol-
ogy along myelinated axons.
Transcriptional factor binding factors are essen-
tial elements in the gene regulatory networks.
Directly experimental identification of such tran-
scriptional factors is not practical or efficient in
many situations. Conlon, Liu, Lieb, and Liu (2003)
provided one method based on least regression to
integrating microarray data and transcriptional
factor binding sites (TFBS) patterns. Curran, Liu,
Long, and Ge (2004) provided a logistic regres-
sion approach to joint mining an internal
microarray database and a corresponding TFBS
database (http://transfac.gbf.de/).
FUTURE TRENDS
As more and more new technologies platform are in-
troduced for medical research, it is imaginable that
data will continue to grow. For example, Affymetrix
recently introduced its SNP chip which contains
100,000 human SNPs. If such a technology were ap-
plied to a clinical trial with 10000 subjects, the SNP
data alone will be a 10000 by 100000 table in addition
to data from potential other technologies such as
proteomics, metabonomics and bio-imaging.
Associated with the growth of data will be the in-
creasing need for effective data management and data
integration. More efficient data retrieval system will
be needed as well as system that can accommodate
large scale and diversified data.
Besides the growth of data management system, it is
foreseeable that integrated data analysis will become
more and more routine. At this point, most of the data
analysis software is only capable of analyzing small
scale, isolated data sets. So, the challenges to the
informatics field and statistics field will continue to
grow dramatically.
CONCLUSION
Microarray technology has generated vast amount of
data for very interesting data mining research. How-
TEAM LinG
Mining Microarray Data
ever, as the central dogma indicates, microarray only
provides one snapshot of the biological system. Se-
quencing, proteomics technology, metabolite profiling
provide different view about the biological system and it
will remain a challenge to deal with the joint data mining
of data from such diverse technology platform.
There are significant challenges with respect to devel-
opment of data mining technology to deal with data
generated from different technology platform. The chal-
lenges to deal with the combined data will be even more
difficult. Besides methodological challenges, how to or-
ganize the data, how to develop standards for data gen-
erated from different platforms and how to develop com-
mon terminologies, all remain to be answered.
REFERENCES
Alizadeh, A.A. et al. (2000). Different types of diffuse
large b-cell lymphoma identified by gene expression
profiling. Nature, 403(6769), 503-511.
Blalock, E.M., Geddes, J.W., Chen, K.C., Porter, N.M.,
Markesbery, W.R., & Landfield, P.W. (2004). Incipient
Alzheimers disease: Microarray correlation analyses
reveal major transcriptional and tumor suppressor re-
sponses. PNAS, 101(7), 2173-8.
Bolstad, B.M., Irizarry, R.A., Astrand, M., & Speed, T.P.
(2003). A comparison of normalization methods for
high density oligonucleotide array data based on bias
and variance. Bioinformatics, 19(2), 185-193.
Chu, S., DeRisi, J., Eisen, M., Mulholland, J., Botstein, D.,
Brown, P.O. et al. (1998). The transcriptional program of
sporulation in budding yeast. Science, 282(5389), 699-
705.
Conlon, E.M., Liu, X.S., Lieb, J.D., & Liu, J.S. (2003).
Integrating regulatory motif discovery and genome-
wide expression analysis. PNAS, 100(6), 3339-3344.
Curran, M., Liu, H., Long, F., & Ge, N. (2003). Statisti-
cal methods for joint data mining of gene expression and
DNA sequence database. SIGKDD Explorations Spe-
cial Issue on Microarray Data Mining, 5, 122-129.
Dudoit, S., Fridlyand, J., & Speed, T.P. (2002). Com-
parison of discrimination methods for the classifica-
tion of tumors using gene expression data. Journal of
the American Statistical Association, 97(457), 77-87.
Dudoit, S., Yang, Y.H., Callow, M.J., & Speed, T.P.
(2002). Statistical methods for identifying genes with
differential expression in replicated cDNA microarray
experiments. Statistical Sinica, 12(1), 111-139.
Durbin, B.P., Hardin, J.S., Hawkins, D.M., & Rocke, D.M.
(2002). A variance-stabilizing transformation of gene ex- M
pression microarray data. Bioinformatics, 18(Supplement
1), S105-S110.
Eisen, M.B., Spellman, P.T., Brown, P.O., & Botstein, D.
(1998). Cluster analysis and display of genome-wide ex-
pression patterns. PNAS, 95, 14863-14868.
Golub, T.R., Slonim, D.K., Tamayo, P., Huard, C.,
Gaasenbeek, M., Mesirov, J.P., et al. (1999). Molecular
classification of cancer: Class discovery and class pre-
diction by gene Expression monitoring. Science,
286(5439), 531-537.
Hosack, D.A., Dennis, G. Jr., Sherman, B.T., Lane, H.C., &
Lempicki, R.A. (2003). Identifying biological themes within
lists of genes with EASE. Genome Biology, 4(9), R60.
Huang, E., Cheng, S.H., Dressman, H., Pittman, J., Tsou,
M.H., Horng, C.F. et al. (2003). Gene expression predictors of
breast cancer outcomes. Lancet, 361, 1590-1596.
Irizarry, R.A., Hobbs, B., Collin, F., Beazer-Barclay, Y.D.,
Antonellis, K.J., Scherf, U. et al. (2003) Exploration, nor-
malization, and summaries of high density oligonucle-
otide array probe level data. Biostatistics, 4(2), 249-264.
Lee, H.K., Hsu, A.K., Sajdak, J., Qin, J., & Pavlidis, P. (2004).
Coexpression analysis of human genes across many microarray
data sets. Genome Res, 14(6), 1085-1094.
Li, C., & Wong, W.H. (2001). Model-based analysis of
oligonucleotide arrays: Expression index computation
and outlier detection. PNAS, 98(1), 31-36.
Liu, L., Hawkins, D.M., Ghosh, S., & Young, S.S. (2003).
Robust singular value decomposition analysis of
microarray data. PNAS, 100(23), 13167-13172.
The International Human Genome Mapping Consortium
(IHGMC). (2001). A physical map of the human ge-
nome. Nature, 409(6822), 934-941.
Ross, D.T., Scherf, U., Eisen, M.B., Perou, C.M., Rees,
C., Spellman, P., et al. (2000). Systematic variation in
gene expression patterns in human cancer cell lines.
Nature Genetics, 24, 227-234.
Venter, J. et al. (2001). The human genome. Science,
291(5507), 1304-51.
West, M., Blanchette, C., Dressman, H., Huang, E.,
Ishida, S., Spang, R., et al. (2001). Predicting the clini-
cal status of human breast cancer by using gene expres-
sion profiles. PNAS, 98, 11462-11467.
813
TEAM LinG

KEY TERMS
Bioinformatics: The analysis of biological informa-
tion using computers and statistical techniques; the sci-
ence of developing and utilizing computer databases and
algorithms to accelerate and enhance biological research
Gene Ontology (GO): GO is three structured, con-
trolled vocabularies that describe gene products in terms
of their associated biological processes, cellular com-
ponents and molecular functions in a species-indepen-
dent manner (http://www.geneontology.org ).
Metabonomics: The evaluation of tissues and biologi-
cal fluids for changes in metabolite levels that result from
toxicant-induced exposure.
814
Mining Microarray Data
MIAME: Minimum Information About a Microarray
Experiment. It is a standard for data from a microarray
experiment.
Normalization: A process to remove systematic varia-
tion in microarray experiment. Examples of such system-
atic variation includes between array variation, dye bias,
etc.
Proteomics: is the large-scale study of proteins, par-
ticularly their structures and functions.
SNP: Single Nucleotide Polymorphism. The same
sequence from two individuals can often be single base
pair changes. Such change can be useful genetic markers
and might explain why certain individual respond to
certain drug while others dont.
TEAM LinG

Mining Quantitative and Fuzzy
Association Rules
Hong Shen
Japan Advanced Institute of Science and Technology, Japan
Susumu Horiguchi
Tohoku University, Japan
INTRODUCTION
The problem of mining association rules from databases
was introduced by Agrawal, Imielinski, & Swami (1993). In
this problem, we give a set of items and a large collection
of transactions, which are subsets (baskets) of these
items. The task is to find relationships between the occur-
rences of various items within those baskets. Mining
association rules has been a central task of data mining,
which is a recent research focus in database systems and
machine learning and shows interesting applications in
various fields, including information management, query
processing, and process control.
When items contain quantitative and categorical val-
ues, association rules are called quantitative association
rules. For example, a quantitative association rule derived
from a regional household living standard investigation
database has the following form:
age [50,55] married house.
Here, the first item, age, is numeric, and the second
item is categorical. Categorical attributes can be con-
verted to numerical attributes in a straightforward way by
enumerating all categorical values and mapping them to
numerical values.
An association rule becomes a fuzzy association rule
if its items contain probabilistic values that are defined by
fuzzy sets.
BACKGROUND
Many results on mining quantitative association rules
can be found in Li, Shen, and Topor (1999) and Miller and
Yang (1997). A common method for quantitative associa-
tion rule mining is to map numerical attributes to binary
attributes, then use algorithms of binary association rule
mining. A popular technique for mapping numerical at-
tributes is to attribute discretization that converts a con-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
815
M
tinuous attribute value range to a set of discrete intervals
and then map all the values in each interval to an item of
binary values (Dougherty, Kohavi, & Sahami, 1995).
Two classical discretization methods are equal-width
discretization, which divides the attribute value range
into N intervals of equal width without considering the
population (number of instances) within each interval,
and equal-depth (or equal-cardinality) discretization,
which divides the attribute value range into N intervals of
equal populations without considering the similarity of
instances within each interval. Examples of using these
methods are given in Fukuda, Morimoto, Morishita, and
Tokuyama (1996); Miller and Yang (1997); and Srikant and
Agrawal (1996).
To overcome the problems of sharp boundaries
(Gyenesei, 2001) and expressiveness (Kuok, Fu, & Wong,
1998) in traditional discretization methods, methods for
mining fuzzy association rules were suggested. Many
traditional algorithms, such as Apriori, MAFIA, CLOSET,
and CHARM, were employed to discover fuzzy associa-
tion rules. However, the number of fuzzy attributes is
usually at least double the number of attributes; therefore,
these algorithms require huge computational times. To
reduce the computation cost of association mining, vari-
ous parallel algorithms based on count, data, and candi-
date distribution, along with other suitable strategies,
were suggested (Han, Karypis, & Kumar, 1997; Shen,
Liang, & Ng, 1999; Shen, 1999a; Zaki, Parthasarathy, &
Ogihara, 2001). Recently, a parallel algorithm for mining
fuzzy association rules, which divides the set of fuzzy
attributes into independent partitions based on the natu-
ral independence among fuzzy sets defined by the same
attribute, was proposed (Phan & Horiguchi, 2004b).
MAIN THRUST
This article introduces recent results of our research on
this topic in two prospects: mining quantitative associa-
tion rules and mining fuzzy association rules.
TEAM LinG

Mining Quantitative Association Rules
One method that was proposed is an adaptive numerical
value discretization method that considers both value
density and value distance of numerical attributes and
produces better quality intervals than the two classical
methods (Li et al., 1999). This method repeatedly selects
a pair of adjacent intervals that have the minimum differ-
ence to merge until a given criterion is met. It requires
quadratic time on the number of attribute values, because
each interval initially contains only a single value, and all
the intervals may be merged into one large interval con-
taining all the values in the worst case. A method of linear-
scan merging for quantizing numeric attribute values that
can be implemented in linear time sequentially and linear
cost in parallel was also proposed (Shen, 2001). This
method takes into consideration the maximal intrainterval
distance and load balancing simultaneously to improve
the quality of merging. In comparison with the existing
results in the same category, the algorithm achieves a
linear time speedup.
Suppose that a numerical attribute has m distinct
values, I = {x0 , x1 ,...x m1 } , where attribute value xi has ni
m 1
occurrences in the database (weight). Let N = i =0 ni be
the total number of attribute value occurrences, called
instances. Without loss of generality, we further assume
that xi < xi+1 for all 0<i<m-2 (otherwise, we can simply sort
these values). Define P to be a set of maximal disjoint
intervals on I, where interval IuP contains a sequence of
v 1
attribute values {xu, xu+1,,x v-1} and Nu = i = u ni instances,
v is the index of the next interval after Iu in P, and 0<u<v<m.
We also assume that Iu has a representative center, cu.
Initially, Iu contains only xu, which is also its representa-
tive center.
We define the maximal intrainterval distance, denoted
by D*(Iu; cu), as follows:
D * (cu , cv ) = max | xi cu |
i
Assume that two adjacent
intervals, I u = {xu ,...xv 1} and I v = {xv ,...xw1} , contain
v 1 w 1
N u = i =u ni and N v = i = v ni attribute value occurrences
v 1
and have representative centers cu = i =u xi ni / N u and
w 1
cv = i =v xi ni / N v , respectively, and 0<u<v<m. The union,
I u = I u U I v , of the two intervals containing (v-u)+(w-
w 1
v)=w-u attribute values and N u + N v = i =u ni instances to-
816
Mining Quantitative and Fuzzy Association Rules
tal thus has its representative center given by the average
weighted value of (cu, nu) and (cv, nv):
v 1 w 1
cu i =u ni + cv i =u ni cu N u + cv N v
cu = w 1
=
i =u
ni Nu + Nv
An optimal interval merge scheme produces a mini-
mum number of intervals whose maximal intrainterval
distances are each within a given threshold and whose
populations are as equal as possible. Assume that the
threshold for the maximal intrainterval difference is d,
which can be the average interinterval differences of all
the adjacent interval pairs or can be given by the system.
For k intervals, let the average population (support) of
each interval be N k = N / k and the population deviation
of interval Iu be u =| N u N k | , where Nu is the actual
population of Iu. Initially, Iu={xu} and 0<u<m-1. Our strat-
egy leads to the following algorithm for interval merging:
1. Partition {I0, I1, , Im-1} into a minimum number of
intervals such that each interval has a maximal
intrainterval distance not greater than d.
2. Assume that Step 1 produces k inter-
vals: {I u , I u ,..., I u } and 0=u0< u1< uk-1<m-1. For
0 1 k 1
I u j = [ X u j : cu j ] ,where X u j = {xu j , xu j +1 ,..., xu j +1 1 } and
cu j is the representative center of I u j , check to see
if moving xu 1 to I u will result in a better load
j +1 j +1
balance while preserving the maximal intrainterval
distance property, and do so if it will.
Noticing that x 0<x1 < < xm-1, we can implement Step
1 simply by using a linear scan to form appropriate seg-
ments of intervals after a single pass. Starting from I0 ,
merge Iu with Iu+j for j = 1,2, , until the next merge would
result in Ius maximal intrainterval distance greater than
the threshold; continue this process until no interval to be
merged remains. This process requires time O(m).
Step 2 examines every adjacent pair of intervals after
the merge, requiring, at most, m-1 steps. Each step checks
the changes of population deviation by moving uj+1-1
instances from I u to I u . That is, it considers whether the
j j +1
following condition holds:
| u j u j |<| u j +1 +u j +1 | ,
where u =| N u nu N k | and +u j =| N u j+1 nu j+1 1 N k | .
j j j +1
TEAM LinG
Mining Quantitative and Fuzzy Association Rules
Do the move only when the condition holds. Clearly, this
step requires O(m) time as well.
It was shown that the above linear-scan implementa-
tion indeed produces the minimum number of intervals
satisfying the intrainterval distance property.
Mining Fuzzy Association Rules
Let D be a relational database with I ={i1, , in} as the set
of n attributes and T={t1, ,tm} as the set of m records.
Using the fuzzification method, we associate each at-
tribute, iu, with a set of fuzzy sets: Fiu = { f i1u ,..., f iuk } . A fuzzy
association rule (Gyenesei, 2001; Kuok et al., 1998) is an
implication as follows:
X is A = Y is B ,
where X,Y I are disjointedly frequent itemsets. A and B
are sets of disjoint fuzzy sets corresponding to attributes
in X and Y: A= { f x ,... f x } , B = { f y ,... f y } , f x Fx , f y Fy .
1 p 1 p i i
A fuzzy itemset is now defined as a pair (<X, A>) of an
itemset X and a set A of fuzzy sets associated with at-
tributes in X. The support factor, denoted fs(<X, A>), is
determined by the formula
m 64 nodes. Each node consists of four 322MHz PowerPC
{
v =1
x1 (tv x1 ) ... x p (tv x p )} / T ,
where X ={x1,, xp} and tv is the vth record in T; is the T-
norm operator in fuzzy logic theory; x (tv xu ) is calculated
u
as
mxu (tv xu ), mxu (tv xu ) wxu
0 otherwise,
where mx is the membership function of fuzzy set f x ;
u u
and wx is a user-specified threshold of membership func-
u important association rules that exist in many real-life
tion mx . u
Each fuzzy attribute is a pair of the original attribute
name accompanied by the fuzzy set name. We perceive that
any fuzzy association rule never contains two fuzzy at-
tributes sharing a common original attribute in I. For
example, the rule Age_Old<
Cholesterol_High<Age_Young HeartDisease_Yes is
invalid because it contains Age_Old and Age_Young,
both of which derive from a common original attribute, Age.
There are two main reasons for this assumption. First,
fuzzy attributes sharing a common original attribute are M
usually mutually exclusive in meaning, so they would
largely reduce the support of rules in which they are
contained together. Second, such a rule would not be
worthwhile and would carry little meaning. Hence, we can
conclude that all fuzzy attributes in the same rule are
independent with respect to the fact that no pair of fuzzy
attributes whose original attributes are identical exists.
This observation is the foundation of our new parallel
algorithm.
The idea of partitioning algorithm is to divide the
original set of fuzzy attributes into separated parts (each
for a processor), so that every part retains at least one
fuzzy attribute for each original attribute. To do so, we
divide according to several original attributes, so that the
number of fuzzy attributes reduced at each processor is
maximized. After dividing the set of all the fuzzy at-
tributes for parallel processors, we can use any tradi-
tional algorithms, such as Apriori, CHARM, and so forth,
to mine local association rules. Finally, the local results
j j
at processors are gathered to constitute the overall
result. Space limitations prevent a detailed discussion of
the partitioning algorithm (FDivision) and the parallel
algorithm (PFAR) (Phan & Horiguchi, 2004b)
The PFAR algorithm was implemented using MPI
standard on the SP2 parallel system, which has a total of
604e processors, 512 MB local memory, and 9.1GB local
disks. In the experiments, we used only 24 processors
(PEs) on 24 nodes. The testing data included synthetical
data and real world databases of heart disease (created
by George John, 1994, statlog-adm@ncc.up.pt,
bob@stams.strathclyde.ac.uk). The experimental results
showed that the performance of PFAR is satisfactory
(Phan & Horiguchi, 2004a).
FUTURE TRENDS
Quantitative and fuzzy association rules are two types of
applications. Future research trends include mining them
among data that have structural properties such as se-
quence, time series, and spatiality. Mining these types of
rules in multidimensional databases is also a challenging
problem. These complex data mining tasks require tech-
niques not only for data mining but also for data represen-
tation, reduction, transformation, and visualization.
Completion of these tasks depends on successful integra-
tion of all the relevant techniques and effective applica-
tion of those techniques.
817
TEAM LinG

CONCLUSION
We have summarized research activities in mining quan-
titative and fuzzy association rules and some recent
results of our research on these topics. These results,
together with those given in Shen (1999b, 2005), show a
complete picture of the major achievements made within
our data mining research projects. We hope that this
article serves as a useful reference to the researchers
working in this area and that its contents provide interest-
ing techniques and tools to solve these two challenging
problems, which are significant in various applications.
REFERENCES
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
associations between sets of items in massive databases.
Proceedings of the ACM SIGMOD International Confer-
ence on Management of Data (pp. 207-216).
Dehaspe, L., & Raedt, L. D. (1997). Mining association
rules in multiple relations. Proceedings of the Seventh
International Workshop on Inductive Logic Program-
ming, 1297 (pp. 125-132).
Dougherty, J., Kohavi, R., & Sahami, M. (1995). Super-
vised and unsupervised discretization of continuous fea-
tures. Proceedings of the 12th International Conference
on Machine Learning (pp. 194-202).
Fukuda, T., Morimoto, Y., Morishita, S., & Tokuyama, T.
(1996). Mining optimized association rules for numeric
attributes. Proceedings of the 15th ACM SIGACT-
SIGMOD (pp. 182-191).
Gyenesei, A. (2001). A fuzzy approach for mining quanti-
tative association rules. Acta Cybernetica, 15(2), 305-
320.
Han, E-W., Karypis, G., & Kumar, V. (1997). Scalable
parallel data mining for association rules. ACM SIGMOD
Record, 26(2), 277-288.
Kuok, C. M., Fu, A., & Wong, M. H. (1998). Mining fuzzy
association rules in databases. ACM SIGMOD Record,
27(1), 41-46.
Li, J., Shen, H., & Topor, R. (1999). An adaptive method of
numerical attribute merging for quantitative association
rule mining. Proceedings of the 1999 International Com-
puter Science Conference (pp. 31-40), Hong Kong.
Miller, R. J., & Yang, Y. (1997). Association rules over
interval data. ACM SIGMOD Record, 26(2), 452-460.
818
Mining Quantitative and Fuzzy Association Rules
Phan, X. H., & Horiguchi, S. (2004a). A parallel algorithm
for mining fuzzy association rules (Tech. Rep. No. IS-RR-
2004-014, JAIST).
Phan, X. H., & Horiguchi, S. (2004b). Parallel data mining
for fuzzy association rules (Tech. Rep).
Shen, H. (1999a). Fast subset statistics and its applica-
tions in data mining. Proceedings of the International
Conference on Parallel and Distributed Processing Tech-
niques and Applications (pp. 1476-1480).
Shen, H. (1999b). New achivements in data mining. In J.
Sun (Ed.), Science: Advancing into the new millenium
(pp. 162-178). Beijing: Peoples Education.
Shen, H. (2001). Optimal attribute quantization for mining
quantitative association rules. Proceedings of the 2001
Asian High-Performance Computing Conference, Aus-
tralia.
Shen, H. (2005). Advances in mining association rules. In
John Wang (Ed.), Encyclopedia of data warehousing
and mining. Hershey, PA: Idea Group Reference.
Shen, H., Liang, W., & Ng, J. K-W. (1999). Efficient
computation of frequent itemsets in a subcollection of
multiple set families. Informatica, 23(4), 543-548.
Srikant, R., & Agrawal, R. (1996). Mining quantitative
association rules in large relational tables. ACM SIGMOD
Record, 25(2), 1-8.
Zaki, M. J., Parthasarathy, S., & Ogihara, M. (2001).
Parallel data mining for association rules on shared-
memory systems. Knowledge and Information Systems,
3(1), 1-29.
KEY TERMS
Attribute Discretization: The process of converting
a (continuous) attribute value range to a set of discrete
intervals and representing all the values in each interval
with a single (interval) label.
Attribute Fuzzification: A process that converts
numeric attribute values into membership values of fuzzy
sets.
Equal-Depth Discretization: A technique that divides
the attribute value range into intervals of equal popula-
tion (value density).
Equal-Width Discretization: A technique that di-
vides the attribute value range into intervals of equal
width (length).
TEAM LinG
Mining Quantitative and Fuzzy Association Rules

Fuzzy Association Rule: An implication rule showing

the conditions of co-occurrence of itemsets that are M
defined by fuzzy sets whose elements have probabilistic
values.
Quantitative Association Rule: An implication rule
containing items of quantitative and/or categorical val-
ues that shows in a given database.

819

TEAM LinG
820
Model Identification through Data Mining
Diego Liberati
Consiglio Nazionale delle Ricerche, Italy
INTRODUCTION
In many fields of research as in everyday life, one has to
face a huge amount of data, often not completely homo-
geneous and many times without an immediate grasp of
an underlying simple structure. A typical example is the
growing field of bio-informatics, where new technologies,
like the so-called micro-arrays, provide thousands of
gene expression data on a single cell in a simple and
quickly integrated way. Further, the everyday consumer
is involved in a process not so different from a logical
point of view, when the data associated with the
consumers identity contributes to a large database of
many customers whose underlying consumer trends are
of interest to the distribution market.
The large number of variables (i.e., gene expressions,
goods) for so many records (i.e., patients, customers)
usually are collected with the help of wrapping or ware-
housing approaches in order to mediate among different
repositories, as described elsewhere in the encyclopedia.
Then, the problem arises of reconstructing a synthetic
mathematical model, capturing the most important rela-
tions between variables. This will be the focus of the
present contribution.
A possible approach to blindly building a simple linear
approximating model is to resort to piece-wise affine
(PWA) identification (Ferrari-Trecate et al., 2003). The
rationale for this model will be explored in the second part
of the methodological section of this article.
In order to reduce the dimensionality of the problem,
thus simplifying both the computation and the subse-
quent understanding of the solution, the critical problems
of selecting the most salient variables must be solved. A
possible approach is to resort to a rule induction method,
like the one described in Muselli and Liberati (2002) and
recalled in the first methodological part of this contribu-
tion. Such a strategy offers the advantage of extracting
underlying rules and implying conjunctions and/or dis-
junctions between such salient variables. The first idea of
their non-linear relationships is provided as a first step to
designing a representative model,using the selected vari-
ables.
The joint use of the two approaches recalled in this
article, starting from data without known background
information about their relationships, first allows a reduc-
tion in dimensionality without significant loss in informa-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
tion, and later to infer logical relationships. Finally, it
allows the identification of a simple input-output model of
the involved process that also could be used for control-
ling purposes.
BACKGROUND
The two tasks of selecting salient variables and identify-
ing their relationships from data may be sequentially
accomplished with various degrees of success in a variety
of ways. Principal components order the variables from
the most salient to the least, but only under a linear
framework. Partial least squares allow an extension to
non-linear models, provided that one has prior informa-
tion on the structure of the involved non-linearity; in fact,
the regression equation needs to be written before iden-
tifying its parameters. Clustering may operate in an unsu-
pervised way without the a priori correct classification of
a training set (Booley, 1998). Neural networks are known
to use embedded rules with the indirect possibility (Taha
& Ghosh, 1999)of making rules explicit or to underline the
salient variables. Decision trees (Quinlan, 1994), a popu-
lar framework, can provide a satisfactory answer to both
questions.
MAIN THRUST
Recently, a different approach has been suggestedHam-
ming clustering. This approach is related to the classical
theory exploited in minimizing the size of electronic circuits,
with the additional ability to obtain a final function able to
generalize everything from the training dataset to the most
likely framework describing the actual properties of the
data. In fact, the Hamming metric tends to cluster samples
with code that is less distant. This is likely to be natural if
variables are redundantly coded via thermometer (for nu-
meric variables) or used for only-one (for logical variables)
code (Muselli & Liberati, 2000). The Hamming clustering
approach reflects the following remarkable properties:
It is fast, exploiting (after the mentioned binary
coding) just logical operations instead of floating
point multiplications.
TEAM LinG
Model Identification through Data Mining
It directly provides a logical understandable expres-
sion (Muselli & Liberati, 2002), which is the final
synthesized function directly expressed as the OR
of ANDs of the salient variables, possibly negated.
When the variables are selected, a mathematical model
of the underlying generating framework still must be
produced. At this point, a first hypothesis of linearity may
be investigated (usually being only a very rough approxi-
mation) where the values of the variables are not close to
the functioning point around which the linear approxima-
tion is computed.
Building a non-linear model is far from easy; the
structure of the non-linearity needs to be a priori knowl-
edge, which is not usually the case. A typical approach
consists of exploiting a priori knowledge, when available,
to define a tentative structure, then refining and modify-
ing it on the training subset of data, and finally retaining
the structure that best fits a cross-validation on the
testing subset of data. The problem is even more complex
when the collected data exhibit hybrid dynamics (i.e., their
evolution in time is a sequence of smooth behaviors and
abrupt changes).
An alternative approach is to infer the model directly
from the data without a priori knowledge via an identifica-
tion algorithm capable of reconstructing a very general
class of a piece-wise affine model (Ferrari-Trecate et al.,
2003). This method also can be exploited for the data-
driven modeling of hybrid dynamic systems, where logic
phenomena interact with the evolution of continuous-
valued variables. Such an approach will be described
concisely later, after a more detailed drawing of the rules-
oriented mining procedure, and some applications will be
discussed briefly.
Binary Rule Generation and Variable
Selection While Mining Data
The approach followed by Hamming clustering in mining
the available data to select the salient variables and to
build the desired set of rules consists of the three steps
in Table 1.
Step 1: A critical issue is the partition of a possibly
continuous range in intervals, whose number and
limits may affect the final result. The thermometer
code then may be used to preserve ordering and
distance (in the case of nominal input variables, for
which a natural ordering cannot be defined, instead
adopting the only-one). The simple metric used is
the Hamming distance, computed as the number of
different bits between binary strings. In this way,
the training process does not require floating point
computation but rather basic logic operations. This
Table 1. The three steps executed by Hamming clustering
to build the set of rules embedded in the mined data M
1. The input variables are converted into binary strings
via a coding designed to preserve distance and, if
relevant, ordering.
2. The OR of ANDs expression of a logical function
is derived from the training examples coded in the
binary form of step 1.
3. In the OR final expression, each logical AND pro-
vides intelligible conjunctions or disjunctions of the
involved variables, ruling the analyzed problem.
is one reason for the algorithm speed and for its
insensitivity to precision.
Step 2: Classical techniques of logical synthesis are
specifically designed to obtain the simplest AND-
OR expression able to satisfy all the available input-
output pairs without an explicit attitude to generali-
zation. To generalize and infer the underlying rules
at every iteration, by Hamming clustering groups
together in a competitive way, binary strings have
the same output and are close to each other. A final
pruning phase does simplify the resulting expres-
sion, further improving its generalization ability.
Moreover, the minimization of the involved vari-
ables intrinsically excludes the redundant ones,
thus enhancing the very salient variables for the
investigated problem. The low (quadratic) compu-
tational cost allows for managing quite large
datasets.
Step 3: Each logical product directly provides an
intelligible rule, synthesizing a relevant aspect of
the underlying system that is believed to generate
the available samples.
Identification of Piece-wise Affine
Systems Through a Clustering
Technique
Once the salient variables have been selected, it may be
of interest to capture a model of their dynamical interac-
tion. Piece-wise affine identification exploits K-means
clustering that associates data points in multivariable
space in such a way that jointly determines a sequence
of linear submodels and their respective regions of opera-
tion without even imposing continuity at each change in
the derivative. In order to obtain such a result, the five
steps reported in Table 2 are executed.
Step 1: The model is locally linear; small sets of data
points close to each other likely belong to the same
submodel. For each data point, a local set is built,
821
TEAM LinG

collecting the selected points together with a given
number of its neighbors (whose cardinality is one of
the parameters of the algorithm). Each local set will
be pure if made of points really belonging to the same
single linear subsystem; otherwise, it is mixed.
Step 2: For each local dataset, a linear model is
identified through a usual least squares procedure.
Pure sets belonging to the same submodel give
similar parameter sets, while mixed sets yield isolated
vectors of coefficients, looking as outliers in the
parameter space. If the signal-to-noise ratio is good
enough, and if there are not too many mixed sets (i.e.,
the number of data points is more than the number of
submodels to be identified, and the sampling is fair
in every region), then the vectors will cluster in the
parameter space around the values pertaining to each
submodel, apart from a few outliers.
Step 3: A modified version of the classical K-means,
whose convergence is guaranteed in a finite number
of steps (Ferrari-Trecate et al., 2003), takes into
account the confidence on pure and mixed local sets
in order to cluster the parameter vectors.
Step 4: Data points are then classified, each belong-
ing to a local dataset one-to-one related to its gen-
erating data point, which is classified according to
the cluster to which its parameter vector belongs.
Step 5: Both the linear submodels and their regions
are estimated from the data in each subset. The
coefficients are estimated via weighted least squares,
taking into account the confidence measures. The
shape of the polyhedral region characterizing the
domain of each model may be obtained via linear
support vector machines (Vapnik, 1998), easily solved
via linear/quadratic programming.
A Few Applications
The field of application of the proposed approach is
intrinsically wide (both tools are most general and quite
powerful, especially if combined together). Here, only a
few suggestions will be drawn with reference to already
obtained results or to some application under consider-
Table 2. The five steps for piece-wise affine identification
1. The local datasets neighboring each sample are built.
2. The local linear models are identified through least squares.
3. The parameters vectors are clustered through a modified
K-means.
4. The data points are classified according to the clustered
parameter vectors.
5. The submodels are estimated together with their domains.
822
Model Identification through Data Mining
ation for ongoing research projects. The field of life
science plays a central role because of its relevance to
science and to society.
A growing acceptance of international relevance, in
which the described approaches are being used to
provide a contribution, is the so-called field of systems
biologya feedback model of how proteins interact with
each other and with nucleic acids within the cell, both of
which are needed to better understand the control mecha-
nisms of the cellular cycle. This is especially evident with
respect to duplication (such as cancer, when such
mechanisms become out of control). Such an under-
standing will hopefully encourage a drive to personal
therapy, when everybodys gene expression will be cor-
related to the amount of corresponding proteins in-
volved in the cellular cycle. Moreover, a new computa-
tional paradigm could arise by exploiting biological com-
ponents like cells instead of the usual silicon hardware,
thus overcoming some technological issues and possi-
bly facilitating neuroinformatics.
The study of systems biology as a leading edge of the
large field of bioinformatics, begins by analyzing data
from so-called micro-arrays. They are little standard
chips, where thousands of gene expressions may be
obtained from the same cell material, thus providing a
large amount of data whose handling with the usual
deterministic approaches is not conceivable, and whose
fault is its inability to obtain significant synthetic infor-
mation. Thus, matrices of as many subjects as available,
possibly grouped in homogeneous categories for super-
vised training, each one carrying thousands of gene
expressions, are the natural input to algorithms like
Hamming clustering. The desired output are rules able to
classify; for instance, patients affected by different tu-
mors from healthy subjects, on the basis of a few iden-
tified genes, whose set is the candidate basis for the
piece-wise linear model describing their complex interac-
tion in such a particular class of subjects.
Also, without deeper insight into the cell, the iden-
tification of the prognostic factors in oncology is already
occuring with Hamming clustering (Paoli et al., 2000) and
also providing a hint about their interaction, which is not
explicit in the outcome of a simple neural network (Drago
et al., 2002).
FUTURE TRENDS
Beside improvements in the algorithmic approaches (also
implying the possibility of taking into account potential
a priori knowledge), a wide range of applications will
benefit from the proposed ideas in the near future; some
of which are outlined here and partially recalled in Table
3.
TEAM LinG
Model Identification through Data Mining

Drug design would benefit from the a priori forecast-
ing provided by Hamming clustering about hydrophilic
behavior of the not yet experimented pharmacological
molecule, on the basis of the known properties of some
possible radicals and in the track of Manga et al. (2003)
within the frame of computational methods for the predic-
tion of drug-likeness (Clark & Pickett, 2000). In principle,
such not quite different in silico predictions from the
systems biology expectation are of paramount relevance
to pharmaceutical companies to save money in designing
minimally invasive drugs controlling kidney metabolism
that would have less of an affect on the liver the more
hydrophilic they are.
Compartmental behavior of drugs then may be
analysed via piece-wise identification by collecting in
vivo data samples and clustering them within the more
active compartment at each stage, instead of the classical
linear system identification that requires non-linear algo-
rithms. The same is true for metabolism, such as glucose
in diabetes or urea in renal failure. Dialysis can be modeled
as a compartmental process in a sufficiently accurate way.
A piece-wise linear approach is able to simplify the iden-
tification even on a single patient, when population data
are not available to allow a simple linear deconvolution
approach (Liberati & Turkheimer, 1999) (whose result is
only an average knowledge of the overall process, with-
out taking into special account the very subject under
analysis). Moreover, compartmental models are perva-
sive, like in ecology, wildlife, and population studies;
potential applications in that direction are almost never
ending.
Many physiological processes are switching natu-
rally or intentionally from an active to quiescent state, like
hormone pulses whose identification (Sartorio et al., 2002)
is important in growth and fertility diseases as well as in
doping assessment. In that respect, the most fascinating
human organ is probably the brain, whose study may be
undertaken today either in the sophisticated frame of M
functional nuclear magnetic imaging or in the simple way
of EEG recording. Multidimensional (three space vari-
ables plus time) images or multivariate time series provide
an abundance of raw data to mine in order to understand
which kind of activation is produced in correspondence
with an event (Maieron et al., 2002) or a decision. A brain
computer-interfacing device then may be built that is able
to reproduce ones intention to perform a movenot
directly possible for the subject in some impaired physical
conditionsand command a proper actuator. Both Ham-
ming clustering and piece-wise affine identification would
improve the only partially successful approach based on
artificial neural networks (Babiloni et al., 2000). A simple
drowsiness detector based on the EEG may be designed, as
well as a flexible anaesthesia/hypotension level detector,
without needing a time-varying more precise, but more
costly, identification. A psychological stress indicator
may be inferred, outperforming (Pagani et al., 1991). The
multivariate analysis (Liberati et al., 1997), possibly taking
into account the input stimulation so useful in approaching
difficult neurological tasks (like modeling electro
encephalographic coherence in Alzheimers disease pa-
tients (Locatelli et al., 1998)) or non-linear effects in muscle
contraction (Orizio et al., 1996) would be outperformed by
piece-wise linear identification, even in time-varying con-
texts like epilepsy.
Industrial applications, of course, are not excluded
from the field of possible interests. In Ferrari-Trecate et al.
(2003), for instance, the classification and identification
of the dynamics of an industrial transformer are performed
via the piece-wise approach, with a much simpler cost, and
no really significant reduction of performances with re-
spect to the non-linear modeling described in Bittanti, et
al. (2001).

Table 3. A summary of selected applications of the proposed approaches

System Biology and Bioinformatics: To identify the interaction of the main proteins involved in cell cycle.
Drug Design: To forecast desired and undesired behavior of the final molecule from components.
Compartmental Modeling: To identify number, dimensions, and exchange rates of communicating reservoirs.
Hormone Pulses Detection: To detect true pulses among the stochastic oscillations of the baseline.
Sleep Detector: To forecast drowsiness as well as to identify sleep stages and transitions.
Stress Detector: To identify the psychological state of a subject from multivariate analysis of biological signals.
Prognostic Factors: To identify the interaction between selected features (e.g., in oncology).
Pharmacokinetics and Metabolism: To identify diffusion and metabolic time constants from time series
sampled in blood.
Switching Processes: To identify abrupt or possibly smoother commutations within the process duty cycle.
Brain Computer Interfacing: To identify a decision taken within the brain and propagate it directly form
brain waves.
Anesthesia Detector: To monitor the level of anesthesia and provide close-loop control.
Industrial Applications: A wide spectrum of logical or dynamic-logical hybrid problems may be faced (i.e., the
tracking of a transformer).

823

TEAM LinG

CONCLUSION
The proposed approaches are very powerful tools for
quite a wide spectrum of applications in and beyond data
mining, providing an up-to-date answer to the quest of
formally extracting knowledge from data and sketching a
model of the underlying process.
ACKNOWLEDGMENTS
Marco Muselli had the key role in conceiving and devel-
oping the recalled clustering algorithms, while Giancarlo
Ferrari-Trecate has been determinant in extending them to
model identification as well as in kindly revising the
present contribution. Their stimulating and friendly inter-
action over the past few years is warmly acknowledged.
An anonymous reviewer is also gratefully acknowledged
for indicating how to improve the writing in several
critical points.
REFERENCES
Babiloni, F. et al. (2000). Comparison between human and
ANN detection of laplacian-derived electroencephalographic
activity related to unilateral voluntary movements. Comput
Biomed Res, 33, 59-74.
Bittanti, S., Cuzzola, F.A., Lorito, F., & Poncia, G. (2001).
Compensation of nonlinearities in a current transformer
for the reconstruction of the primary current. IEEE T
Control System Techn, 9(4), 565-573.
Boley, D.L. (1998). Principal direction divisive partition-
ing. Data Mining and Knowledge Discovery, 2(4),
325-344.
Clark, D.E., & Pickett, S.D. (2000). Computational methods
for the prediction of drug-likeness. Drug Discovery
Today, 5(2), 49-58.
Drago, G.P., Setti, E., Licitra, L., & Liberati, D. (2002).
Forecasting the performance status of head and neck
cancer patient treatment by an interval arithmetic pruned
perceptron. IEEE T Bio-Med Eng, 49(8), 782-787.
Ferrari, T.G., Muselli, M., Liberati, D., & Morari, M. (2003).
A clustering technique for the identification of piecewise
affine systems. Automatica, 39, 205-217.
Liberati, D., Cursi, M., Locatelli, T., Comi, G., & Cerutti, S.
(1997). Total and partial coherence of spontaneous and
evoked EEG by means of multi-variable autoregressive
processing. Med Biol Eng Comput, 35(2), 124-130.
824
Model Identification through Data Mining
Liberati, D., & Turkheimer, F. (1999). Linear spectral
deconvolution of catabolic plasma concentration decay
in dialysis. Med Biol Eng Comput, 37, 391-395.
Locatelli, T., Cursi, M., Liberati, D., Franceschi, M., & Comi,
G. (1998). EEG coherence in Alzheimers disease.
Electroenceph Clin Neurophys, 106(3), 229-237.
Maieron, M. et al. (2002). An ARX model-based approach
to the analysis of single-trial event-related fMRI data.
Proceedings of the 8th International Conference on
Functional Mapping of the Human Brain, Sendai,
Japan.
Manga, N., Duffy, J.C., Rowe, P.H., & Cronin, M.T.D.
(2003). A hierarchical quantitative structure-activity rela-
tionship model for urinary excretion of drugs in humans
as predictive tool for biotrasformation. Quantita-
tive Structure-Activity Relationship Comb Sci, 22,
263-273.
Muselli, M., & Liberati, D. (2000). Training digital circuits
with Hamming clustering. IEEE T Circuits and Systems
I: Fundamental Theory and Applications, 47, 513-527.
Muselli, M., & Liberati, D. (2002). Binary rule generation
via Hamming clustering. IEEE T Knowledge and Data
Eng, 14, 1258-1268.
Orizio, C., Liberati, D., Locatelli, C., DeGrandis, D., &
Veicsteinas, A. (1996). Surface mechanomyogram re-
flects muscle fibres twitches summation. J Biomech,
29(4), 475-481.
Pagani, M. et al. (1991). Sympatho-vagal interaction dur-
ing mental stress: A study employing spectral analysis of
heart rate variability in healthy controls and in pa-
tients with a prior myocardial infarction. Circulation,
83(4), 43-51.
Paoli, G. et al. (2000). Hamming clustering techniques for
the identification of prognostic indices in patients with
advanced head and neck cancer treated with radiation
therapy. Med Biol Eng Comput, 38, 483-486.
Quinlan, J.R. (1994). C4.5: Programs for machine learn-
ing. San Francisco: Morgan Kaufmann.
Sartorio, A., De Nicolao, G., & Liberati, D. (2002). An
improved computational method to assess pituitary re-
sponsiveness to secretagogue stimuli. Eur J Endocrinol,
147(3), 323-332.
Taha, I., & Ghosh, J. (1999). Symbolic interpretation of
artificial neural networks. IEEE T Knowledge and Data
Eng., 11, 448-463.
TEAM LinG
Model Identification through Data Mining
Vapnik, V. (1998). Statistical learning theory. New York:
Wiley.
KEY TERMS
Bioinformatics: The processing of the huge amount
of information pertaining to biology.
Hamming Clustering: A fast binary rule generator
and variable selector able to build understandable logical
expressions by analyzing the Hamming distance between
samples.
Hybrid Systems: Their evolution in time is composed
by both smooth dynamics and sudden jumps.
Micro-Arrays: Chips where thousands of gene ex-
pressions may be obtained from the same biological cell M
material.
Model Identification: Definition of the structure and
computation of its parameters best suited to mathemati-
cally describe the process underlying the data.
Rule Generation: The extraction from the data of the
embedded synthetic logical description of their relation-
ships.
Salient Variables: The real players among the many
apparently involved in the true core of a complex business.
Systems Biology: The quest of a mathematical model
of the feedback regulation of proteins and nucleic acids
within the cell.
825
TEAM LinG
826
Modeling Web-Based Data in a Data Warehouse
Hadrian Peter
University of the West Indies, Barbados
Charles Greenidge
University of the West Indies, Barbados
INTRODUCTION
Good database design generates effective operational
databases through which we can track customers, sales,
inventories, and other variables of interest. The main
reason for generating, storing, and managing good data
is to enhance the decision-making process. The tool used
during this process is the decision support system (DSS).
The information requirements of the DSS have become so
complex, that it is difficult for it to extract all the necessary
information from the data structures typically found in
operational databases. For this reason, a new storage
facility called a data warehouse has been developed. Data
in the data warehouse have characteristics that are quite
distinct from those in the operational database (Rob &
Coronel, 2002).
The data warehouse extracts or obtains its data from
operational databases as well as from other sources, thus
providing a comprehensive data pool from which useful
information can be extracted. The other sources represent
the external data component of the data warehouse. Exter-
nal information in corporate data warehouses has in-
creased during the past few years because of the wealth
of information available, the desire to work more closely
with third parties and business partners, and the Internet.
In this paper, we focus on the Internet as the source
of the external data and propose a model for representing
such data. The model represents a timely intervention by
two important technologies (i.e., data warehousing and
search engine technology) and is a departure from current
thinking of major software vendors (e.g., IBM, Oracle,
MicroSoft) that seeks to integrate multiple tools into one
environment with one administration. The paper is orga-
nized as follows: background on the topic is discussed in
the following section, including a literature review on the
main areas of the paper; the main thrust section discusses
the detailed model, including a brief analysis of the
model; finally, we identify and explain the future trends
of the topic.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
Data Warehousing
Data warehousing is a process concerned with the collec-
tion, organization, and analysis of data typically from
several heterogeneous sources with an aim to augment
end-user business functions (Hackathorn, 1998; Strehlo,
1996). It is intended to provide a working model for the
easy flow of data from operational systems to decision
support systems. The data warehouse structure includes
three major levels of informationgranular data, archival
data, and summary dataand the metadata to support
them (Inmon, Zachman & Geiger, 1997).
Data warehousing:
1. Is centered on ad hoc end-user queries posed by
business users rather than by information system
experts.
2. Is concerned with off-line, historical data rather
than online, volatile, operational type data.
3. Must efficiently handle larger volumes of data than
those handled by normal operational systems.
4. Must present data in a form that coincides with the
expectations and understanding of the business
users of the system rather than that of the informa-
tion system architects.
5. Must consolidate data elements. Diverse opera-
tional systems will refer to the same data in different
ways.
6. Relies heavily on meta-data. The role of meta-data
is particularly vital, as data must remain in its proper
context over time.
The main issues in data warehousing design are:
1. performance versus flexibility;
2. cost;
3. maintenance.
TEAM LinG
Modeling Web-Based Data in a Data Warehouse

Details of data warehousing/warehouses issues are
provided in Berson and Smith (1997), Bischoff and Yevich
(1996), Devlin (1998), Hackathorn (1995, 1998), Inmon
(2002), Mattison (1996), Rob and Coronel (2002), Becker
(2002), Kimball and Ross (2002), Lujan-Mora and Trujillo (2003).
acted on by the search engine. Results coming from the
search engine also can be processed prior to being re- M
layed to the warehouse. This intermediate, independent
meta-data bridge is an important concept in this model.

External Data and Search Engines THE DETAILED MODEL

External data represent an essential ingredient in the Motivation/Justification

production of decision-support analysis in the data ware-
house environment (Inmon, 2002). The ability to make
effective decisions often requires the use of information
that is generated outside the domain of the person making
the decision (Parsaye, 1996). External data sources pro-
vide data that cannot be found within the operational
database but are relevant to the business (e.g., stock
prices, market indicators, and competitors data).
Once in the warehouse, the external data go through
processes that enable them to be interrogated by the end
user and business analysts seeking answers to strategic
questions. This external data may augment, revise, or
even challenge the data residing in the operational sys-
tems (Barquin & Edelstein, 1997).
Our model provides a cooperative nexus between the
data warehouse and search engine, and is aimed at en-
hancing the collection of external data for end-user que-
ries originating in the data warehouse (Bischoff & Yevich,
1996). The model grapples with the complex design differ-
ences between data warehouse and search engine tech-
nologies by introducing an independent, intermediate,
data-staging layer.
Search engines have been used since the 1990s to
grant to millions access to global external data via the
Internet (Goodman, 2000; Sullivan, 2000). In spite of their
shortcomings (Chakrabarti, et al., 2001; Soudah, 2000),
search engines still represent the best approach to the
retrieval of Internet-based external data (Goodman, 2000;
Sullivan, 2000). Importantly, under our model, queries
originating with the business expert on the warehouse
side can be modified at an intermediate stage before being
To better understand the importance of the new model, we
must examine the search engine and meta-data engine
components. The meta-data engine cooperates closely
with both the data warehouse and search engine architec-
tures to facilitate queries and efficiently handle results. In
this paper, we propose a special-purpose search engine,
which is a hybrid between the traditional, general-pur-
pose search engine and the meta-search engine.
The hybrid approach we have taken is an effort to
maximize the efficiency of the search process. Although
the engine does not need to produce results in a real-time
mode, the throughput of the engine may be increased by
using multi-threading.
The polling of the major commercial engines by our
engine ensures the widest possible access to Internet
information (Bauer et al., 2002; Pfaffenberger, 1996; Ray et
al., 1998; Sullivan, 2000).
The special-purpose hybrid search engine in Figure 1
goes through the following steps each time a query is
processed:
1. Retrieve major search engines home pages HTML
documents;
2. Parse files and submit GET type queries to major
engines;
3. Retrieve results and use links found in results to
form starting seed links;
4. Perform depth/breath first traversal using seed links;
and
5. Capture results from no. 4 to local disk.

Figure 1. Query and retrieval in hybrid search engine

User Filter

Final Download
Query Results
Submit Major PARSE
Query Engines HTML FILES
FILES
Store
Links
Download Seed
Results Follow Links
Invoke Meta- HTML Seed
Data Engine Links
FILES
on Results

827

TEAM LinG

Architecture
It is evident that our proposed system uses the Internet to
supplement the external data requirements of the data
warehouse. To function effectively, the system must ad-
dress the issues surrounding the architectures of the data
warehouse and search engine. The differences are in the
following areas:
1. Subject Area Specificity
2. Granularity
3. Time Basis (Time Stamping)
4. Frequency of Updates
5. Purity of Data
To address these areas, we need a flexible architecture
that can provide a bridge over which relevant data can be
transmitted. This bridge needs to be a communication
channel, as well, so that an effective relationship between
the data warehouse and search engine domains is main-
tained. One approach is to integrate the search engine into
the warehouse. This approach, though possible, is not
expedient, because the warehouse environment is not an
operational type environment but one optimized for the
analysis of enterprise-wide historical data.
Typical commercial search engines are composed of
a crawler (spider) and an indexer (mite). The indexer is used
to codify results into a database for easy querying. Our
approach reduces the complexity of the search engine by
moving the indexing tasks to the meta-data engine. The
purpose of the meta-data engine is to facilitate an auto-
matic information retrieval mode. Figure 2 illustrates the
concept of the bridge between data warehouse and search
engine environments.
The meta-data engine provides a useful transaction
point for both queries and results. It is useful architectur-
ally and practically to keep the search engine and data
warehouse separate. The separation of the three architec-
tures means that they can be optimized independently. The
meta-data engine performs a specialized task of interfacing
Figure 2. Meta-data engine operation
Handle Query
Filter
from D.W.
Meta-Data
Format
Provide Results to
D.W.
828
Modeling Web-Based Data in a Data Warehouse
between two distinct environments, and, as such, we
believe that it should be developed separately.
We now briefly examine each of the four components
of the meta-data engine operation as shown in Figure 2.
First, queries originating from the warehouse will have to
be filtered to ensure they are actionable; that is, non-
redundant and meet the timeliness and other processing
requirements of the search engine component. Second,
incoming queries may have to be modified in order to
meet the goal of being actionable or in order to meet
specific efficiency/success criteria for the search engine.
For example, the recall of results may be enhanced, if
certain terms are included/excluded from the original
query coming from the warehouse. Third, after results are
retrieved, some minimal analysis of those results will
have to occur to determine either the success or failure
of the retrieval process or to verify the content type or
structure (e.g., HTML, XML, .pdf) of the retrieved re-
sults. Due to the ability of search engines to return large
volumes of irrelevant results, some preliminary analysis
is warranted. More advanced versions of the engine
would tackle semantic issues. The fourth component
(labeled format in the diagram) will be invoked to produce
plain text from the recognizable document formats or,
alternatively, to convert from one known document for-
mat to another.
Off-the-shelf tools exist online to parse HTML, XML,
and other Internet document formats as well as provide
language toolkits from which these four important areas
of meta-data engine construction can be done. The Web
provides a sprawling proving ground of semi-structured,
structured, and unstructured data that must be handled
using multiple syntactic, structural, and semantic tech-
niques. The meta-data engine seeks to architecturally
isolate these techniques from the data warehouse and
search engine domains.
Also, in our proposed model, the issue of relevance
must be addressed. Therefore, a way must be found to
sort out relevant from non-relevant documents. For ex-
ample, we are interested in questions such as the follow-
Submit Modified
Modify
Queries to S.E.
Analyze
Handle Results
from S.E.
TEAM LinG
Modeling Web-Based Data in a Data Warehouse
ing: Does every page containing the word Java specifi-
cally address Java programming issues or tools?
The Need for Data Synchronization
Data synchronization is an important issue in data ware-
housing. Data warehouses are designed to provide a
consistent view of data emanating from several sources.
The data from these sources must be combined, in a
staging area outside of the warehouse, through a process
of transformation and integration. To achieve this goal
requires that the data drawn from the multiple sources be
synchronized. Data synchronization can only be guaran-
teed by safeguarding the integrity of the data and ensur-
ing that the timeliness of the data is preserved.
We propose that the warehouse transmit a request
for external data to the search engine. Once the eventual
results are obtained, they are left for the warehouse to
gather as it completes a new refresh cycle. This process
is not intended to provide an immediate response (as
required by a Web surfer) but to lodge a request that will
be answered at some convenient time in the future.
At their simplest, data originating externally may be
synchronized by their date of retrieval as well as by
system statistics such as last time of update, date of
posting, and page expiration date. More sophisticated
schemes could include verification of dates of origin, use
of Internet archives, and automated analysis of content to
establish dates. For example, online newspaper articles
often carry the date of publication close to the headlines.
An analysis of such a documents content would reveal
a date field in a position adjacent to the heading for the story.
Analysis of the Model
The major strengths of our model are its:
Independence;
Value-added approach; and
Flexibility and security
This model exhibits (a) logical independence (ensures
that each component of the model retains its integrity); (b)
physical independence (allows summarized results of
external data to be transferred via network connections to
any remote physical machines; and (c) administrative
independence (administration of each of the three compo-
nents in the model is a specialized activity; hence, the
need for three separate administrators).
In terms of its value-added approach, the model maxi-
mizes the utility of acquired data by adding value to this
data long after their day-to-day usefulness has expired.
The ability to add pertinent external data to volumes of
aging internal data extends the usefulness of the internal
data. With careful selection of external data, organiza-
tions with data warehouses can reap disproportionate M
benefits over competitors who do not use their historical
or external data to maximum effect.
Our model also allows for flexibility. By allowing de-
velopment to take place in three languagesquery lan-
guage SQL (e.g., for the data warehouse), Perl for the
meta-data engine, and Java for the search enginethe
strengths of each can be utilized effectively. By allowing
flexibility in the actual implementation of the model, we
can increase efficiency and decrease development time.
Our model allows for security and integrity to be
maintained. The warehouse need not expose itself to
dangerous integrity questions or to increasingly mali-
cious threats on the Internet. By isolating the module that
interfaces with the Internet from the module that carries
vital internal data, the prospects for good security are
improved.
Our model also relieves information overload. By us-
ing the meta-data engine to manipulate external data, we
are indirectly removing time-consuming filtering activi-
ties from users of the data warehouse.
FUTURE TRENDS
It is our intention to incorporate portals (Hall, 1999; Raab,
2000) and vortals (Chakrabarti et al., 2001; Rupley, 2000)
in our continued research in data warehousing. A goal of
our model is to provide a subject-oriented bias of the
warehouse when satisfying queries involving the identi-
fication and extraction of external data from the Internet.
The following three potential problems are still to be
addressed in our model:
1. It relies on fast-changing search engine technologies;
2. It introduces more system overhead and the need
for more administrators;
3. It results in the storage of large volumes of irrel-
evant/worthless information.
Other issues to be addressed by our model are:
1. The need to investigate how the search engine
component can make smarter choices of which links
to follow;
2. The need to test the full prototype in a live data
warehouse environment;
3. Providing a more detailed comparison between our
model and the traditional external data approaches.
4. Determining how the traditional Online Transaction
Processing (OLTP) systems will funnel data into
this model.
829
TEAM LinG

We are confident that further research with our model
will provide the techniques for handling the aforemen-
tioned issues. One possible approach is to incorporate
languages such as WebSQL (Mihaila, 1996) and Squeal
(Spertus, 1999) in our model.
CONCLUSION
The model presented in this paper is an attempt to bring
the data warehouse and search engine into cooperation to
satisfy the external data requirements for the warehouse.
However, the new model anticipates the difficulty of
simply merging two conflicting architectures. The ap-
proach we have taken is the introduction of a third inter-
mediate layer called the meta-data engine. The role of this
engine is to coordinate data flows to and from the respec-
tive environments. The ability of this layer to filter extra-
neous data retrieved from the search engine, as well as
compose domain-specific queries, will determine its success.
In our model, the data warehouse is seen as the agent
that initiates the search for external data. Once a search
has been started, the meta-data engine component takes
over and eventually returns results. The model provides
for querying of general-purpose search engines, but a
simple configuration allows for subject-specific engines
(vortals) to be used in the future. When specific engines
designed for specific niches become widely available, the
model stands ready to accommodate this change.
We believe that WebSQL and Squeal can be used in
our model to perform the structured queries that will
facilitate speedy identification of relevant documents. In
so doing, we think that these tools will allow for faster
development and testing of a prototype and, ultimately,
a full-fledged system
REFERENCES
Barquin, R., & Edelstein, H. (Eds.). (1997). Planning and
designing the data warehouse. Upper Saddle River, NJ:
Prentice-Hall.
Bauer, A., Hmmer, W., Lehner, W., & Schlesinger, L.
(2002). A decathlon in multidimensional modeling: Open
issues and some solutions. DaWaK, 274-285.
Becker, S.A. (Ed.). (2002). Data warehousing and Web
engineering. Hershey, PA: IRM Press.
Berson, A., & Smith, S.J. (1997). Data warehousing, data
mining and olap. New York: McGraw-Hill.
830
Modeling Web-Based Data in a Data Warehouse
Bischoff, J., & Yevich, R. (1996). The superstore: Building
more than a data warehouse. Database Programming and
Design, 9(9), 220-229.
Chakrabarti, S. (2002). Mining the Web: Analysis of
hypertext and semi-structured data. New York: Morgan
Kaufmann.
Chakrabarti, S., van den Berg, M., & Dom, B. (2001).
Focused crawling: A new approach to topic-specific
Web resource discovery. Retrieved January, 2004, from
http://www8.org/w8-papers/5a-search-query/crawling/
Day, A. (2004). Data warehouses. American City and
County, 119(1), 18.
Devlin, B. (1998). Meta-data: The warehouse atlas. DB2
Magazine, 3(1), 8-9.
Goodman, A. (2000). Searching for a better way. Re-
trieved July 25, 2002, from http://www.traffick.com
Hackathorn, R. (1995). Data warehousing energizes your
enterprise. Datamation, 38-45.
Hackathorn, R. (1998). Web farming for the data ware-
house. New York: Morgan Kaufmann.
Hall, C. (1999). Enterprise information portals: Hot air or
hot technology [Electronic version]. Business Intelli-
gence Advisor, 111(11).
Imhoff, C., Galemmo, N., & Geiger, J.G. (2003). Mastering
data warehouse design: Relational and dimensional
techniques. New York: John Wiley and Sons.
Inmon, W.H. (2002). Building the data warehouse. New
York: John Wiley and Sons.
Inmon, W.H., Zachman, J.A., & Geiger, J.G. (1997). Data
stores, data warehousing, and the Zachman framework:
Managing enterprise knowledge. New York: McGraw-Hill.
Kimball, R. (1996). Dangerous preconceptions. Retrieved
June 13, 2002, from http://pwp.starnetinc.com/larryg/
index.html
Kimball, R., & Ross, M. (2002). The data warehouse
toolkit: The complete guide to dimensional modeling.
New York: John Wiley and Sons.
Lujan-Mora, S., & Trujillo, J. (2003). A comprehensive
method for data warehouse design. DMDW.
Marakas, G.M. (2003). Modern data warehousing, min-
ing, and visualization: Core concepts. Upper Saddle
River, NJ: Prentice-Hall College Division.
TEAM LinG
Modeling Web-Based Data in a Data Warehouse
Mattison, R. (1996). Data warehousing: Strategies, tech-
nologies and techniques. New York: McGraw-Hill.
McElreath, J. (1995). Data warehouses: An architectural
perspective. CA: Computer Sciences Corp.
Mihaila, G.A. (1996). WebSQLAn SQL-like query lan-
guage for the World Wide Web [masters thesis]. Univer-
sity of Toronto.
Parsaye, K. (1996). Data mines for data warehouses. Da-
tabase Programming and Design, 9(9).
Peralta, V., & Ruggia, R. (2003). Using design guidelines
to improve data warehouse logical design. DMDW.
Pfaffenberger, B. (1996). Web search strategies. MIS
Press.
Raab, D.M. (1999). Enterprise information portals [Elec-
tronic version]. Relationship Marketing Report.
Ray, E.J., Ray, D.S., & Seltzer, R. (1998). The Alta Vista
search revolution. CA: Osborne/McGraw-Hill.
Rob, P., & Coronel, C. (2002). Database systems: Design,
implementation, and management. New York: Thomson
Learning.
Rupley, S. (2000). From portals to vortals. PC Magazine.
Sander-Beuermann, F., & Schomburg, M. (1998). Internet
information retrievalThe further development of meta-
search engine technology. Proceedings of the Internet
Summit, Internet Society.
Schneider, M. (2003). Well-formed data warehouse struc-
tures, design and management of data warehouses 2003.
Proceedings of the 5th International Workshop
DMDW2003, Berlin, Germany.
Soudah, T. (2000). Search, and you shall find. Retrieved
July 17, 2002, from http://searchenginesguides.com
Spertus, E., & Stein, L.A. (1999). Squeal: A structured
query language for the Web. Retrieved August 10, 2002,
from http://www9.org/w9cdrom/222/222.html
Strehlo, K. (1996). Data warehousing: Avoid planned
obsolescence. Datamation, 38-45. M
Sullivan, D. (2000). Search engines review chart. Re-
trieved June 10, 2002, from http://searchenginewatch.com
Zghal, H.B., Faiz, S., & Ghezala, H.B (2003). Casme: A case
tool for spatial data marts design and generation, design
and management of data warehouses 2003. Proceedings
of the 5th International Workshop DMDW2003, Berlin,
Germany.
WEBSITES OF INTEREST
www.perl.com
www.cpan.com
www.xml.com
http://java.sun.com
http://semanticweb.org
KEY TERMS
Decision Support System (DSS): An interactive ar-
rangement of computerized tools tailored to retrieve and
display data regarding business problems and queries.
External Data: Data originating from other than the
operational systems of a corporation.
Granular Data: Data representing the lowest level of
detail that resides in the data warehouse.
Metadata: Data about data; in the data warehouse, it
describes the contents of the data warehouse.
Operational Data: Data used to support the daily
processing a company does.
Refresh Cycle: The frequency with which the data
warehouse is updated (e.g., once a week).
Transformation: The conversion of incoming data
into the desired form.
831
TEAM LinG
832
Moral Foundations of Data Mining
Kenneth W. Goodman
University of Miami, USA
INTRODUCTION
It has become a commonplace observation that scien-
tific progress often, if not usually, outstrips or precedes
the ethical analyses and tools that society increasingly
relies on and even demands. In the case of data mining
and knowledge discovery in databases, such an observa-
tion would be mistaken. There are, in fact, a number of
useful ethical precedents, strategies, and principles avail-
able to guide those who request, pay for, design, main-
tain, use, share, and sell databases used for mining and
knowledge discovery. These conceptual tools and the
need for them will vary as one is using a database to,
say, analyze cosmological data, identify potential cus-
tomers, or run a hospital. But these differences should
not be allowed to mask the ability of applied ethics to
provide practical guidance to those who work in an
exciting and rapidly growing new field.
BACKGROUND
Data mining is itself a hybrid discipline, embodying
aspects of computer science, artificial intelligence,
cryptography, statistics, and logic. In greater or lesser
degree, each of these disciplines has noted and ad-
dressed the ethical issues that arise in their practice. In
statistics, for instance, leading professional organiza-
tions have ratified codes of ethics that address issues
ranging from safeguarding privileged information and
avoiding conflicts of interest or sponsor bias (Interna-
tional Statistical Institute, 1985) to the avoidance of
any tendency to slant statistical work toward predeter-
mined outcomes (American Statistical Association,
1999).
It is in computer ethics, however, that one finds the
earliest, sustained, and most thoughtful literature
(Bynum, 1985; Johnson & Snapper, 1985; Ermann, Wil-
liams & Gutierrez, 1990; Forester & Morrison, 1994;
Johnson, 1994) in addition to ethics codes by profes-
sional societies (Association for Computing Machin-
ery, 1992; IEEE, 1990). Traditional issues in computer
ethics include privacy and workplace monitoring, hack-
ing, intellectual property, and appropriate uses and us-
ers. The intersection of computing and medicine has
also begun to attract interest (Goodman, 1998a).
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
What is clear about this landscape is that terms we
attach to issues privacy, for instance can mask
significant differences according as one uses a com-
puter to keep track of warehouse stock or arrest records
or real estate transactions or sexually transmitted dis-
eases. Moreover, ethical issues take on somewhat dif-
ferent aspects depending on whether a computer and its
data storage media are used by an individual, a business,
a university, or a government. Atop this is the general
purpose to which the machine is put: science, business,
law enforcement, public health, or national security.
This triad content, user, and purpose frames the
space in which ethical issues arise.
MAIN THRUST
One can identify a suite of ethical issues that arise in data
mining. All are tethered in one way or another to issues
encountered in computing, statistics, and kindred fields.
The question of whether data mining, or any discipline
for that matter, presents unique or unprecedented issues
is open to dispute. Issues between or among disciplines
often vary by degree more than by kind. If it is learned or
inferred from a database that Ms. Garcia prefers blue
frocks, it might be the case that her privacy has been
violated. But if it is learned or inferred that she has HIV,
the stakes are altogether different. The ability to make
ever-more-fine-grained inferences from very large data-
bases increases the importance of ethics in data mining.
Privacy, Confidentiality, and Consent
It is common to distinguish between privacy and confi-
dentiality by applying the former to humans claim or
desire to control access to themselves or information
about them, and the latter, more narrowly, to specific
units of that information. Privacy, if you will, is about
people; confidentiality is about information. Privacy is
broader, and it includes interest in information protec-
tion and control.
An important correlate of privacy is consent. One
cannot control information without being asked for
permission, or at least informed of information use. A
business that is creating a customer database might
collect data surreptitiously, arguably infringing on pri-
TEAM LinG
Moral Foundations of Data Mining
vacy. Or it might publicly thought not necessarily
individually disclose the collection. Such disclo-
sures are often key components of privacy policies.
Privacy policies sometimes seek permission in advance
to obtain and archive personal data or, more frequently,
disclose that data are being collected and then provide a
mechanism for individuals to opt out. The question
whether such opt-out policies are adequate to give indi-
viduals opportunities to control use of their data is
subject to widespread debate. In another context, a
government will collect data for vital statistics or public
health databases. Such uses, at least in democratic soci-
eties, may be justified on grounds of the implied con-
sent of those to whom the information applies and who
would benefit from its collection.
It is not clear how much or what kind of consent
would be necessary to provide ethical warrant for data
mining of personal information. The problem of ad-
equate consent is complicated by what may be hypoth-
esized to be widespread ignorance about data mining and
its capabilities. As elsewhere, some solutions to this
ethical problem might be identified or clarified by
empirical research related to public understanding of
data-mining technology, individuals preference for (lev-
els of) control over use of their information, and similar
considerations. The U.S. Department of Health and
Human Services has, for instance, supported research
on the process of informed consent for biomedical
research. Data mining warrants a kindred research pro-
gram. Among the issues to be clarified by such research
are the following:
To what extent do individuals want to control
access to their information?
What are the differences between consent to ac-
quire data for one purpose and consent for second-
ary uses of that data?
How do individual preferences or inclinations to
consent vary along with the data miner? (That is, it
may be hypothesized that some or many individu-
als will be sanguine about data mining by trusted
public health authorities, but opposed to data min-
ing by [certain] business entities or governments.)
It should be noted that many information exchanges
especially including those generally involving the
most sensitive or personal data are at least partly
governed by professionalism standards. Thus, doctor-
patient and lawyer- or accountant-client relationships
traditionally, if not legally, impose high standards for
the protection of information acquired during the course
of a professional relationship.
Appropriate Uses and Users of Data
Mining Technology M
It should be uncontroversial to point out that not all data
mining or knowledge discovery is done by appropriate
users, and not all uses enjoy equal moral warrant. A data-
mining police state may not be said to operate with the
same moral traction as a government public health ser-
vice in a democracy. (We may one day need to inquire
whether use of data-mining technology by a government
is itself grounds for identifying it as repressive.) Simi-
larly, given two businesses (insurance companies, say),
it is straightforward to report that the one using data-
mining technology to identify trends in accidents to
better offer advice about preventing accidents is on firm
moral footing, as opposed to one that identifies trends
in accidents to discriminate against minorities.
One way to carve the world of data mining is at the
public/private joint. Public uses are generally by gov-
ernments or their proxies, which can include universi-
ties and corporate contractors, and can employ data from
private sources (such as credit card information). Public
data mining can, at least in principle, claim to be in the
service of some collective good. The validity of such a
claim must be assessed and then weighed against damage
or threats to other public goods or values.
In the United States, the General Accounting Office,
a research and investigative branch of Congress, identi-
fied 199 federal data-mining projects and found that of
these, 54 mined private sector data, with 36 involving
personal information. There were 77 projects using data
from other federal agencies, and, of these, 46 involve
personal information from the private sector. The per-
sonal information, apparently used in these projects
without explicit consent, is said to include student loan
application data, bank account numbers, credit card
information, and taxpayer identification numbers. The
projects served a number of purposes, the top six of
which are given as improving service or performance
(65 projects), detecting fraud, waste and abuse (24),
analyzing scientific and research information (23),
managing human resources (17), detecting criminal
activities or patterns (15) and analyzing intelligence
and detecting terrorist activities (14) (General Ac-
counting Office, 2004).
Is losing confidentiality in credit card transactions a
fair exchange for improved government service? Re-
search? National security? These questions are the fo-
cus of sustained debate.
In the private sphere, data miners enjoy fewer oppor-
tunities to claim that their work will result in collective
benefit. The strongest warrant for private or for-profit
833
TEAM LinG

data mining is that corporations have fiduciary duties to
shareholders and investors, and these duties can only or
are best discharged by using data-mining technology. In
many jurisdictions, the use of personal information,
including health and credit data, is governed by laws that
include requirements for privacy policies and notices.
Because these often include opt-out provisions, it is
clear is that individuals must as a matter of strategy,
if not moral self-defense take some responsibility for
safeguarding their own information.
A further consideration is that for the foreseeable
future, most people will continue to have no idea of what
data mining is or what it is capable of. For this reason, the
standard bar for consent to use data must be set very high
indeed. It ought to include definitions of data mining and
must explicitly disclose that the technology is able to
infer or discover things about individuals that are not
obvious from the kinds of data that are willingly shared:
that databases can be linked; that data can be inaccurate;
that decisions based on this technology can affect credit
eligibility and employment; and so on (OLeary, 1995).
Indeed, the very point of data mining is the discovery of
trends, patterns and (putative) facts that are not obvious
or knowable on the surface. This must become a central
feature of privacy notice disclosure.
Further, all data-mining entities ought to develop or
adopt and then adhere to policies and guidelines govern-
ing their practice (OLeary, 1995).
Error and Uncertainty
Data analysis is probabilistic. Very large datasets are
buggy and contain inaccuracies. It follows that patterns,
trends, and other inferences based on algorithmic analy-
sis of databases are themselves open to challenge on
grounds of accuracy and reliability. Moreover, when
databases are proprietary, there are disincentives to op-
erate with standards for openness and transparency that
are morally required and increasingly demanded in
government, business, and science. What emerges is an
imperative to identify and follow best practices for data
acquisition, transmission, and analysis. Although some
errors are forgivable, those caused by sloppiness, low
standards, haste, or inattention can be blameworthy. When
the stakes are high, errors can cause extensive harm. It
would indeed be a shame to solve problems of privacy,
consent, and appropriate use only to fail to do the job one
set in the first place.
Perhaps the most interesting kind of data-mining
error is that which identifies or relies on illusory pat-
terns or flawed inferences. A business data-mining op-
eration that errs in predicting the blue frock market may
disappoint, but an error by data miners for a manned
space program, biomedical research project, public health
834
Moral Foundations of Data Mining
surveillance system, or national security agency could
be catastrophic. This means that good ethics requires
good science. That is, it is not adequate to collect and
analyze data for worthy purposes one must also seek
and stand by standards for doing so well.
One feature of data mining that makes this excep-
tionally difficult is its very status as a science. It is
customary in most empirical inquiries to frame and
then test hypotheses. Although philosophers of sci-
ence disagree about the best or most productive meth-
ods for conducting such inquiries, there is broad agree-
ment that experiments and tests generally offer more
rigor and produce more reliable results and insights
than (mere) observation of patterns between or among
variables. This is a well-known problem in domains in
which experiments are impossible or unethical. In epi-
demiology and public health, for instance, it is often
not possible to test a hypothesis by using the experi-
mental tools of control groups, double blinding, and
placebos (one could not intentionally attempt to give
people a disease by different means in order to identify
the most dangerous routes of transmission). For this
reason, epidemiologists and public health scientists
are usually mindful of the fact that their studies might
fail to reveal a causal relation, but rather, no more than
a statistical correlation.
Put differently, when there is no hypothesis to test,
it is not possible to know what will be found until it is
discovered (Fule & Roddick, 2004, p. 160).
There is a precedent for addressing this kind of
challenge. In meta-analysis, or the concatenation and
reanalysis of others results, statisticians endured with-
ering criticism of questionable methods, bias intro-
duced by initial data selection and over-broad conclu-
sions. They have responded by refining their tech-
niques and improving their processes. Meta-analysis
remains imperfect, but scientists in numerous disci-
plines (perhaps most noteworthily in the biomedical
sciences, where in some instances meta-analysis has
altered standards of patient care) have come to rely on
it (Goodman, 1998b).
FUTURE TRENDS
Data mining has, in a short time, become ubiquitous.
Like many hybrid sciences, it was never clear who
owned it and therefore who should assume responsi-
bility for its practice. Attention to data mining has
increased in part because of concern about its use by
governments and the extent to which such use will
infringe on civil liberties. This has, in turn, led to
renewed scrutiny, a positive development. It must, how-
ever, be emphasized that while issues related to privacy
TEAM LinG
Moral Foundations of Data Mining
and confidentiality are of fundamental concern, they are
by no means the only ethical issues raised by data mining
and, indeed, attention to them at the expense of the other
issues identified here would be a dangerous oversight.
The task of identifying ethical issues and proposing
solutions, approaches, and best (ethical) practices itself
requires additional research. It is even possible to evalu-
ate data minings ethical sensitivity, that is, the extent
to which rule generation itself takes ethical issues into
account (Fule & Roddick, 2004).
A broad-based research program will use the con-
tent, user, purpose triad to frame and test hypotheses
about the best ways to protect widely shared values; it
might be domain specific, emphasizing those areas that
raise the most interesting issues: business and econom-
ics, public health (including bioterrorism prepared-
ness), scientific (including biomedical) research, gov-
ernment operations (including national security), and so
on. Some issues/domains raise keenly important issues
and concerns. These include genetics and national secu-
rity surveillance.
Genetics and Genomics
The completion of the Human Genome Project in 2001
is correctly seen as a scientific watershed. Biology and
medicine will never be the same. Although it has been
recognized for some time that the genome sciences are
dependent on information technology, the ethical is-
sues and challenges raised by this symbiosis have only
been sketched (Goodman, 1996; Goodman, 1999). The
fact that vast amounts of human and other genetic infor-
mation can be digitized and analyzed for clinical, re-
search, and public health purposes has led to the cre-
ation of a variety of databases and, consequently, estab-
lishment of the era of data mining in genomics. It could
not be otherwise: [T]he amount of new information is
so great that data mining techniques are essential in
order to obtain knowledge from the experimental data
(Larraaga, Menasalvas, & Pea, 2004).
The ethical issues that arise at the intersection of
genetics and data mining should be anticipated: privacy
and confidentiality, appropriate uses and users, and
accuracy and uncertainty. Each of these embodies spe-
cial features. Confidentiality of genetic information
applies not only to the source of genetic material but
also, in one degree or another, to his or her relatives.
The concepts of appropriate use and user are strained
given extensive collaborations between and among
corporate, academic, and government entities. And chal-
lenges of accuracy and uncertainty are magnified in case
the results of data mining are applied in medical practice.
National Security Surveillance
M
Although it would be blameworthy for vulnerable soci-
eties to fail to use all appropriate tools at their disposal
to prevent terrorist attacks, data-mining technology
poses new and interesting challenges to the concept of
appropriate tool. There are concerns at the outset that
data acquisition for whatever purpose may be
inappropriate. This is true whether police, military, or
other government officials are collecting information
on notecards or in huge, machine-tractable databases.
Knowledge discovery ups the ante considerably.
The privacy rights of citizens in a democracy are not
customarily bartered for other benefits, such as secu-
rity. The rights are what constitute the democracy in the
first place. On the other hand, rights are not absolute,
and there might be occasions on which morality permits
or even requires infringements. Moreover, when a data-
base is created, appropriate use and error/uncertainty
become linked: appropriate use includes (at least tac-
itly) the notion that such use will achieve its specified
ends. To the degree that error propagation or uncer-
tainty can impeach data-mining results, the use itself is
correspondingly impeached.
CONCLUSION
The intersection of computing and ethics has been a
fertile ground for scholars and policymakers. The rapid
growth and extraordinary power of data mining and
knowledge discovery in databases bids fair to be a
source of new and interesting ethical challenges.
Precedents in kindred domains, such as health
informatics, offer a platform on which to begin to
prepare the kind of thoughtful and useful conceptual
tools that will be required to meet those challenges.
By identifying the key nodes of content, user, and
purpose, and the leading ethical issues of privacy/con-
fidentiality, appropriate use(r), and error/uncertainty,
we can enjoy some optimism that the future of data
mining will be guided by thoughtful rules and policies
and not the narrower (but not invalid) interests of entre-
preneurs, police, or scientists who hesitate or fail to
weigh the goals of their inquiries against the needs and
expectations of the societies that support their work.
REFERENCES
American Statistical Association. (1999). Ethical guide-
lines for statistical practice. Retrieved from http://
835
TEAM LinG

www.amstat.org/profession/index.cfm?fus
eaction=ethicalstatistics
Association for Computing Machinery. (1992). Code of
ethics and professional conduct. Retrieved from http://
www.acm.org/constitution/code.html
Bynum, T. W. (Ed.). (1985). Computers and ethics. New
York: Blackwell.
Ermann, M. D., Williams, M. B., & Gutierrez, C. (Eds.).
(1990). Computers, ethics, and society. New York:
Oxford University Press.
Forester, T., & Morrison, P. (1994). Computer ethics:
Cautionary tales and ethical dilemmas in computing
(2nd ed.). Cambridge, MA: MIT Press.
Fule, P., & Roddick, J. F. (2004). Detecting privacy and
ethical sensitivity in data mining results. In V. Estivill-
Castro (Ed.), Proceedings of the 27th Australasian
Computer Science Conference (pp. 159-166).
General Accounting Office. (2004). Data mining: Federal
efforts cover a wide range of uses (Rep. No. GAO-04-548).
Washington, DC: U.S. General Accounting Office. Re-
trieved from http://www.gao.gov/new.items/d04548.pdf
Goodman, K. W. (1996). Ethics, genomics, and informa-
tion retrieval. Computers in Biology and Medicine, 26,
223-229.
Goodman, K. W. (Ed.). (1998a). Ethics, computing and
medicine: Informatics and the transformation of health
care. Cambridge, MA: Cambridge University Press.
Goodman, K. W. (1998b). Meta-analysis: Conceptual,
ethical and policy issues. In K. W. Goodman (Ed.),
Ethics, computing and medicine: Informatics and the
transformation of health care (pp. 139-167). Cambridge,
MA: Cambridge University Press.
Goodman, K. W. (1999). Bioinformatics: Challenges revis-
ited. MD Computing, 16(3),17-20.
IEEE. (1990). IEEE code of ethics. Retrieved from http://
www.ieee.org/portal/index.jsp?pageID= corp_level1&
path=about/whatis& file=code.xml&xsl=generic.xsl
International Statistical Institute. (1985). Declaration on
professional ethics. Retrieved from http://www.cbs.nl/
isi/ethics.htm
836
Moral Foundations of Data Mining
Johnson, D. O. (1994). Computer ethics. Englewood Cliffs,
NJ: Prentice Hall.
Johnson, D. O., & Snapper, J. W. (Eds.). (1985). Ethical
issues in the use of computers. Belmont, CA:
Wadsworth.
Larraaga, P., Menasalvas, E., & Pea, J. M. (2004).
Data mining in genomics and proteomics. Artificial
Intelligence in Medicine, 31, iii-iv.
OLeary, D. E. (1995, April). Some privacy issues in knowl-
edge discovery: The OECD personal privacy guidelines.
IEEE Expert, 48-52.
KEY TERMS
Applied Ethics: The branch of ethics that empha-
sizes not theories of morality but ways of analyzing and
resolving issues and conflicts in daily life, the profes-
sions and public affairs.
Codes of Ethics: More or less detailed sets of
principles, standards, and rules aimed at guiding the
behavior of groups, usually of professionals in busi-
ness, government, and the sciences.
Confidentiality: The claim, right, or desire that
personal information about individuals should be kept
secret or not disclosed without permission or informed
consent.
Ethics: The branch of philosophy that concerns
itself with the study of morality.
Informed (or Valid) Consent: The process by which
people make decisions based on adequate information,
voluntariness, and capacity to understand and appreciate
the consequences of whatever is being consented to.
Morality: A common or collective set of rules,
standards, and principles to guide behavior and by which
individuals and society can gauge an action as being
blameworthy or praiseworthy.
Privacy: Humans claim or desire to control access
to themselves or information about them; privacy is
broader than confidentiality, and it includes interest in
information protection and control.
TEAM LinG

Mosaic-Based Relevance Feedback for Image
Retrieval
Odej Kao
University of Paderborn, Germany
Ingo la Tendresse
Technical University of Clausthal, Germany
INTRODUCTION
A standard approach for content-based image retrieval
(CBIR) is based on the extraction and comparison of
features usually related to dominant colours, shapes,
textures and layout (Del Bimbo, 1999). These features
are a-priori defined and extracted, when the image is
inserted into the database. At query time the user sub-
mits a similar sample image (query-by-sample-image)
or draws a sketch (query-by-sketch) of the sought
archived image. The similarity degree of the current
query image and the target images is determined by
calculation of a multidimensional distance between the
corresponding features. The computed similarity values
allow the creation of an image ranking, where the first k,
usually k=32 or k=64, images are considered retrieval
hits. These are chained in a list called ranking and then
presented to the user. Each of these images can be used
as a starting point for a refined search in order to
improve the obtained results.
The assessment of the retrieval result is based on a
subjective evaluation of whole images and their position
in the ranking. An important disadvantage of the re-
trieval with content-based features and the presentation
of the resulting images as ranking is that the user is
usually not aware, why certain images are shown on the
top positions and why certain images are ranked low or
not presented at all. Furthermore, users are also inter-
ested which sketch properties are decisive for the con-
sideration and rejection of the images, respectively. In
case of primitive features like colour these questions
can be often answered intuitively. Retrieval with com-
plex features considering for example texture and lay-
out creates rankings, where the similarity between the
query and the target images is not always obvious. Thus,
the user is not satisfied with the displayed results and
would like to improve the query, but it is not clear to
him/her, which parts of the querying sketch or the
sample image should be modified and improved accord-
ing to the desired targets. Therefore, a suitable feedback
mechanism is necessary.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
837
M
BACKGROUND
Relevance feedback techniques are often used in image
databases in order to gain additional information about
the sought image set (Rui, Huang, & Mehrotra, 1998).
The user evaluates the retrieval results and selects for
example positive and negative instances, thus in the
subsequent retrieval steps the search parameters are
optimised and the corresponding images/features are
supplied with higher weights (Mller & Pun, 2000; Rui
& Huang, 2000). Moreover, additional query images can
be considered and allow a more detailed specification
of the target image (Baudisch, 2001). However, the
more complex is the learning model, the more difficult
is the analysis and the evaluation of the retrieval results.
Users in particular those with limited expertise in
image processing and retrieval are not able to detect
misleading areas in the image/sketch with respect to the
applied retrieval algorithms and to modify the sample
image/sketch appropriately. Consequently, an iterative
search is often reduced to a random process of param-
eter optimisation.
The consideration of the existing user knowledge is the
main objective of many feedback techniques. In case of
user profiling (Cox, Miller, Minka, Papathomas, & Yianilos,
2000) the previously completed retrieval sessions are
analysed in order to obtain additional information about
users preferences. Furthermore, the selection actions
during the current session are monitored and images simi-
lar to those are given higher weights. A hierarchical ap-
proach named multi-feedback ranking separates the query-
ing image in several regions and allows a more detailed
search (Mirmehd & Perissamy, 2001).
Techniques for permanent feedback guide the user
through the entire retrieval process. An example for this
approach is implemented by the system Image Retro:
based on a selected sample image a number of images
are removed from the possible image set. By analysing
these images, the user develops an intuition about prom-
ising starting points (Vendrig, Worring, & Smeulders,
2001). In case of the fast feedback, the user receives the
TEAM LinG

Figure 1. Compilation of a mosaic-based ranking
current ranking after every modification of the query
sketch. The correspondence between the presented rank-
ing and the last modification helps the user to identify
sketch improvements leading in the desired retrieval
directions and to undo inappropriate sketch drawings
immediately (Veltkamp, Tanase, & Sent, 2001).
FEEDBACK METHOD FOR QUERY-
BY-SKETCH
This section describes a feedback method, which helps
the user to improve the query sketch and to receive the
desired results. Subsequently, the query sketch is sepa-
rated into regions and each area is compared with the
corresponding subsection of all images in the database.
The most similar sections are grouped in a mosaic, thus
the user can detect well-defined and misleading areas of
his/her query sketch quickly. The creation of a mosaic
consisting of k areas with the most similar sections is
shown in Figure 1.
The corresponding pseudo-code algorithm for mo-
saic-based retrieval and ranking presentation is given in
Algorithm 1.
The criterion cm for the similarity computation of the
corresponding sections can be identical with the main
retrieval criterion (cm = c) or be adapted to specific
Algorithm 1. Pseudo code of the mosaic build-up
procedure mosaic(sketch s, ranking R, criterion c m , grid g)
begin
divide s using a uniform grid g into k fields sk
initialise similarity[1 k] = 0, image[1 k]= none
for each image i R Figure 3. Mosaic based on an adaptive grid
begin
scale i to i* with dim[i*] = dim[s]
divide i* using grid g into k fields ik
for j=[1 k] do
if similarity[j] < cm(sj, ij)
then similarity[j] = cm(sj, ij), image[j] = ij
od
end
mosaic M = j=1k image[j]
end
838
Mosaic-Based Relevance Feedback for Image Retrieval
Figure 2. An example for a mosaic feedback
images or users (cm c). The size of the mosaic sections
can also be selected by the user. Here grids consisting of
1616 and 3232 pixel blocks are evaluated.
An example for the resulting mosaics is found in
Figure 2. On the left hand side the sought image is
shown, which was approximated by the user-defined
sketch in the middle. On the right hand side the resulting
1616 mosaic containing the best section hits of all images
in the databases is presented. A manual post-processing
of the mosaic clarifies the significance of individual mo-
saic blocks: Those sections, which are parts of the sought
image, are framed with a white box. Furthermore, sections
from images of the same category in this case other
pictures with animals are marked with white circles.
An extension of the fixed grids is realised by adaptive
mosaics, where neighbouring sections depending on a
given similarity threshold are merged into larger, homo-
geneous areas, thus the user can evaluate individual
sections more easily. Figure 3 shows a sample image and
the corresponding mosaic-based ranking, which is gained
using an adaptive grid.
In these manually post-processed cases additional
information is provided to the user, which sketch re-
gions are already sufficiently prepared (boxes) and which
regions have the right direction but still need minor
corrections (circles). Finally, all other regions do not
satisfy the similarity criterions and have to be re-drawn.
With this information the user can focus on the most
promising areas and modify these in a suitable manner.
For the usage of this feedback method in real-world
applications it is necessary to evaluate, whether typical
users are able to detect suitable and misleading regions
intuitively and thus to improve the original sketch.
For the performance evaluation of the developed mo-
saic technique and measuring the retrieval quality we
TEAM LinG
Mosaic-Based Relevance Feedback for Image Retrieval
created a test image database with pictures of various
categories and selected a wavelet-based feature for the
retrieval (Kao & Joubert, 2001). Wavelet-based features
have in common, that the semantic interpretation of the
extracted coefficients and thus the produced rankings is
rather difficult, in particular for users without signifi-
cant background in image processing. Nevertheless, such
features perform well in general image sets and are
applied in a variety of systems (Jacobs, Finkelstein, &
Salesin, 1995). The application of the mosaic-based
feedback could improve the usability of these powerful
features by helping the user to find and to modify the
decisive image regions.
The experiments considered fine-grained mosaics
with a 1616 grid, a 3232 grid, and adaptive grids. Simul-
taneously, a traditional ranking for the current query was
created, which contained the best k images for the given
sketch and the selected similarity criterion. Both rankings
were subsequently distributed to 16 users, who analysed
those and based on the derived knowledge improved
the sketch appropriately.
The obtained results were evaluated manually in or-
der to resolve following basic questions related to pos-
sible applications of the mosaic feedback:
Can a mosaic be used as a sole presentation form
for image ranking and thus replace the traditional
list/table-based ranking?
Are users able to recognise, which parts of the
mosaic represent sections of relevant images, and
which parts should be modified?
Can we expect a significant improvement of the
sketch quality, if the relevant sections are cor-
rectly marked/recognised?
A query of an image database is considered success-
ful, if the sought image is returned within the first k
retrieval hits. Analogously, in case of mosaic-based
result presentation, image retrieval is evaluated as suc-
cessful, if at least one of the mosaic blocks is a part of
the sought image. An analysis of the produced rankings
shows, that the recall rate improved dependent on the
used colour space up to 32.75% (la Tendresse & Kao,
2003; Skubowius, 2002). However, it is not clear, whether
the user will be able to establish a relation between the
presented section and the image he/she is looking for.
Therefore, we asked 16 test persons to study a number of
images and the corresponding mosaics. The similarity of
the images to the given sketch was not obvious, as the
images were found during the mosaic-based retrieval,
but not in case of the traditional retrieval. The test
persons had 180 seconds time to mark all mosaic sec-
tions, which in their opinion match an original image
section. They found approximately 65% of the relevant
sections per mosaic and needed on average 63 seconds
study time.
In order to evaluate the mosaic as a feedback tech-
M
nique we selected 26 sketches, which did not lead to the
desired images during the traditional and mosaic-based
retrieval and asked 16 test persons to improve the
quality of the sketches. Eight test persons received the
target image, the old sketch as well as the correspond-
ing mosaic and had 300 seconds to correct the colours,
shapes, layout, etc. The other eight persons should
improve the sketch without the mosaic feedback. All
modified sketches were submitted to the CBIR-system
and produced following results: 46.59% of all sketches
changed according to the mosaic feedback led to a
successful traditional retrieval, i.e. the sought image
was returned within the 16 most similar images. In
opposite only 34.24% of the sketches modified with-
out mosaic feedback enabled a successful retrieval. If
a mosaic-based ranking was presented, then the recall
quote increased up to 68.75% (feedback allowed) and
up to 54.35% otherwise. Moreover, the desired image
is placed on average two positions above the ranking
position prior modification.
FUTURE TRENDS
The introduction of novel and powerful methods for
relevance feedback in image, video and audio databases
is an emerging and fast on-going process. Improve-
ments related to integration of semantic information/
combination of multiple attributes in the query (Heesch
& Rger, 2003; Santini, Gupta, & Jain, 2001), auto-
matic detection/modification of misleading areas, 3D
result visualisation techniques beyond the simple rank-
ing list (Cinque, Levialdi, Malizia, & Olsen, 1998),
queries considering multiple instances (Santini & Jain,
2000) and many more are investigated in number of
research groups.
The presented mosaic-based feedback technique
easies the modification process by showing significant
sketch regions to the user. A re-definition of the entire
sketch is however time-consuming and usually avoided
by the users. Instead, minor modifications are applied
to the sketch and another query run is started. By taking
this into account, the feedback process and more
decisive the next ranking can be improved, if the user
is guided to the modification of those regions, which
exhibit the largest chances of success. Thus, the au-
tonomous detection of misleading sketch sections and
system-based suggestions for modification are most
desirable in the short-term development focus. For this
purpose the convergence behaviour of individual sketch
sections towards the desired target is analysed. All results
839
TEAM LinG

can be combined in a so-called convergence map, which is
subsequently processed by a number of rules. Based on the
output of the analysis process, several actions can be
proposed. For example, if the given sketch is completely on
the wrong route, the user is asked to re-draw the sketch
entirely. If only parts of the sketch are misleading, the user
should be asked to pay more attention to those. Finally,
even some of the necessary modifications can also be
performed/proposed automatically, for example if intuitive
features such as colours are used for the retrieval. In this
case the user is advised to use more red or green colour in
some sections or to change the average illumination of the
sketch.
An additional visual help for the user during creation
of the query sketch is given by a permanent evaluation of
the archived images and presentation of the most prom-
ising images in a three-dimensional space. The current
sketch builds the centre of this 3D-space and the archived
images are arranged according to their similarity dis-
tance around the sketch. The nearest images are the most
similar images, thus the user can copy some aspects of
those images in order to move the retrieval in the
desired direction. After each new line the distance to all
images is re-computed and the images in the 3D-space
are re-arranged. Thus, the user can see the effects of the
performed modification immediately and proceed in
the same direction or remove the latest modifications
and try another path.
CONCLUSIONS
This article described a mosaic-based feedback method,
which helps image database users to recognise and to
modify misleading areas in the querying sketch and to
improve the retrieval results. Performance measure-
ments showed that the proposed method easies the
querying process and allows the users to modify their
sketches in a suitable manner.
REFERENCES
Baudisch, P. (2001). Dynamic information Filtering. PhD
Thesis, GMD Research Series 2001, No. 16. GMD
Forschungszentrum.
Cinque, L., Levialdi, S., Malizia, A., & Olsen, K.A. (1998).
A multi-dimensional image browser. Journal of Visual
Languages and Computing, 9(1), 103-117.
Cox, I.J., Miller, M.L., Minka, T.P., Papathomas, T., &
Yianilos, P.N. (2000). The bayesian image retrieval sys-
840
Mosaic-Based Relevance Feedback for Image Retrieval
tem, pichunter: Theory, implementation and psychophysi-
cal experiments. In IEEE Transactions on Image Process-
ing, 9(1), 20-37.
Del Bimbo, A. (1999). Visual information retrieval. Mor-
gan Kaufmann Publishers.
Heesch, D., & Rger, S. M. (2003). Relevance feedback for
content-based image retrieval: What can three mouse
clicks achieve? In Advances in Information Retrieval,
LNCS 2633 (pp. 363-376).
Jacobs, C.E., Finkelstein, A., & Salesin, D.H. (1995). Fast
multiresolution image querying. In Proceedings of ACM
Siggraph (pp. 277-286).
Kao, O., & Joubert, G.R. (2001). Efficient dynamic image
retrieval using the trous wavelet transformation. In
Advances in Multimedia Information Processing, LNCS
2195 (pp. 343-350).
La Tendresse, I., & Kao, O. (2003). Mosaic-based sketch-
ing interface for image databases. Journal of Visual
Languages and Computing 14(3), 275-293.
La Tendresse, I., Kao, O., & Skubowius, M. (2002). Mosaic
feedback for sketch training and retrieval improvement. In
Proceedings of the IEEE Conference on Multimedia and
Expo (ICME 2002) (pp. 437-440).
Mirmehdi M., & Perissamy, R. (2001). CBIR with percep-
tual region features. In Proceedings of the 12th British
Machine Vision Conference (pp. 51-520).
Mller, H., Mller, W., Squire, D., Marchand-Maillet, S.,
& Pun, T. (2000). Learning features weights from user
behavior in content-based image retrieval. In Proceed-
ings of ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (Workshop on
Multimedia Data Mining MDM/KDD2000).
Rui, Y., & Huang, T.S. (2000). Optimizing learning in image
retrieval. In Proceeding of IEEE International Confer-
ence on Computer Vision and Pattern Recognition (pp.
236-245).
Rui, Y., Huang, T.S., & Mehrotra, S. (1998). Relevance
feedback techniques in interactive content-based Image
retrieval. In Proceedings of SPIE 3312 (pp. 25-36).
Santini, S., Gupta, A., & Jain, R. (2001). Emergent seman-
tics through interaction in image databases. IEEE Trans-
actions on Knowledge Data Engineering, 13(3), 337-351.
Santini, S., & Jain, R. (2000). Integrated browsing and
querying for image databases. IEEE MultiMedia, 7(3), 26-
39.
TEAM LinG
Mosaic-Based Relevance Feedback for Image Retrieval
Veltkamp, R.C., Tanase, M., & Sent, D. (2001). Features in
content-based image retrieval systems: A survey. Kluwer
Academic Publishers.
Vendrig, J., Worring, M., & Smeulders, A.W.M. (2001).
Filter image browsing: Interactive image retrieval by using
database overviews. Multimedia Tools and Applica-
tions, 15(1), 83-103.
KEY TERMS
Content-Based Image Retrieval: Search for suit-
able image in a database by comparing extracted fea-
tures related to color, shape, layout and other specific
image characteristics.
Feature Vector: Data that describes the content of
the corresponding image. The elements of the feature
vector represent the extracted descriptive information
with respect to the utilised analysis.
Query By Sketch, Query by Painting: A widespread
query type where the user is not able to present a similar
sample image. Therefore the user sketches the looked-for
image with a few drawing tools. It is not necessary to do M
this correctly in all aspects.
Query-By-Pictorial-Example: The query is formu-
lated by using a user-provided example image for the
desire retrieval. Both query and stored image are analysed
in the same way.
Ranking: List of the most similar images extracted
from the database according to the querying sketch/
image or other user-defined criteria. The ranking dis-
plays the retrieval results to the user.
Relevance Feedback: The user evaluates the qual-
ity of the individual items in the ranking based on the
subjective expectations / specification of the sought
item. Subsequently, the user supplies the system with
the evaluation results (positive/negative instances,
weights ) and re-starts the query. The system consid-
ers the user-knowledge while computing the similarity.
Similarity: Correspondence of two images. The simi-
larity is determined by comparing the extracted feature
vectors, for example by a metric or distance function.
841
TEAM LinG
842
Multimodal Analysis in Multimedia Using
Symbolic Kernels
Hrishikesh B. Aradhye
SRI International, USA
Chitra Dorai
IBM T. J. Watson Research Center, USA
INTRODUCTION
The rapid adoption of broadband communications tech-
nology, coupled with ever-increasing capacity-to-price
ratios for data storage, has made multimedia information
increasingly more pervasive and accessible for consum-
ers. As a result, the sheer volume of multimedia data
available has exploded on the Internet in the past decade
in the form of Web casts, broadcast programs, and stream-
ing audio and video. However, indexing, search, and
retrieval of this multimedia data is still dependent on
manual, text-based tagging (e.g., in the form of a file name
of a video clip). However, manual tagging of media con-
tent is often bedeviled by an inadequate choice of key-
words, incomplete and inconsistent terms used, and the
subjective biases of the annotator introduced in his or her
descriptions of content adversely affecting accuracy in
the search and retrieval phase. Moreover, manual anno-
tation is extremely time-consuming, expensive, and
unscalable in the face of ever-growing digital video col-
lections. Therefore, as multimedia get richer in content,
become more complex in format and resolution, and grow
in volume, the urgency of developing automated content
analysis tools for indexing and retrieval of multimedia
becomes easily apparent.
Recent research towards content annotation, struc-
turing, and search of digital media has led to a large
collection of low-level feature extractors, such as face
detectors and recognizers, videotext extractors, speech
and speaker identifiers, people/vehicle trackers, and event
locators. Such analyses are increasingly processing both
visual and aural elements to result in large sets of
multimodal features. For example, the results of these
multimedia feature extractors can be
Real-valued, such as shot motion magnitude, audio
signal energy, trajectories of tracked entities, and
scene tempo
Discrete or integer-valued, such as the number of
faces detected in a video frame and existence of a
scene boundary (yes/no)
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Ordinal, such as shot rhythm, which exhibits partial
neighborhood properties (e.g., metric, accelerated,
decelerated)
Nominal, such as identity of a recognized face in a
frame and text recognized from a superimposed
caption1
Multimedia metadata based on such a multimodal
collection of features pose significant difficulties to sub-
sequent tasks such as classification, clustering, visual-
ization, and dimensionality reduction all which tradi-
tionally deal with only continuous-valued data. Common
data-mining algorithms employed for these tasks, such as
Neural Networks and Principal Component Analysis
(PCA), often assume a Euclidean distance metric, which is
appropriate only for real-valued data. In the past, these
algorithms could be applied to symbolic domains only
after representing the symbolic labels as integers or real
values or to a feature space transformation to map each
symbolic feature as multiple binary features. These data
transformations are artificial. Moreover, the original fea-
ture space may not reflect the continuity and neighbor-
hood imposed by the integer/real representation.
This paper discusses mechanisms that extend tasks
traditionally limited to continuous-valued feature spaces,
such as (a) dimensionality reduction, (b) de-noising, (c)
visualization, and (d) clustering, to multimodal multimedia
domains with symbolic and continuous-valued features.
To this end, we present four kernel functions based on
well-known distance metrics that are applicable to each of
the four feature types. These functions effectively define
a linear or nonlinear dot product of real or symbolic feature
vectors and therefore fit within the generic framework of
kernel space machines. The framework of kernel functions
and kernel space machines provides classification tech-
niques that are less susceptible to overfitting when com-
pared with several data-driven learning-based classifiers.
We illustrate the usefulness of such symbolic kernels
within the context of Kernel PCA and Support Vector
Machines (SVMs), particularly in temporal clustering and
tracking of videotext in multimedia. We show that such
TEAM LinG
Multimodal Analysis in Multimedia Using Symbolic Kernels
analyses help capture information from symbolic feature
spaces, visualize symbolic data, and aid tasks such as
classification and clustering and therefore are eminently
useful in multimodal analysis of multimedia.
BACKGROUND
Early approaches to multimedia content analysis dealt
with multimodal feature data in two primary ways. Either
a learning technique such as Neural Nets was used to find
patterns in the multimodal data after mapping symbolic
values into integers, or the multimodal features were
segregated into different groups according to their modes
of origin (e.g., into audio and video features), processed
separately, and the results from the separate processes
were merged by using some probabilistic mechanism or
evidence combination method. The first set of methods
implicitly assumed the Euclidean distance as an underly-
ing metric between feature vectors. Although this may be
appropriate for real-valued data, it imposes a neighbor-
hood property on symbolic data that is artificial and is
often inappropriate. The second set of methods essen-
tially dealt with each category of multimodal data sepa-
rately and fused the results. They were thus incapable of
leading to novel patterns that can arise if the data were
treated together as a whole. As audiovisual collections of
today provide multimodal information, they need to be
examined and interpreted together, not separately, to
make sense of the composite message (Bradley, Fayyad,
& Mangasarian, 1998).
Recent advances in machine-learning and data analy-
sis techniques, however, have enabled more sophisti-
cated means of data analyses. Several researchers have
attempted to generalize the existing PCA-based frame-
work. For instance, Tipping (1999) presented a probabilis-
tic latent-variable framework for data visualization of
binary and discrete data types. Collins and co-workers
(Collins, Dasgupta, & Schapire, 2001) generalized the
basic PCA framework, which inherently assumes Gaussian
features and noise, to other members of the exponential
family of functions. In addition to these research efforts,
Kernel PCA (KPCA) has emerged as a new data represen-
tation and analysis method that extends the capabilities
of the classical PCA which is traditionally restricted to
linear feature spaces to feature spaces that may be
nonlinearly correlated (Scholkopf, Smola, & Muller, 1999).
In this method, the input vectors are implicitly projected
on a high-dimensional space by using a nonlinear map-
ping. Standard PCA is then applied to this high-dimen-
sional space. KPCA avoids explicit calculation of high-
dimensional projections with the use of kernel functions,
such as radial basis functions (RBF), high-degree polyno-
mials, or the sigmoid function. KPCA has been success-
fully used to capture important information from large,
nonlinear feature spaces into a smaller set of principal M
components (Scholkopf et al., 1999). Operations such as
clustering or classification can then be carried out in this
reduced dimensional space. Because noise is eliminated
as projections on eigen-vectors with low eigen-values,
the final reduced space of larger principal components
contains less noise and yields better results with further
data analysis tasks such as classification.
Although many conventional methods have been
previously developed for extraction of principal compo-
nents from nonlinearly correlated data, none allowed for
generalization of the concepts to dimensionality reduc-
tion of symbolic spaces. The kernel-space representation
of KPCA presents such an opportunity. However, since
its inception, applications of KPCA have been primarily
limited to domains with real-valued, nonlinearly corre-
lated features despite the recent literature on defining
kernels over several discrete objects such as sequences,
trees, graphs, as well as many other types of objects.
Moreover, recent techniques like the Fisher kernel ap-
proach by Jaakkola and Haussler (1999) can be used to
systematically derive kernels from generative models,
which have been demonstrated quite successfully in the
rich symbolic feature domain of bioinformatics. Against
the backdrop of these emerging collections of research,
the work presented in this paper uses the ideas of Kernel
PCA and symbolic kernel functions to investigate the yet
unexplored problem of symbolic domain principal compo-
nent extraction in the context of multimedia. The kernels
used here are designed based on well-known distance
metrics, namely Hamming distance, Cityblock distance,
and the Edit distance metric, and have been previously
used for string comparisons in several domains, including
gene sequencing.
With these and other symbolic kernels, multimodal
data from multimedia analysis containing real and sym-
bolic values can be handled in a uniform fashion by using,
say, an SVM classifier employing a kernel function that is
a combination of Euclidean, Hamming, and Edit Distance
kernels. Applications of the proposed kernel functions to
temporal analysis of videotext data demonstrate the util-
ity of this approach.
MAIN THRUST
Distance Kernels for Multimodal Data
Kernel-based classifiers such as SVMs and Neural Net-
works use linear, Radial Basis Function (RBF), or polyno-
mial functions as kernels that first (implicitly) transform
input data into a higher dimensional feature space and
then process them in this space. Many of the common
843
TEAM LinG
 Multimodal Analysis in Multimedia Using Symbolic Kernels
kernels assume a Euclidean distance to compare feature
vectors. Symbolic kernels, in contrast, have been less
commonly used in the published literature. We use the
following distance-based kernel functions in our analysis.
1. Linear (Euclidean) Kernel Function for Real-valued
Features: This is the most commonly used kernel
function for linear SVMs and other kernel-based
algorithms. Let x and z be two feature vectors. Then
the function
K l (x, z ) = x T z (1)
defines a Euclidean distance-based kernel function.
KPCA with linear kernel function reduces to the
standard PCA. The linear kernel trivially follows
Mercers condition for kernel validity, that is, the
matrix comprising pairwise kernel function values for
any finite subset of feature vectors selected from the
feature space is guaranteed to be positive
semidefinite.
2. Hamming Kernel Function for Nominal Features:
Let the number of features be N. Let x and z be two
feature vectors, that is, N-dimensional symbolic vec-
tors from a finite, symbolic feature space XN. Then the
function
N
K h (x, z ) = N (xi , z i ) (2)
i =1
where (xi , z i ) = 0 if xi = z i and 1 otherwise, defines
a Hamming distancebased kernel function and fol-
lows the Mercers condition for kernel validity
(Aradhye & Dorai, 2002). The equality, xi = z i , refers
to symbol/label match.
3. Cityblock Kernel Function for Discrete Features:
Using the preceding notation, the function
N
K c (x, z ) = M i xi z i (3)
i =1
where the ith feature is Mi-ary, defines a Cityblock
distancebased kernel function and follows the
Mercers condition for kernel validity (Aradhye &
Dorai, 2002).
4. Edit Kernel Function for Stringlike Features: We
define an edit kernel between two strings as
K e (x, z ) = max(len(x ), len(z )) E (x, z ) (4)
where E (x, z ) is the edit distance between the two
strings, defined conventionally as the minimum num-
844
ber of change, delete, and insert operations re-
quired to convert one string into another, and
len(x ) is the length of string x. In theory, Edit
distance does not obey Mercer validity, as has
been recently proved by Cortes and coworkers
(Cortes, Haffner, & Mohri, 2002, 2003). However,
empirically, the Kernel matrices generated by the
edit kernel are often positive definite, justifying the
practical use of Edit distancebased kernels.
Hybrid Multimodal Kernel
Having defined these four basic kernel functions for
different modalities of features, we are now in a position
to define a multimodal kernel function that encompasses
all types of common multimedia features. Let any given
feature vector x be comprised of a real-valued feature set
x r , a nominal feature set xh , a discrete-valued feature set
xc , and a string-style feature xe , such that
x = [x r xh xc x e ]. Then, because a linear combi-
nation of valid kernel functions is a valid kernel function,
we define
K m (x, z ) = K l (xr , z r ) + K h (xh , z h ) + K c (xc , z c ) + K l (xe , z e )
(5)
where K m (x, z ) is our multimodal kernel and , , , and
are constants. Such a hybrid kernel can now be
seamlessly used to analyze multimodal feature vectors
that have real and symbolic values, without imposing
any artificial integer mapping of symbolic labels and
further obtaining the benefits of analyzing disparate data
together as one. The constants , , , and can be
determined in practice either by a knowledge-based analy-
sis of the relative importance of the different types of
features or by empirical optimization.
Example Multimedia Application:
Videotext Postprocessing
Video sequences contain a rich combination of images,
sound, motion, and text. Videotext, which refers to super-
imposed text on images and video frames, serves as an
important source of semantic information in video
streams, besides speech, close caption, and visual con-
tent in video. Recognizing text superimposed on video
frames yields important information such as the identity
of the speaker, his/her location, topic under discussion,
sports scores, product names, associated shopping data,
and so forth, allowing for automated content description,
TEAM LinG
Multimodal Analysis in Multimedia Using Symbolic Kernels
search, event monitoring, and video program categoriza-
tion. Therefore, a videotext based Multimedia Description
Scheme has recently been adopted into the MPEG-7 ISO/
IEC standard to facilitate media content description
(Dimitrova, Agnihotri, Dorai, & Bolle, 2000). To this end,
videotext extraction and recognition is an important task
of an automated video content analysis system. Figure 1
shows three illustrative frames containing videotext taken
from MPEG videos of different genres.
However, unlike scanned paper documents, videotext
is superimposed on often changing backgrounds com-
prising moving objects with a rich variety of color and
texture. In addition, videotext is often of low resolution
and suffers from compression artifacts. Due to these
difficulties, existing OCR algorithms result in low accu-
racy when applied to the problem of videotext extraction
and recognition. On the other hand, we observed that
temporal videotext often persists on the screen over a
span of time (approximately 30 seconds) to ensure read-
ability, resulting in many available samples of the same
body of text over multiple frames of video. This redun-
dancy can be exploited to improve recognition accuracy,
although erroneous text extraction and/or incorrect char-
acter recognition by a classifier may make the strings
dissimilar from frame to frame. Often no single instance of
the text may lead to perfect recognition, underscoring the
need for intelligent postprocessing.
In the existing literature, unfortunately, temporal con-
tiguity analysis of videotext is implemented by using ad-
hoc thresholds and heuristics for the following reasons.
First of all, due to missed or merged characters, the same
string may be perceived to be of different lengths on
different frames. We thus have feature vectors of varying
lengths. Secondly, the exact duration of the persistence
of videotext is unknown a priori. Two consecutive frames
can have completely different strings. Thirdly, videotext
can be in scrolling motion. Because multiple moving text
blocks can be present in the same video frame, it is
nontrivial to recognize which videotext objects from con-
secutive frames are instances of the same text.
In light of these difficulties, we present brief illustra-
tive examples of dimensionality reduction and visualiza-
Figure 1. Illustrative frames with videotext
tion of feature vectors comprising strings of recognized
videotext. Experiments with our Edit distance kernel in- M
vestigated the use of KPCA for analyzing the temporal
contiguity of videotext using these feature vectors.
Videotext Clustering and Change Detection: Fig-
ure 2 shows the first two components obtained by
applying KPCA with the Edit distance kernel to a set
of strings recognized from 20 consecutive frames.
These frames contain instances of two distinct
strings. Without any assumed knowledge, KPCAs
use of the Edit distance kernel clearly shows two
distinct clusters corresponding to these two strings.
Videotext Tracking and Outlier Detection: Figure
3 shows the first three principal components ob-
tained by applying KPCA with our Edit distance
kernel to a set of strings recognized from 20 con-
secutive frames. These frames contain instances of
videotext scrolling across the screen. In this three-
dimensional plot, we can see a visual representation
of the changing content of videotext as a trajectory
in the principal component space, and locating
outliers from the trajectory indicates the appear-
ance of other strings in the video frames.
These results show that the symbolic kernels can
assist significantly in automated agglomeration and track-
ing of recognized text as well as effective data visualiza-
tion. In addition, multimodal feature vectors constructed
from recognized text strings and frame motion estimates
can now be analyzed jointly by using our hybrid kernel for
media content characterization.
FUTURE TRENDS
One of the big hurdles facing media management systems
is the semantic gap between the high-level meaning sought
by user queries in search for media and the low-level
features that we actually compute today for media index-
ing and description. Computational Media Aesthetics, a
promising approach to bridging the gap and building
845
TEAM LinG
 Multimodal Analysis in Multimedia Using Symbolic Kernels
Figure 2. Videotext clustering
high-level semantic descriptions for media search and
navigation services, is founded upon an understanding
of media elements and their individual and joint roles in
synthesizing meaning and manipulating perceptions, with
a systematic study of media productions (Dorai &
Venkatesh, 2001). The core trait of this approach is that in
order to create effective tools for automatically under-
standing video, we need to be able to interpret the data
with its makers eye. In order to realize the potential of this
approach, it becomes imperative that all sources of de-
scriptive information, audio, video, text, and so forth need
to be considered as a whole and analyzed together to
derive inferences with certain level of integrity. With the
ability to treat multimodal features as an integrated fea-
ture set to describe media content during classification
and visualization, new higher level semantic mappings
from low-level features can be achieved to describe media
content. The symbolic kernels are promising an initial step
in that direction to facilitate rigorous joint feature analysis
in various media domains.
CONCLUSION
Traditional integer representation of symbolic multimedia
feature data for classification and other data-mining tasks
is artificial, as the symbolic space may not reflect the
continuity and neighborhood relations as imposed by
integer representations. In this paper, we use distance-
based kernels in conjunction with kernel space methods
such as KPCA to handle multimodal data, including sym-
bolic features. These symbolic kernels, as shown in this
846
Figure 3. Videotext tracking and visualization
paper, help apply traditionally numeric methods to sym-
bolic spaces without any forced integer mapping for
important tasks such as data visualization, principal com-
ponent extraction, and clustering in multimedia and other
domains.
REFERENCES
Aradhye, H., & Dorai, C. (2002). New kernels for analyzing
multimodal data in multimedia using kernel machines.
Proceedings of the IEEE International Conference on
Multimedia and Expo, Switzerland, 2 (pp. 37-40).
Bradley, P. S., Fayyad, U. M., & Mangasarian, O. (1998).
Data mining: Overview and optimization opportunities
(Tech. Rep. No. 98-01). Madison: University of Wiscon-
sin, Computer Sciences Department.
Collins, M., Dasgupta, S., & Schapire, R. (2001). A gener-
alization of principal component analysis to the exponen-
tial family. In T.G. Dietterich, S. Becker, & Z. Ghahramani
(Eds.), Advances in neural information processing sys-
tems 14 (pp. 617-624). Cambridge, MA: MIT Press.
Cortes, C., Haffner, P., & Mohri, M. (2002). Rational
kernels. In S. Becker, S. Thrun, & K. Obermayer (Ed.),
Advances in neural information processing systems 15
(pp. 41-56). Cambridge, MA: MIT Press.
Cortes, C., Haffner, P., & Mohri, M. (2003). Positive
definite rational kernels. Proceedings of the 16th Annual
Conference on Computational Learning Theory (pp. 41-
56), USA.
TEAM LinG
Multimodal Analysis in Multimedia Using Symbolic Kernels
Dimitrova, N., Agnihotri, L., Dorai, C., & Bolle, R. (2000,
October). MPEG-7 videotext descriptor for superimposed
text in images and video. Signal Processing: Image Com-
munication, 16, 137-155.
Dorai, C., & Venkatesh, S. (2001, October). Computational
media aesthetics: Finding meaning beautiful. IEEE Mul-
timedia, 8(4), 10-12.
Jaakkola, T. S., & Haussler, D. (1999). Exploiting genera-
tive models in discriminative classifiers. In M. S. Kearns,
S. A. Solla, & D. A. Cohn (Eds.), Advances in neural
information processing systems 11 (pp. 487-493). Cam-
bridge, MA: MIT Press.
Scholkopf, B., Smola, A., & Muller, K. R. (1999). Kernel
principal component analysis. In B. Scholkopf, C. J. C.
Burges, & A. J. Smola (Eds.), Advances in kernel methods:
SV learning (pp. 327-352). Cambridge, MA: MIT Press.
Tipping, M. E. (1999). Probabilistic visualisation of high-
dimensional binary data. In M. S. Kearns, S. A. Solla, & D.
A. Cohn (Eds.), Advances in neural information process-
ing systems 11 (pp. 592-598). Cambridge, MA: MIT Press.
KEY TERMS
Dimensionality Reduction: The process of transfor-
mation of a large dimensional feature space into a space
comprising a small number of (uncorrelated) components.
Dimensionality reduction allows us to visualize, catego-
rize, or simplify large datasets.
Kernel Function: A function that intrinsically defines
the projection of two feature vectors (function argu-
ments) onto a high-dimensional space and a dot product
therein.
Mercers Condition: A kernel function is said to obey
Mercers condition for kernel validity iff the kernel matrix M
comprising pairwise kernel evaluations over any given
subset of the feature space is guaranteed to be positive
semidefinite.
MPEG Compression: Video/audio compression stan-
dard established by Motion Picture Experts Group. MPEG
compression algorithms use psychoacoustic modeling of
audio and motion analysis as well as DCT of video data for
efficient multimedia compression.
Multimodality of Feature Data: Feature data is said to
be multimodal if the features can be characterized as a
mixture of real-valued, discrete, ordinal, or nominal
values.
Principal Component Analysis (PCA): One of the
oldest modeling and dimensionality reduction techniques.
PCA models observed feature data as a linear combination
of a few uncorrelated, Gaussian principal components and
additive Gaussian noise.
Videotext: Text graphically superimposed on video
imagery, such as caption text, headline news, speaker
identity, location, and so on.
ENDNOTE
1
The term symbolic is loosely used in this paper to
mean discrete, ordinal, and nominal features.
847
TEAM LinG
848
Multiple Hypothesis Testing for Data Mining
Sach Mukherjee
University of Oxford, UK
INTRODUCTION
A number of important problems in data mining can be
usefully addressed within the framework of statistical
hypothesis testing. However, while the conventional
treatment of statistical significance deals with error
probabilities at the level of a single variable, practical
data mining tasks tend to involve thousands, if not mil-
lions, of variables. This Chapter looks at some of the
issues that arise in the application of hypothesis tests to
multi-variable data mining problems, and describes two
computationally efficient procedures by which these
issues can be addressed.
BACKGROUND
Many problems in commercial and scientific data min-
ing involve selecting objects of interest from large
datasets on the basis of numerical relevance scores
(object selection). This Section looks briefly at the
role played by hypothesis tests in problems of this kind.
We start by examining the relationship between rel-
evance scores, statistical errors and the testing of hy-
potheses in the context of two illustrative data mining
tasks. Readers familiar with conventional hypothesis
testing may wish to progress directly to the main part of
the Chapter.
As a topical example, consider the differential analy-
sis of gene microarray data (Piatetsky-Shapiro &
Tamayo, 2004; Cui & Churchill, 2003). The data consist
of expression levels (roughly speaking, levels of activ-
ity) for each of thousands of genes across two or more
conditions (such as healthy and diseased). The data
mining task is to find a set of genes which are differen-
tially expressed between the conditions, and therefore
likely to be relevant to the disease or biological process
under investigation. A suitably defined mathematical
function (the t-statistic is a canonical choice) is used to
assign a relevance score to each gene and a subset of
genes selected on the basis of the scores. Here, the
objects being selected are genes.
As a second example, consider the mining of sales
records. The aim might be, for instance, to focus mar-
keting efforts on a subset of customers, based on some
property of their buying behavior. A suitably defined
function would be used to score each customer by rel-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
evance, on the basis of his or her records. A set of
customers with high relevance scores would then be
selected as targets for marketing activity. In this ex-
ample, the objects are customers.
Clearly, both tasks are similar; each can be thought
of as comprising the assignment of a suitably defined
relevance score to each object and the subsequent se-
lection of a set of objects on the basis of the scores. The
selection of objects thus requires the imposition of a
threshold or cut-off on the relevance score, such that
objects scoring higher than the threshold are returned as
relevant. Consider the microarray example described
above. Suppose the function used to rank genes is simply
the difference between mean expression levels in the
two classes. Then the question of setting a threshold
amounts to asking how large a difference is sufficient to
consider a gene relevant. Suppose we decide that a
difference in means exceeding x is large enough: we
would then consider each gene in turn, and select it as
relevant if its relevance score equals or exceeds x.
Now, an important point is that the data are random
variables, so that if measurements were collected again
from the same biological system, the actual values
obtained for each gene might differ from those in the
particular dataset being analyzed. As a consequence of
this variability, there will be a real possibility of obtain-
ing scores in excess of x from genes which are in fact
not relevant.
In general terms, high scores which are simply due to
chance (rather than the underlying relevance of the
object) lead to the selection of irrelevant objects; er-
rors of this kind are called false positives (or Type I
errors). Conversely, a truly relevant object may have an
unusually low score, leading to its omission from the
final set of results. Errors of this kind are called false
negatives (or Type II errors). Both types of error are
associated with identifiable costs: false positives lead
to wasted resources, and false negatives to missed op-
portunities. For example, in the market research con-
text, false positives may lead to marketing material
being targeted at the wrong customers; false negatives
may lead to the omission of the right customers from
the marketing campaign. Clearly, the rates of each kind
of error are related to the threshold imposed on the
relevance score: an excessively strict threshold will
minimize false positives but produce many false nega-
tives, while an overly lenient threshold will have the
TEAM LinG
Multiple Hypothesis Testing for Data Mining
opposite effect. Setting an appropriate threshold is
therefore vital to controlling errors and associated costs.
Statistical hypothesis testing can be thought of as a
framework within which the setting of thresholds can be
addressed in a principled manner. The basic idea is to
specify an acceptable false positive rate (i.e. an accept-
able probability of Type I error) and then use probability
theory to determine the precise threshold which corre-
sponds to that specified error rate. A general discussion
of hypothesis tests at an introductory level can be found
in textbooks of statistics such as DeGroot and Schervish
(2002), or Moore and McCabe (2002); the standard
advanced reference on the topic is Lehmann (1997).
Now, let us assume for the moment that we have only
one object to consider. The hypothesis that the object is
irrelevant is called the null hypothesis (and denoted by
H0), and the hypothesis that it is relevant is called the
alternative hypothesis (H1). The aim of the hypothesis
test is to make a decision regarding the relevance of the
object, that is, a decision as to which hypothesis should
be accepted. Suppose the relevance score for the object
under consideration is t. A decision regarding the rel-
evance of the object is then made as follows:
(1) Specify an acceptable level of Type I error p*.
(2) Use the sampling distribution of the relevance
score under the null hypothesis to compute a
threshold score corresponding to p*. Let this
threshold score be denoted by c.
(3) If t c, reject the null hypothesis and regard the
object as relevant. If t < c, regard the object as
irrelevant.
The specified error level p* is called the signifi-
cance level of the test and the corresponding threshold
c the critical value.
Hypothesis testing can alternatively be thought of as
a procedure by which relevance scores are converted
into corresponding error probabilities. The null sam-
pling distribution can be used to compute the probabil-
ity p of making a Type I error if the threshold is set at
exactly t, i.e. just low enough to select the given object.
This then allows us to assert that the probability of
obtaining a false positive if the given object is to be
selected is at least p. This latter probability of Type I
error is called a P-value. In contrast to relevance scores,
P-values, being probabilities, have a clear interpreta-
tion. For instance, if we found that an object had a t-
statistic value of 3 (say), it would be hard to tell whether
the object should be regarded as relevant or not. How-
ever, if we found the corresponding P-value was 0.001,
we would know that if the threshold were set just low
enough to include the object, the false positive rate
would be 1 in 1000, a fact that is far easier to interpret.
MAIN THRUST
M
We have seen that in the case of a single variable,
relevance scores obtained from test statistics can be
easily converted into error probabilities called P-val-
ues. However, practical data mining tasks, such as min-
ing microarrays or consumer records, tend to be on a
very large scale, with thousands, even millions of ob-
jects under consideration. Under these conditions of
multiplicity, the conventional P-value described above
no longer corresponds to the probability of obtaining a
false positive.
An example will clarify this point. Consider once
again the microarray analysis scenario, and assume that
a suitable relevance scoring function has been chosen.
Now, suppose we wish to set a threshold corresponding
to a false positive rate of 0.05. Let the relevance score
whose P-value is 0.05 be denoted by t 05. Then, in the case
of a single variable/gene, if we were to set the threshold
at t 05, the probability of obtaining a false positive would
be 0.05. However, in the multi-gene setting, it is each of
the thousands of genes under study that is effectively
subjected to a hypothesis test with the specified error
probability of 0.05. Thus, the chance of obtaining a false
positive is no longer 0.05, but much higher. For in-
stance, if each of 10000 genes were statistically inde-
pendent, (0.05 10000) = 500 genes would be mistak-
enly selected on average! In effect, the very threshold
which implied a false positive rate of 0.05 for a single
gene now leaves us with hundreds of false positives.
Multiple hypothesis testing procedures address the
issue of multiplicity in hypothesis tests and provide a
way of setting appropriate thresholds in multi-variable
problems. The remainder of this Section describes two
well-known multiple testing methods (the Bonferroni
and False Discovery Rate methods), and discusses their
advantages and disadvantages.
Table 1 summarizes the numbers of objects in vari-
ous categories, and will prove useful in clarifying some
of the concepts presented below. The total number of
objects under consideration is m, of which m1 are rel-
evant and m0 are irrelevant. A total of S objects are
selected, of which S1 are true positives and S0 are false
positives. We follow the convention that variables relat-
ing to irrelevant objects have the subscript 0 (to
signify the null hypothesis) and those relating to rel-
evant objects the subscript 1 (for the alternative hy-
pothesis). Note also that fixed quantities (e.g. the total
number of objects) are denoted by lower-case letters,
while variable quantities (e.g. the number of objects
selected) are denoted by upper-case letters.
The initial stage of a multi-variable analysis follows
from our discussion of basic hypothesis testing and is
849
TEAM LinG

Table 1. Summary table for multiple testing, following
Benjamini and Hochberg (1995)
Selected Not selected
Irrelevant objects S0 m0-S0
Relevant objects S1 m1-S1
Total S m-S
common to the methods discussed below. The data is
processed as follows:
(1) Score each of the m objects under consideration
using a suitably chosen test statistic f. If the data
corresponding to object j is denoted by D j, and the
corresponding relevance score by Tj:
Tj = f (D j )
Let the scores so obtained be denoted by T1, T2,Tm.
(2) Convert each score T j to a corresponding P-value Pj
by making use of the relevant null sampling distri-
bution:
null sampling
distribution
Tj Pj
These P-values are called nominal P-values and
represent per-test error probabilities. The procedure by
which a P-value corresponding to an observed test statis-
tic is computed is not described here, but can be found in
any textbook of statistics, such as those mentioned pre-
viously. Essentially, the null cumulative distribution func-
tion of the test statistic is used to compute the probabil-
ity of Type I error at threshold T j.
(3) Arrange the P-values obtained in the previous step
in ascending order (smaller P-values correspond to
objects more likely to be relevant). Let the or-
dered P-values be denoted by:
P(1) P( 2)  P( m )
Thus, each object has a corresponding P-value. In order
to select a subset of objects, it will therefore be sufficient
to determine a threshold in terms of nominal P-value.
The Bonferroni Method
This is perhaps the simplest method for correcting P-
values. We first specify an acceptable probability of
850
Multiple Hypothesis Testing for Data Mining
committing at least one Type I error, and then calculate
a corresponding threshold in terms of nominal P-value.
Total
Procedure
m0
m1
m
(1) Specify an acceptable probability p* of at least
one Type I error being committed.
(2) Recall that P(1) P(2) P(m) represent the
ordered P-values. Find the largest i which satis-
fies the following inequality:
p*
P( i )
m
Let the largest i found be denoted k.
(3) Select the k objects corresponding to P-values
P(1), P(2) P(k) as relevant.
If we assume that all m tests are statistically inde-
pendent, it is easy to show that this simple procedure
does indeed guarantee that the probability of obtaining
at least one false positive is p*.
Advantages
The basic Bonferroni procedure is extremely
simple to understand and use.
Disadvantages
The Bonferroni procedure sets the threshold to
meet a specified probability of making at least
one Type I error. This notion of error is called
Family Wise Error Rate, or FWER, in the sta-
tistical literature. FWER is a very strict control
of error, and is far too conservative for most
practical applications. Using FWER on datasets
with large numbers of variables often results in
the selection of a very small number of objects,
or even none at all.
The assumption of statistical independence be-
tween tests is a strong one, and almost never
holds in practice.
The FDR Method of Benjamini and
Hochberg
An alternative way of thinking about errors in multiple
testing is the false discovery rate (FDR) (Benjamini &
Hochberg, 1995). Looking at Table 1 we can see that of
S objects selected, S0 are false positives. FDR is simply
TEAM LinG
Multiple Hypothesis Testing for Data Mining
the average proportion of false positives among the
objects selected:
S The FDR procedure outlined above, while simple and
FDR E 0
S tions, and while better than Bonferroni is still often too
Where, E[] denotes expectation. The Benjamini and
Hochberg method allows us to compute a threshold
corresponding to a specified FDR.
Procedure
(1) Specify an acceptable FDR q*
(2) Find the largest i which satisfies the following
inequality:
i Resampling Based Methods
P( i ) q*
m
Let the largest i found be denoted k.
(3) Select the k objects corresponding to P-values
P(1), P (2) P(k) as relevant.
Under the assumption that the m tests are statisti-
cally independent, it can be shown that this procedure
leads to a selection of objects such that the FDR is
indeed q*.
Advantages
The FDR concept is far less conservative than the
Bonferroni method described above.
Specifying an acceptable proportion q* of false
positives is an intuitively appealing way of con-
trolling the error rate in object selection.
The FDR method tells us in advance that a certain
proportion of results are likely to be false posi-
tives. As a consequence, it becomes possible to
plan for the associated costs.
Disadvantages
Again, the assumption of statistically independent
tests rarely holds.
FUTURE TRENDS
A great deal of recent work in statistics, machine learn-
ing and data mining has focused on various aspects of
multiple testing in the context of object selection.
Some important areas of active research which were not
discussed in detail are briefly described below.
Robust FDR Methods
M
computationally efficient, makes several strong assump-
conservative for practical problems (Storey & Tibshirani,
2003). The recently introduced q-value (Storey, 2003)
is a more sophisticated approach to FDR correction and
provides a very robust methodology for multiple test-
ing. The q-value method makes use of the fact that P-
values are uniformly distributed under the null hypoth-
esis to accurately estimate the FDR associated with a
particular threshold. Estimated FDR is then used to set
an appropriate threshold. The q-value approach is an
excellent choice in many multi-variable settings.
In recent years a number of resampling based methods
have been proposed for multiple testing (see e.g. Westfall
& Young, 1993). These methods make use of
computationally intensive procedures such as the boot-
strap (Efron, 1982; Davison & Hinckley, 1997) to per-
form non-parametric P-value corrections. Resampling
methods are extremely powerful and make fewer strong
assumptions than methods based on classical statistics,
but in many data mining applications the computational
burden of resampling may be prohibitive.
Machine Learning of Relevance
Functions
The methods described in this chapter allowed us to
determine threshold relevance scores, but very little
attention was paid to the important issue of choosing an
appropriate relevance scoring function. Recent research
in bioinformatics (Broberg, 2003; Mukherjee, 2004a)
has shown that the effectiveness of a scoring function
can be very sensitive to the statistical model underlying
the data, in ways which can be difficult to address by
conventional means. When fully labeled data (i.e. datasets
with objects flagged as relevant/irrelevant) are avail-
able, canonical supervised algorithms (see e.g. Hastie,
Tibshirani, & Friedman , 2001) can be used to learn
effective relevance functions. However, in many cases
- microarray data being one example - fully labeled data
is hard to obtain. Recent work in machine learning
(Mukherjee, 2004b) has addressed the problem of learn-
ing relevance functions in an unsupervised setting by
exploiting a probabilistic notion of stability; this ap-
proach turns out to be remarkably effective in settings
where underlying statistical models are poorly under-
stood and labeled data unavailable.
851
TEAM LinG

CONCLUSION
An increasing number of problems in industrial and
scientific data analysis involve multiple testing, and
there has consequently been an explosion of interest in
the topic in recent years. This Chapter has discussed a
selection of important concepts and methods in mul-
tiple testing; for further reading we recommend
Benjamini and Hochberg (1995) and Dudoit, Shaffer, &
Boldrick, (2003).
REFERENCES
Benjamini, Y., & Hochberg, Y. (1995). Controlling the
false discovery rate: A practical and powerful approach
to multiple testing. J. Roy. Stat. Soc. B 57, 289-300.
Broberg, P. (2003). Statistical methods for ranking differ-
entially expressed genes. Genome Biology, 4(6).
Cui, X., & Churchill, G. (2003). Statistical tests for differ-
ential expression in cDNA microarray experiments. Ge-
nome Biology, 4(210).
Davison, A. C., & Hinckley, D. V. (1997). Bootstrap
methods and their applications. Cambridge University
Press.
DeGroot, M. H., & Schervish, M. J. (2002). Probability
and statistics (3rd ed.). Addison-Wesley.
Dudoit, S., Shaffer, J. P., & Boldrick, J. C. (2003).
Multiple hypothesis testing in microarray experiments.
Statistical Science, 18(1), 71-103.
Efron, B. (1982). The jacknife, the bootstrap, and
other resampling plans. Society for Industrial and
Applied Mathematics.
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The
elements of statistical learning. Springer-Verlag.
Lehmann, E. L. (1997). Testing statistical hypotheses
(2nd ed.). Springer-Verlag.
Moore, D. S., & McCabe, G. P. (2002). Introduction to
the practice of statistics (4 th ed.). W. H. Freeman.
Mukherjee, S. (2004a). A theoretical analysis of gene
selection. Proceedings of the IEEE Computer Society
Bioinformatics Conference 2004 (CSB 2004). IEEE
press.
Mukherjee, S. (2004b). Unsupervised learning of ranking
functions for high-dimensional data (Tech. Rep. No.
PARG-04-02). University of Oxford, Department of
Engineering Science.
852
Multiple Hypothesis Testing for Data Mining
Piatetsky-Shapiro, G., & Tamayo, P. (2003). Microarray
Data mining: Facing the Challenges. SIGKDD Explora-
tions, 5(2).
Storey, J. D. (2003). The positive false discovery rate: A
Bayesian interpretation and the q-value. Annals of Statis-
tics, 31, 2013-2035.
Storey, J. D., & Tibshirani, R. (2003). Statistical signifi-
cance for genome-wide studies. Proceedings of the Na-
tional Academy of Sciences, 100 (pp. 9440-9445).
Westfall, P. H., & Young, S. S. (1993). Resampling-based
multiple testing: Examples and methods for p-value
adjustment. John Wiley & Sons.
KEY TERMS
Alternative Hypothesis: The hypothesis that an
object is relevant. In general terms, the alternative hy-
pothesis refers to the set of events that is the comple-
ment of the null hypothesis.
False Discovery Rate (FDR): The expected proportion
of false positives among the objects selected as relevant.
False Negative: The error committed when a truly
relevant object is not selected. More generally, a false
negative occurs when the null hypothesis is erroneously
accepted. Also called Type II error.
False Positive: The error committed when an object
is selected as relevant when it is in fact irrelevant. More
generally, a false positive occurs when the null hypoth-
esis is erroneously rejected. Also called Type I error.
Gene Microarray Data: Measurements of mRNA
abundances derived from biochemical devices called
microarrays. These are essentially measures of gene
activity.
Hypothesis Test: A formal statistical procedure by
which an interesting hypothesis (the alternative hypoth-
esis) is accepted or rejected on the basis of data.
Multiple Hypothesis Test: A formal statistical proce-
dure used to account for the effects of multiplicity in a
hypothesis test.
Null Hypothesis: The hypothesis that an object is
irrelevant. In general terms, it is the hypothesis we wish
to falsify on the basis of data.
P-Value: The P-value for an object is the probability
of obtaining a false positive if the threshold is set just
high enough to include the object among the set selected
TEAM LinG
Multiple Hypothesis Testing for Data Mining
as relevant. More generally, it is the false positive rate
corresponding to an observed test statistic.
Random Variable: A variable characterized by ran-
dom behavior in assuming its different possible values.
Sampling Distribution: The distribution of values
obtained by applying a function to random data.
Test Statistic: A relevance scoring function used in
a hypothesis test. Classical test statistics (such as the t- M
statistic) have null sampling distributions which are
known a priori. However, under certain circumstances
null sampling distributions for arbitrary functions can
be obtained by computational means.
853
TEAM LinG
854
Music Information Retrieval
Alicja A. Wieczorkowska
Polish-Japanese Institute of Information Technology, Poland
INTRODUCTION
Music information retrieval is a multi-disciplinary re-
search on retrieving information from music. This re-
search involves scientists from traditional, music, and
digital libraries; information science; computer sci-
ence; law; business; engineering; musicology; cognitive
psychology; and education (Downie, 2001).
BACKGROUND
A huge amount of audio resources, including music data,
is becoming available in various forms, both analog and
digital. Notes, CDs, and digital resources of the World
Wide Web are growing constantly in amount, but the
value of music information depends on how easy it can
be found, retrieved, accessed, filtered, and managed
(Fingerhut, 1997; International Organization for Stan-
dardization, 2003).
Music information retrieval consists of quick and
efficient searching for various types of audio data of
interest to the user, filtering them in order to receive
only the data items that satisfy the users preferences
(International Organization for Standardization, 2003).
This broad domain of research includes:
Audio retrieval by content;
Auditory scene analysis and recognition (Rosenthal
& Okuno, 1998);
Music transcription;
Denoising of old analog recordings; and
Other topics discussed further in this article
(Wieczorkowska & Ras, 2003).
The topics are interrelated, since the same or similar
techniques can be applied for various purposes. For
instance, source separation, usually applied in auditory
scene analysis, is used also for music transcription and
even restoring (denoising) of all recordings. The re-
search on music information retrieval has many applica-
tions. The most important include automatic production
of music score on the basis of the presented input,
retrieval of music pieces from huge audio databases, and
restoration of old recordings. Generally, the research
within the music information retrieval domain is fo-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
cused on harmonic structure analysis, note extraction,
melody and rhythm tracking, timbre and instrument
recognition, classification of type of the signal (speech,
music, pitched vs. non-pitched), and so forth.
The research basically uses digital audio recordings,
where sound waveform is digitally stored as a sequence
of discrete samples representing the sound intensity at
a given time instant, and MIDI files, storing information
on parameters of electronically synthesized sounds
(voice, note on, note off, pitch bend, etc.). Sound analy-
sis and data mining tools are used to extract information
from music files in order to provide the data that meet
users needs (Wieczorkowska & Ras, 2001).
MAIN THRUST
Music information retrieval domain covers a broad range
of topics of interest, and various types of music data are
investigated in this research. Basic techniques of digital
sound analysis for these purposes come from speech
processing focused on automatic speech recognition
and speaker identification (Foote, 1999). Sound de-
scriptors calculated this way can be added to the audio
files in order to facilitate content-based searching of
music databases.
The issue of representation of music and multimedia
information in a form that allows interpretation of the
informations meaning is addressed by MPEG-7 stan-
dard, named Multimedia Content Description Interface.
MPEG-7 provides a rich set of standardized tools to
describe multimedia content through metadata (i.e.,
data about data), and music information description also
has been taken into account in this standard (Interna-
tional Organization for Standardization, 2003).
The following topics are investigated within music
information retrieval domain:
Auditory scene analysis, which focuses on various
aspects of music like timbre description, sound
harmonicity, spatial origin, source separation, and
so forth (Bregman, 1990). Timbre is defined sub-
jectively as this feature of sound that distinguishes
two sounds of the same pitch, loudness, and dura-
tion. Therefore, subjective listening tests are of-
ten performed in this research, but also signal-
TEAM LinG
Music Information Retrieval
processing techniques are broadly applied here. One
of the main topics of computational auditory scene
analysis is automatic separation of individual sound
sources from a mixture. It is difficult with mixtures of
harmonic instrument sounds, where spectra overlap.
However, assuming time-frequency smoothness of
the signal, sound separation can be performed, and
when sound changes in time are observed, onset,
offset, amplitude, and frequency modulation have
similar shapes for all frequencies in the spectrum;
thus, a demixing matrix can be estimated for them
(Virtanen, 2003; Viste & Evangelista, 2003). Audio
source separation techniques also can be used to
source localization for auditory scene analysis. These
techniques, like independent component analysis,
originate from speech recognition in cocktail party
environment, where many sound sources are present.
Independent component analysis is used for finding
underlying components from multidimensional sta-
tistical data, and it looks for components that are
statistically independent (Vincent et al., 2003).
Computational auditory scene recognition aims at
classifying auditory scenes into predefined classes,
using audio information only. Examples of auditory
scenes are various outside and inside environments,
like streets, restaurants, offices, homes, cars, and
so forth. Statistical and nearest neighbor algorithms
can be applied for this purpose. In the nearest
neighbor algorithm, the class (type of auditory
scene, in this case) is assigned on the basis of the
distance of the investigated sample to the nearest
sample, for which the class membership is known.
Various acoustic features, based on Fourier spec-
tral analysis (i.e., mathematic transform, decom-
posing the signal into frequency components), can
be applied to parameterize the auditory scene for
classification purposes. Effectiveness of this re-
search approaches 70% correctness for about 20
auditory scenes (Peltonen et al., 2002).
Query-by-humming systems, which search me-
lodic databases using sung queries (Adams et al.,
2003). This topic represents audio retrieval by
contents. Melody usually is quantized coarsely
with respect to pitch and duration, assuming mod-
erate singing abilities of users. Music retrieval
system takes such an aural query (i.e., a motif or a
theme) as input, and searches the database for the
piece from which this query comes. Markov mod-
els, based on Markov chains, can be used for
modeling musical performances. Markov chain is
a stochastic process for which the parameter is
discrete time values. In Markov sequence of events,
the probability of future states depends on the
present state; in this case, states represent pitch (or
set of pitches) and duration (Birmingham et al.,
2001). Query-by-humming is one of more popular M
topics within music information retrieval domain.
Audio retrieval-by-example for orchestral music
aims at searching for acoustic similarity in an
audio collection, based on analysis of the audio
signal. Given an example audio document, other
documents in a collection can be ranked by simi-
larity on the basis of long-term structure; specifi-
cally, the variation of soft and louder passages,
determined from envelope of audio energy vs.
time in one or more frequency bands (Foote, 2000).
This research is a branch of audio retrieval by
content. Audio query-by-example search also can
be performed within a single document when
searching for sounds similar to the selected sound
event. Such a system for content-based audio re-
trieval can be based on a self-organizing feature
map (i.e., a special kind of neural network de-
signed by analogy with a simplified model of the
neural connections in the brain and trained to find
relationships in the data). Perceptual similarity
can be assessed on the basis of spectral evolution
in order to find sounds of similar timbre (Spevak
& Polfreman, 2001). Neural networks also are used
in other forms in audio information retrieval sys-
tems. For instance, time-delayed neural networks
(i.e., neural nets with time delay inputs) are applied,
since they perform well in speech recognition ap-
plications (Meier et al., 2000). One of applications
of audio retrieval-by-example is searching for the
piece in a huge database of music pieces with the
use of so-called audio fingerprintingtechnology
that allows piece identification. Given a short pas-
sage transmitted, for instance, via car phone, the
piece is extracted, and, most important, informa-
tion also is extracted on the performer and title
linked to this piece in the database. In this way, the
user may identify the piece of music with very high
accuracy (95%) only on the basis of a small re-
corded (possibly noisy) passage.
Transcription of music, defined as writing down
the musical notation for the sounds that constitute
the investigated piece of music. Onset detection
based on incoming energy in frequency bands and
multi-pitch estimation based on spectral analysis
may be used as the main elements of an automatic
music transcription system. The errors in such a
system may contain additional inserted notes,
omissions, or erroneous transcriptions (Klapuri
et al., 2001). Pitch tracking (i.e., estimation of
pitch of note events in a melody or a piece of
music) is often performed in many music infor-
mation retrieval systems. For polyphonic music,
855
TEAM LinG

polyphonic pitch-tracking and timbre separation in
digital audio is performed with such applications as
score-following and denoising of old analog re-
cordings, which is also a topic of interest within
music information retrieval. Wavelet analysis can
be applied for this purpose, since it decomposes
the signal in time-frequency space, and then musical
notes can be extracted from the result of this decom-
position (Popovic et al., 1995). Simultaneous poly-
phonic pitch and tempo tracking, aiming at automati-
cally inferring a musical notation that lists the pitch
and the time limits of each note, is a basis of the
automatic music transcription. A musical perfor-
mance can be modeled for these purposes using
dynamic Bayesian networks (i.e., directed graphical
models of stochastic processes) (Cemgil et al., 2003).
It is assumed that the observations may be gener-
ated by a hidden process that cannot be directly
experimentally observed, and dynamic Bayesian net-
works represent the hidden and observed states in
terms of state variables, which can have complex
interdependencies. Dynamic Bayesian networks
generalize hidden Markov models, which have one
hidden node and one observed node per observation
time.
Automatic characterizing the rhythm and tempo of
music and audio, revealing tempo and the relative
strength of particular beats, is a branch of research
on automatic music transcription. Since highly
structured or repetitive music has strong beat spec-
trum peaks at the repetition times, it allows tempo
estimation and distinguishing between different
kinds of rhythms at the same tempo. The tempo can
be estimated using beat spectral peak criterion (the
lag of the highest peak exceeding assumed time
threshold) accurately to within 1% in the analysis
window (Foote & Uchihashi, 2001).
Automatic classification of musical instrument
sounds, aiming at accurate identification of musi-
cal instruments playing in a given recording, based
on various sound analysis and data mining tech-
niques (Herrera et al., 2000; Wieczorkowska, 2001).
This research is focused mainly on monophonic
sounds, and sound mixes are usually addressed in
the research on separation of sound sources. In
most cases, sounds of instruments of definite pitch
have been investigated, but recently, research on
percussion also has been undertaken. Various analy-
sis methods are used to parameterize sounds for
instrument classification purposes, including time-
domain description, Fourier, and wavelet analysis.
Classifiers range from statistic and probabilistic
methods, through learning by example, to artificial
intelligence methods. Effectiveness of this research
856
Music Information Retrieval
ranges from about 70% accuracy for instrument
identification to more than 90% for instruments
family (i.e., strings, winds, etc.), approaching 100%
for discriminating impulsive and sustained sounds,
thus even exceeding human performance. Such
instrument sound classification can be included in
automatic music transcription systems.
Sonification, in which utilities for intuitive auditory
display (i.e., in audible form) are provided through
a graphical user interface (Ben-Tal et al., 2002).
Generating human-like expressive musical per-
formances with appropriately adjusted dynamics
(i.e., loudness), rubato (variation in time limits of
notes), vibrato (changes of pitch) (Mantaras &
Arcos, 2002), and identification of musical pieces
representing different types of emotions (ten-
derness, sadness, joy, calmness, etc.), which
music evokes. Emotions in music are gaining the
interest of researchers recently, also including
recognition of emotions in the recordings.
The topics mentioned previously interrelate and
sometimes partially overlap. For instance, both audi-
tory scene analysis and recognition may take into ac-
count a very broad range of recordings containing nu-
merous acoustic elements to identify and analyze. Query
by humming requires automatic transcription of music,
since the input audio samples first must be transformed
into the form based on musical notation, describing
basic melodic features of the query. Audio retrieval by
example and automatic classification of musical in-
strument sounds are both branches of retrieval by con-
tent. Transcription of music requires not only pitch
tracking, but also automatic characterizing the rhythm
and tempo, so these topics overlap. Pitch tracking is
needed in many research topics, including music tran-
scription, query by humming, and even automatic clas-
sification of musical instrument sounds, since pitch is
one of the features characterizing instrumental sound.
Sonification and generating human-like expressive
musical performances both are related to sound syn-
thesis, which are needed to create auditory display or
emotional performance. All these topics are focused
on a broad domain of music and its various aspects.
Results of this research are not always easily mea-
surable, especially in the case of synthesis-based top-
ics, since they usually are validated via subjective tests.
Other topics, like transcription of music, may produce
errors of various importance (i.e., wrong pitch, length,
omission, etc.), and comparison of the obtained tran-
script with the original score can be measured in many
ways, depending on the considered criteria. The easiest
estimation and comparison of results can be performed in
case of recognition of singular sound events or files. In
TEAM LinG
Music Information Retrieval
case of query by example, very high recognition rate has
been reached already (95% of correct piece identifica-
tion via audio fingerprinting), reaching commercial level.
The research on music information retrieval is gain-
ing an increasing interest from the scientific commu-
nity, and investigation of further issues in this domain
can be expected.
FUTURE TRENDS
Multimedia databases and library collections, expand-
ing tremendously nowadays, need efficient tools for
content-based search. Therefore, we can expect an in-
tensification of research effort on music information
retrieval, which may aid searching music data. Espe-
cially, tools for query-by-example and query-by-hum-
ming are needed, as well as tools for automatic music
transcription, so these areas should be investigated
broadly in the near future.
CONCLUSION
Music information retrieval is a broad range of re-
search, focusing on various aspects of possible applica-
tions. The main domains include audio retrieval by
content, automatic music transcription, denoising of
old recordings, generating human-like performances,
and so forth. The results of this research help users find
the audio data they need, even if the users are not
experienced musicians. Constantly growing audio re-
sources evoke a demand for efficient tools to deal with
this enormous amount of data; therefore, music infor-
mation retrieval becomes a dynamically developing field
of research.
REFERENCES
Adams, N.H., Bartsch, M.A., & Wakefield, G.H. (2003).
Coding of sung queries for music information retrieval.
Proceedings of the IEEE Workshop on Applications of
Signal Processing to Audio and Acoustics WASPAA03,
New Paltz, New York.
Ben-Tal, O., Berger, J., Cook, B., Daniels, M., Scavone, G.
& Cook, P. (2002). SONART: The sonification application
research toolbox. Proceedings of the 2002 International
Conference on Auditory Display, Kyoto, Japan.
Birmingham, W.P., Dannenberg, R.D., Wakefield, G.H.,
Bartsch, M.A., Bykowski, D., Mazzoni, D., Meek, C.,
Mellody, M. & Rand, B. (2001). MUSART: Music retrieval
via aural queries. Proceedings of ISMIR 2001 2 nd Annual M
International Symposium on Music Information Re-
trieval, Bloomington, Indiana.
Bregman, A.S. (1990). Auditory scene analysis, the per-
ceptual organization of sound. Cambridge, MS: MIT
Press.
Cemgil, A.T., Kappen, B., & Barber, D. (2003). Genera-
tive model based polyphonic music transcription. Pro-
ceedings of the IEEE Workshop on Applications of
Signal Processing to Audio and Acoustics WASPAA03,
New Paltz, New York.
de Mantaras, R.L., & Arcos, J.L. (2002, Fall). AI and
music: From composition to expressive performance.
AI Magazine, 43-58.
Downie, J.S. (2001). Wither music information re-
trieval: Ten suggestions to strengthen the MIR research
community. Proceedings of the Second Annual Inter-
national Symposium on Music Information Retrieval:
ISMIR 2001, Bloomington, Indiana.
Fingerhut, M. (1997). Le multimdia dans la
bibliothque. Culture et recherche, 61. Retrieved 2004
from http://catalogue.ircam.fr/articles/textes/Fingerhut
97a/.
Foote, J. (1999). An overview of audio information
retrieval. Multimedia Systems, 7(1), 2-11.
Foote, J. (2000). ARTHUR: Retrieving orchestral mu-
sic by long-term structure. Proceedings of the Interna-
tional Symposium on Music Information Retrieval
ISMIR 2000, Plymouth, Massachusetts.
Foote, J., & Uchihashi, S. (2001). The beat spectrum: A
new approach to rhythm analysis. Proceedings of the
International Conference on Multimedia and Expo
ICME 2001, Tokyo, Japan.
Herrera, P., Amatriain, X., Batlle, E., & Serra X. (2000).
Towards instrument segmentation for music content
description: A critical review of instrument classifica-
tion techniques. Proceedings of the International Sym-
posium on Music Information Retrieval ISMIR 2000,
Plymouth, Massachusetts.
International Organization for Standardization ISO/IEC
JTC1/SC29/WG11. (2003). MPEG-7 Overview. Re-
trieved 2004 from http://www.chiariglione.org/mpeg/
standards/mpeg-7/mpeg-7.htm
Klapuri, A., Virtanen, T., Eronen, A., & Seppnen, J.
(2001). Automatic transcription of musical recordings.
Proceedings of the Consistent & Reliable Acoustic Cues
for Sound Analysis CRAC Workshop, Aalborg, Denmark.
857
TEAM LinG

Meier, U., Stiefelhagen, R., Yang, J., & Waibel, A. (2000).
Towards unrestricted lip reading. International Journal
of Pattern Recognition and Artificial Intelligence, 14(5),
571-586.
Peltonen, V., Tuomi, J., Klapuri, A., Huopaniemi, J., &
Sorsa, T. (2002). Computational auditory scene recogni-
tion. Proceedings of the International Conference on
Acoustics Speech and Signal Processing ICASSP, Or-
lando, Florida.
Popovic, I., Coifman, R., & Berger, J. (1995). Aspects
of pitch-tracking and timbre separation: Feature detec-
tion in digital audio using adapted local trigonometric
bases and wavelet packets. Center for Studies in Music
Technology. Retrieved 2004 from http://www-
ccrma.stanford.edu/~brg/research/pc/pitchtrack.html
Rosenthal, D., & Okuno, H.G. (Eds.). (1998). Computa-
tional auditory scene analysis. Proceedings of the IJCAI-
95 Workshop, Mahwah, New Jersey.
Spevak, C., & Polfreman, R. (2001). Sound spottingA
frame-based approach. Proceedings of the Second An-
nual International Symposium on Music Information
Retrieval: ISMIR 2001, Bloomington, Indiana.
Vincent, E., Rodet, X., Rbel, A., Fvotte, C., & Carpentier,
.L. (2003). A tentative typology of audio source separa-
tion tasks. Proceedings of the 4th Symposium on Inde-
pendent Component Analysis and Blind Source Separa-
tion, Nara, Japan.
Virtanen, T. (2003). Algorithm for the separation of
harmonic sounds with time-frequency smoothness con-
straint. Proceedings of the 6th International Confer-
ence on Digital Audio Effects DAFX-03, London, UK.
Viste, H., & Evangelista, G. (2003). Separation of har-
monic instruments with overlapping partials in multi-
channel mixtures. Proceedings of the IEEE Workshop
on Applications of Signal Processing to Audio and
Acoustics WASPAA03, New Paltz, New York.
Wieczorkowska, A. (2001). Musical sound classifica-
tion based on wavelet analysis. Fundamenta
Informaticae Journal, 47(1/2), 175-188.
Wieczorkowska, A., & Ras, Z. (2001). Audio content
description in sound databases. In Proceedings of the
First Asia-Pacific Conference on Web Intelligence, WI
2001, Maebashi City, Japan.
Wieczorkowska, A., & Ras, Z.W. (Eds.). (2003). Music
information retrieval. Journal of Intelligent Information
Systems, 21, 1.
858
Music Information Retrieval
KEY TERMS
Digital Audio: Digital representation of sound wave-
form, recorded as a sequence of discrete samples, repre-
senting the intensity of the sound pressure wave at a
given time instant. Sampling frequency describes the
number of samples recorded in each second, and bit
resolution describes the number of bits used to repre-
sent the quantized (i.e., integer) value of each sample.
Fourier Analysis: Mathematical procedure for
spectral analysis, based on Fourier transform that de-
composes a signal into sine waves, representing fre-
quencies present in the spectrum.
Information Retrieval: The actions, methods, and
procedures for recovering stored data to provide infor-
mation on a given subject.
Metadata: Data about data (i.e., information about
the data).
MIDI: Musical Instrument Digital Interface. MIDI
is a common set of hardware connectors and digital
codes used to interface electronic musical instruments
and other electronic devices. MIDI controls actions
such as note events, pitch bends, and the like, while the
sound is generated by the instrument itself.
Music Information Retrieval: Multi-disciplinary
research on retrieving information from music.
Pitch Tracking: Estimation of pitch of note events
in a melody or a piece of music.
Sound: A physical disturbance in the medium through
which it is propagated. Fluctuation may change rou-
tinely, and such a periodic sound is perceived as having
pitch. The audible frequency range is from about 20 Hz
(hertz, or cycles per second) to about 20 kHz. Harmonic
sound wave consists of frequencies being integer mul-
tiples of the first component (fundamental frequency)
corresponding to the pitch. The distribution of fre-
quency components is called spectrum. Spectrum and
its changes in time can be analyzed using mathematical
transforms, such as Fourier or wavelet transform.
Wavelet Analysis: Mathematical procedure for
time-frequency analysis, based on wavelet transform
that decomposes a signal into shifted and scaled ver-
sions of the original function called wavelet.
TEAM LinG

Negative Association Rules in Data Mining
Olena Daly
Monash University, Australia
David Taniar
Monash University, Australia
INTRODUCTION
Data Mining is a process of discovering new, unexpected,
valuable patterns from existing databases (Chen, Han &
Yu, 1996; Fayyad et. al., 1996; Frawley, Piatetsky-Shapiro
& Matheus, 1991; Savasere, Omiecinski & Navathe, 1995).
Though data mining is the evolution of a field with a long
history, the term itself was introduced only relatively
recently in the 1990s. Data mining is best described as the
union of historical and recent developments in statistics,
artificial intelligence, and machine learning. These tech-
niques then are used together to study data and find
previously hidden trends or patterns within.
Data mining is finding increasing acceptance in sci-
ence and business areas that need to analyze large amounts
of data to discover trends that they could not otherwise
find. Different applications may require different data
mining techniques. The kinds of knowledge that could be
discovered from a database are categorized into associa-
tion rules mining, sequential patterns mining, classifica-
tion, and clustering (Chen, Han & Yu, 1996).
In this article, we concentrate on association rules
mining and, particularly, on negative association rules.
BACKGROUND
Association rules discover database entries associated
with each other in some way (Agrawal, Imielinski & Swami,
1993; Agrawal & Srikant, 1994; Agrawal, et. al., 1996;
Mannila, Toivonen & Verkamo, 1994; Piatetsky-Shapiro,
1991; Srikant & Agrawal, 1995). The example could be
supermarket items purchased in the same transaction,
weather conditions occurring on the same day, stock
market movements within the same trade day, words that
tend to appear in the same sentence. In association rules
mining, we search for sequences of associated entities,
where each subsequence forms association rules as well.
Association rule is an implication of the form AB, where
A and B are database itemsets. A and B belong to the same
database transaction. The discovered sequences and
later on association rules allow to study customers
purchase patterns, stock market movements, and so forth.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
859
N
There are two measures to evaluate association rules:
support and confidence (Agrawal, Imielinski & Swami,
1993). The rule AB has support s, if s% of all transactions
contains both A and B. The rule AB has confidence c,
if c% of transactions that contains A also contains B.
Association rules have to satisfy the user-specified mini-
mum support (minsup) and minimum confidence (minconf).
Such rules with high support and confidence are referred
to as strong rules (Agrawal, Imielinski & Swami, 1993).
The generation of the strong association rules is
decomposed into the following two steps (Agrawal,
Imielinski & Swami, 1993; Agrawal & Srikant, 1994): (i)
discover the frequent itemsets; and (ii) use the frequent
itemsets to generate the association rules.
Itemset is a set of database items. Itemsets that have
support at least equal to minsup are called frequent
itemsets. 1-itemset is called an itemset with one item (A),
2-itemset is called an itemset with two items (AB), k-
itemset is called an itemset with k items. The output of the
first step is all itemsets in the database that have their
support at least equal to the minimum support.
In the second step, all possible association rules will be
generated from each frequent itemset, and the confidence
of the possible rules will be calculated. If the confidence is
at least equal to minimum confidence, the discovered rule
will be listed in the output of the algorithm.
Consider an example database in Figure 1. Let the
minimum support be 40%, the minimum confidence 70%;
the database contains 5 records and 4 different items A, B,
C, D (see Figure 1). In a database with five records,
support 40% means two records. For an item (itemset) to
be frequent in the sample database, it has to occur in two
or more records.
Figure 1. A sample database
1 A,D
2 B,C,D
3 A,B,C
4 A,B,C
5 A,B
TEAM LinG

Discover the Frequent Itemsets
1-itemsets: The support of each database item is
calculated
Support(A)= 4 (records)
Support(B)= 4 (records)
Support(C)= 3 (records)
Support(D)= 2 (records)
All items occur in two or more records, so all of them
are frequent. Frequent 1-itemsets: A, B, C, and D
2-itemsets: From frequent 1-itemsets candidate 2-
itemsets are generated: AB, AC, AD, BC, BD, CD. To
calculate the support of the candidate 2-itemsets,
the database is scanned again.
Support(AB) =3 (records)
Support(AC) =2 (records)
Support(AD) =1 (record)
Support(BC) =3 (records)
Support(BD) =1 (record)
Support(CD) =1 (record)
Only itemsets occurring in two or more records are
frequent. Frequent 2-itemsets: AB, AC, and BC
3-itemsets: From frequent 2-itemsets candidate 3-
itemsets are generated: ABC. In step 3 for candidate
3-itemsets, it is allowed to join frequent 2-itemsets
that only differ in the last item. AB and AC from the
step 2 differ only in the last item, so they are joined
and the candidate 3-itemset ABC is obtained. AB
and BC differ in the first item so a candidate itemset
cannot be produced here; AC and BC do not differ
in the last item, no candidate itemset produced. So
there is only one candidate 3-itemset ABC. To cal-
culate the support of the candidate 3-itemsets the
database is scanned again.
Support(ABC) =2 (records)
Frequent 3-itemsets: ABC
4-itemsets: In step 3, there is only one frequent
itemset, so it is impossible to produce any candidate
4-itemsets. The frequent itemsets generation stops
here. If there was more than one frequent 3-itemset,
then step 4 (and onwards) would be similar to step 3.
The list of frequent itemsets: A, B, C, D, AB, AC, BC,
and ABC.
860
Negative Association Rules in Data Mining
Use the Frequent Itemsets to Generate
the Association Rules
To produce an association rule, at least frequent 2-itemset.
So the generation process starts with 2-itemsets and goes
on with longer itemsets. Every possible rule is produced
from each frequent itemset:
AB: AB
B A
AC: AC
C A
BC: B C
C B
ABC: A BC
B AC
C AB
BC A
AC B
AB C
To distinguish the strong association rules among all
possible rules, the confidence of each possible rule will be
calculated. All support values have been obtained in the
first step of the algorithm.
Confidence(AB)=Support(AB)/Support(A)
Confidence(ABC)=Support(ABC)/Support(AB)
Association rules have been one of the most devel-
oped areas in data mining. Most of research has been done
in positive implications, which is when occurrence of an
item implies the occurrence of another item. Negative
rules also consider negative implications, when occur-
rence of an item implies absence of another item.
MAIN THRUST
Negative itemsets are itemsets that contain both items
and their negations (e.g., AB~C~DE~F). ~C means nega-
tion of the item C (absence of the item C in the database
record).
Negative association rules are rules of a kind AB,
where A and B are frequent negative itemsets. Negative
association rules consider both presence and absence of
items in the database record and mine for negative impli-
cations between database items.
Examples of negative association rules could be
Meat~Fish, which implies that when customers pur-
chase meat at the supermarket, they do not buy fish at the
same time, or ~SunnyWindy, which means no sunshine
TEAM LinG
Negative Association Rules in Data Mining
implies wind, or ~OilStock~PetrolStock, which says if
the price the oil shares does not go up, petrol shares
wouldnt go up either.
In the negative association rules mining, the number of
possible generated negative rules is vast and numerous in
comparison to the positive association rules mining. It is
a critical issue to distinguish only the most interesting
ones among the candidates.
A simple and straightforward method to search nega-
tive association would be to add the negations of absent
items to the database transaction and treat them as addi-
tional items. Then, the traditional positive association
mining algorithms could be run on the extended database.
This approach works in domains with a very limited number
of database attributes. A good example is weather predic-
tion. The database only contains 10-15 different factors
like fog, rain, sunshine, and so forth. A database record
would look like sunshine, ~fog, ~rain, windy. Thus, the
number of database items will be doubled, and the number
of itemsets will increase significantly. Besides, the sup-
port of negations of items can be calculated, based on the
support of positive items. Then, specially created algo-
rithms are required.
The approach works in tasks like weather prediction/
limited stock market analyses but is hardly applicable to
domains with thousands of different database items (sales/
healthcare/marketing). There could be numerous and sense-
less negative rules generated then. That is why special
approaches are required to distinct interesting negative
association rules.
There are two categories in mining negative associa-
tion rules. In category I, a special measure for negative
association rules is created. In category II, the data/first-
generated rules are analyzed to further produce only most
interesting rules with no computational costs.
First, we explain category I. A measure to distinguish
the most valuable negative rules will be employed.
The interest measure may be applied to negative rules
(Wu, Zhang & Zhang, 2002). Interest measure was first
offered for positive rules. The measure is defined by a
formula: Interest(AB)=support(AB)-support(A)*
support(B). The negative rules are generated from infre-
quent itemsets. The mining process for negative rules
starts within 1-frequent itemsets.
A more complicated way to search negative associa-
tion rules is to employ statistical theories (Brin, Motwani
& Silverstein, 1997). Correlations among database items
are calculated. If the correlation is positive, a positive
association rule will be obtained. If the correlation is
negative, a negative association rule will be obtained. The
stronger the correlation, the stronger the generated rule.
In this approach, Chi-squared statistics is employed to
calculate the correlations among database items. Each
different database transaction (market basket) denotes a
cell in a contingency table. The chi-squared statistic is
calculated, which is, in short, a normalized deviation from N
expectation for each cell. From the obtained value, the
strength of the correlation between database items is
estimated.
Now, we explain category II. The data or negative
rules generated in the initial step are analyzed first, and
then the obtained information is employed on the next
steps of generation process.
Mining frequent negative itemsets starts with itemsets
that contain n items and only one negation of an item; for
example, {ABC~D, AB~CD}, where ~C, ~D means ab-
sence of the item C or D. Then, the produced rules with
n items and one negation are analyzed and some informa-
tion is obtained that allows the next step when producing
rules with n items and two negations (A~BC~D) to dis-
regard some itemsets without even considering (Fortes,
Balcazar & Morales, 2001). n takes on values from 1 to the
overall number of different items in the database. The
negative rules mining stops when the maximum number
of negations m in rules has been reached, m provided by
the user, or when all negative rules have been generated,
if m has not been provided.
Another approach in category II is to consider the
hierarchy of items. An example of database hierarchy is
shown in Figure 2.
It is supposed that database items are organized into
the hierarchy ancestor-to-descendant. First, the nega-
tive rules on the very top level are generated and then on
the lower levels. When proceeding to the lower level of
hierarchy, the information obtained from rules on the
higher level is utilized (Daly & Taniar, 2003, 2004). The
hierarchy approach makes the rules more general, which
is crucial for negative association. Any further explora-
tion can be ceased, when no additional knowledge will be
extracted on the lower levels.
For instance, if a rule {Beef Not Juice} is generated,
it is not interesting what kind of juice (because Not
Juice is negation of any kind of juice). In contrast, in the
positive rule {Beef Juice}, it is required to discover
a specific kind of juice and go deeper into the levels.
Figure 2. Hierarchy of items
Items
Stationary Food & Drinks
Writing
Instruments Paper Ink Drinks Food
Pen Pencil Juice Coke Fish Meat Pastry Dairy
Apple Juice Orange Juice Trout Beef Bread Milk Butter
861
TEAM LinG

Though the hierarchy approach requires additional infor-
mation about the database, the way items are organized
into a hierarchy and an experts opinion may be needed.
After the negative association rules have been gener-
ated, some reduction techniques may be applied to refine
the rules. For instance, the reduction techniques could
verify the true support of items (not ancestors) or gener-
alize the sets of negative rules.
FUTURE TRENDS
There has been some research in the area of negative
association mining. The negative association obviously
has provoked more complicated issues than the positive
association mining. In order to overcome the difficulties
of negative association mining, the research community
endeavors to invent new methods and algorithms to
achieve the desirable efficiency. Following are the limita-
tions and trends in the negative association mining.
Issues/Open Research Areas
A vast number of candidate rules for negative asso-
ciation rules in comparison with the positive asso-
ciation rules.
In comparison with the positive association mining, in
the negative association rules, a greater number of candi-
date itemsets is obtained, which would make it technically
impossible to evaluate all and distinguish the rules with
high support/confidence/interest. Besides, the gener-
ated rules may not provide efficient information. Special
approaches, measures, and algorithms should be devel-
oped to solve the issue.
Hardware limitations.
Hardware limitations are crucial for negative associa-
tion rules mining for the following reasons: a huge number
of calculations and assessments required from the CPU;
a storage for vast frequent itemsets trees in the sufficient
main memory; and rapid data interchange with databases
required.
Vast data sets.
With the development of the hardware components, vast
databases could be handled in sufficient time for the companies.
A need for advanced research in negative associa-
tion mining.
862
Negative Association Rules in Data Mining
Some research has been done in the negative associa-
tion rules mining, but there are many unexplored subar-
eas, and advanced research is essential for the areas
development.
A need for special interest measures and algorithms
for negative association vs. positive association.
New interest measures should be invented, new ap-
proaches for negative rules discovered, the ones that
would take advantage of the special properties of the
negative rules.
A need for parallel algorithms for negative associa-
tion rules mining.
Parallel algorithms have taken data mining to the new
extent of mining abilities. As the negative association is
a relatively new research area, the parallel data mining
algorithms have not been developed yet, and they are
absolutely necessary.
A need for new sophisticated reduction techniques.
After the set of negative rules has been generated, one
may apply some reduction and refining techniques to the
set to make sure the rules are not redundant.
The issues require additional research in the area to be
conducted and more scientists to contribute their knowl-
edge, time, and inspiration.
Current Trends
Use of various approaches in negative association
mining.
Researchers are currently trying to invent distinctive
approaches to generate the negative rules and to make
sure they provide valuable information. A straightfor-
ward approach is not acceptable for the negative rules;
support/confidence measures are not enough to distin-
guish the negative rules. Some novel/adapted measures,
particularly for interest measures, will need to be utilized.
Data structure analysis also forms an important part in the
process of negative rule generation.
Specialized mining algorithms development.
The algorithms for positive association rules mining
are not suitable for negative rules, so new and specialized
algorithms often are developed by the researchers cur-
rently working in negative rules mining.
TEAM LinG
Negative Association Rules in Data Mining
Rules reduction development.
Rules reduction techniques have been developed to
refine the final set of the negative rules. Reduction tech-
niques verify the rules quality or generalize the rules.
The researchers are attempting to overcome the vast
search space in negative association mining; they look at the
issue from different points of view to discover what makes
the negative rules more interesting and what measures could
distinguish the highly valuable rules from the rest.
CONCLUSION
This article is a review of the research that has been done
in the negative association rules. Negative association
rules have brought interesting challenges and research
opportunities. This is an open area for future research.
This article describes the main issues and approaches
in negative association rules mining derived from the
literature. One of the main issues is a vast number of
candidate rules for negative association rules in compari-
son with the positive association rules.
There are two categories in mining negative associa-
tion rules. In category I, a special measure for negative
association rules is created. In category II, the data/first-
generated rules are analyzed to further produce only most
interesting rules with no computational costs.
REFERENCES
Agrawal, R. et al. (1996). Fast discovery of association
rules. In U. Fayyad et al. (Eds.), Advances in knowledge
discovery and data mining. American Association for
Artificial Intelligence Press.
Agrawal, R., Imielinski, T., & Swami, A. (1993). Mining
association rules between sets of items in large data-
bases. Proceedings of the Association for Computing
Machinery, Special Interest Group on Management of
Data, International Conference Management of Data.
Agrawal, R., & Srikant, R. (1994). Fast algorithms for
mining association rules in large databases. Proceedings
of the 20th International Conference on Very Large Data
Bases.
Brin, S., Motwani, R., & Silverstein, C. (1997). Beyond
market basket: Generalizing association rules to correla-
tions. Proceedings of Association for Computing Ma-
chinery, Special Interest Group on Management of Data,
International Conference Management of Data.
Chen, M., Han, J., & Yu, P. (1996). Data mining: An
overview from a database perspective. Institute of Elec- N
trical and Electronics Engineers, Transactions on Knowl-
edge and Data Engineering, 8(6), 866-883.
Daly, O., & Taniar, D. (2003). Mining multiple-level nega-
tive association rules. Proceedings of the International
Conference on Intelligent Technologies (InTech03).
Daly, O., & Taniar, D. (2004). Exception rules mining based
on negative association rules. Computational Science
and Its Applications, 3046, 543-552.
Fayyad, U. et al. (1996). Advances in knowledge discov-
ery and data mining. American Association for Artificial
Intelligence Press.
Fortes, I., Balczar, J., & Morales, R. (2001). Bounding
negative information in frequent sets algorithms. Pro-
ceedings of the 4th International Conference of Discovery
Science.
Frawley, W., Piatetsky-Shapiro, G., & Matheus, C. (1991).
Knowledge discovery in databases: An overview. Ameri-
can Association for Artificial Intelligence Press.
Mannila, H., Toivonen, H., & Verkamo, A. (1994). Efficient
algorithms for discovering association rules. Proceed-
ings of the American Association for Artificial Intelli-
gence Workshop on Knowledge Discovery in Databases.
Piatetsky-Shapiro, G. (1991). Discovery, analysis, and
presentation of strong rules. American Association for
Artificial Intelligence Press.
Savasere, A., Omiecinski, E., & Navathe, S. (1995). An
efficient algorithm for mining association rules in large
databases. Proceedings of the 21st International Confer-
ence on Very Large Data Bases.
Srikant, R., & Agrawal, R. (1995). Mining generalized
association rules. Proceedings of the 21st Very Large
Data Bases Conference.
Wu, X., Zhang, C., & Zhang, S. (2002). Mining both
positive and negative association rules. Proceedings of
the 19th International Conference on Machine Learning.
KEY TERMS
Association Rules: An implication of the form AB,
where A and B are database itemsets. Association rules
have to satisfy the pre-set minimum support (minsup) and
minimum confidence (minconf) constraints.
863
TEAM LinG

Confidence: The rule AB has confidence c, if c% of
transactions that contain A, also contains B.
Database Item: An item/entity occurring in the database.
Frequent Itemsets: Itemsets that have support at least
equal to minsup.
Itemset: A set of database items.
864
Negative Association Rules in Data Mining
Negative Association Rules: Rules of a kind AB,
where A and B are frequent negative itemsets.
Negative Itemsets: Itemsets that contain both items
and their negations.
Support: The rule AB has support s, if s% of all
transactions contains both A and B.
TEAM LinG
 865

Neural Networks for Prediction and N

Classification
Kate A. Smith
Monash University, Australia

INTRODUCTION a set of inputs and known outputs. They are a supervised

Neural networks are simple computational tools for exam-
ining data and developing models that help to identify
interesting patterns or structures. The data used to de-
velop these models is known as training data. Once a
neural network has been exposed to the training data, and
has learnt the patterns that exist in that data, it can be
applied to new data thereby achieving a variety of out-
comes. Neural networks can be used to:
learn to predict future events based on the patterns
that have been observed in the historical training
data;
learn to classify unseen data into pre-defined groups
based on characteristics observed in the training
data;
learn to cluster the training data into natural groups
based on the similarity of characteristics in the
training data.
BACKGROUND
learning technique in the sense that they require a set of
training data in order to learn the relationships. With
supervised learning models, the training data contains
matched information about input variables and observ-
able outcomes. Models can be developed that learn the
relationship between these data characteristics (inputs)
and outcomes (outputs). Figure 1 shows the architecture
of a 3-layer MFNN. The weights are updated using the
backpropagation learning algorithm, as summarized in
Table 1, where f(.) is a non-linear activation function such
as 1/(1+exp(- (.)), is the learning rate, and dk is the
desired output of the kth neuron.
The successful application of MFNNs to a wide range
of areas of business, industry and society has been
reported widely. We refer the interested reader to the
excellent survey articles covering application domains of
medicine (Weinstein, Myers, Casciari, Buolamwini, &
Raghavan, 1994), communications (Ibnkahla, 2000), busi-
ness (Smith & Gupta, 2002), finance (Refenes, 1995), and
a variety of industrial applications (Fogelman, Souli, &
Gallinari, 1998).

There are many different neural network models that have MAIN THRUST
been developed over the last fifty years or so to achieve
these tasks of prediction, classification, and clustering. A number of significant issues will be discussed, and
Broadly speaking, these models can be grouped accord- some guidelines for successful training of neural net-
ing to supervised learning algorithms (for prediction and works will be presented in this section.
classification), and unsupervised learning algorithms (for
clustering). This paper focused on the former paradigm. Figure 1. Architecture of MFNN (note: not all weights
We refer the interested reader to Haykin (1994) for a are shown)
detailed account of many neural network models.
J neurons K neurons
According to a recent study (Wong, Jiang, & Lam, (hidden layer) (output layer)
2000), over fifty percent of reported neural network busi-
ness application studies utilise multilayered feedforward
x1 . w21
w11

w12
y1 v11 z1
v21

.
wJ1
neural networks (MFNNs) with the backpropagation learn- vK1
x2 w22
ing rule (Werbos, 1974; Rumelhart & McClelland, 1986). wJ2 y2 z2
This type of neural network is popular because of its
broad applicability to many problem domains of relevance x3 . w13
w23
..
.
. yJ zK
to business and industry: principally prediction, classifi- wJ2
. w 1N+1 wJ3
cation, and modeling. MFNNs are appropriate for solving . w v1j

.
2N+1

problems that involve learning the relationships between -1

w JN+1
-1 .

Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.

TEAM LinG
 Neural Networks for Prediction and Classification

Table 1. Backpropagation learning algorithm for MFNNs

STEP 1: Randomly select an input pattern x to present to the MFNN through the
input layer

STEP 2: Calculate the net inputs and outputs of the hidden layer neurons
N +1
net hj = w ji xi y j = f (net hj )
i =1

STEP 3: Calculate the net inputs and outputs of the K output layer neurons
J +1
net ko = v kj y j z k = f ( net ko )
j =1

STEP 4: Update the weights in the output layer (for all k, j pairs)
vkj vkj + c (d k z k ) z k (1 z k ) y j

STEP 5: Update the weights in the hidden layer (for all i, j pairs)
K
w ji w ji + c 2 y j (1 y j ) x i ( ( d k z k ) z k (1 z k )v kj )
k =1

STEP 6: Update the error term

K
E E + (d k z k ) 2
k =1

and repeat from STEP 1 until all input patterns have been presented (one
epoch).

STEP 7: If E is below some predefined tolerance level (say 0.000001), then

STOP.
Otherwise, reset E = 0, and repeat from STEP 1 for another epoch.

Critical Issues for Neural Networks
Neural networks have not been without criticism, and
there are a number of important limitations that need
careful attention. At this stage it is still a trial and error
process to obtain the optimal model to represent the
relationships in a dataset. This requires appropriate selec-
tion of a number of parameters including learning rate,
momentum rate (if an additional momentum term is added
to steps 4 and 5 of the algorithm), choice of activation
function, as well as the optimal neural architecture. If the
architecture is too large, with too many hidden neurons,
the MFNN will find it very easy to memorize the training
data, but will not have generalized its knowledge to other
datasets (out-of-sample or test sets). This problem is
known as overtraining. Another limitation is the gradient
descent nature of the backpropagation learning algo-
rithm, which causes the network weights to become
trapped in local minima of the error function being
minimized. Finally, neural networks are considered inap-
propriate for certain problem domains where insight and
explanation of the model is required. Some work has been
done on extracting rules from trained neural networks
(Andrews, Diederich, & Tickle, 1995), but other data
mining techniques like rule induction may be better suited
to such situations.
Guidelines for Successful Training
Successful prediction and classification with MFNNs
requires careful attention to two main stages:
866
1. Learning: The developed model must represent
an adequate fit of the training data. The data itself
must contain the relationships that the neural net-
work is trying to learn, and the neural network
model must be able to derive the appropriate
weights to represent these relationships.
2. Generalization: The developed model must also
perform well when tested on new data to ensure that
it has not simply memorized the training data char-
acteristics. It is very easy for a neural network model
to overfit the training data, especially for small
data sets. The architecture needs to be kept small,
and key validation procedures need to be adopted
to ensure that the learning can be generalized.
Within each of these stages there are a number of
important guidelines that can be adopted to ensure effec-
tive learning and successful generalization for prediction
and classification problems (Remus & OConnor, 2001).
To ensure successful learning:
Prepare the Data Prior to Learning the Neural
Network Model: A number of pre-processing
steps may be necessary including cleansing the
data; removing outliers; determining the correct
level of summarisation; converting non-numeric
data (Pyle, 1999).
Normalise, Scale, Deseasonalise and Detrend
the Data Prior to Learning: Time series data
often needs to be deseasonalised and detrended to
enable the neural network to learn the true patterns
in the data (Zhang, Patuwo, & Hu, 1998).

TEAM LinG
Neural Networks for Prediction and Classification
Ensure that the MFNN Architecture is Appro-
priate to Learn the Data: If there are not enough
hidden neurons, the MFNN will be unable to repre-
sent the relationships in the data. Most commercial
software packages extend the standard 3-layer
MFNN architecture shown in Figure 1 to consider
additional layers of neurons, and sometimes in-
clude feedback loops to provide enhanced learning
capabilities. The number of input dimensions can
also be reduced to improve the efficiency of the
architecture if inclusion of some variables does
not improve the learning.
Experiment with the Learning Parameters to
Ensure that the Best Learning is Produced:
There are several parameters in the backpropagation
learning equations that require selection and ex-
perimentation. These include the learning rate c,
the equation of the function f() and its gradient l,
and the values of the initial weights.
Consider Alternative Learning Algorithms to
Backpropagation: Backpropagation is a gradient
descent technique that guarantees convergence only
to a local minimum of the error function. Other local
optimization techniques, such as the Levenberg-
Marquardt algorithm, have gained popularity due to
their increased speed and reduced memory require-
ments (Kinsella, 1992). Recently, researchers have
used more sophisticated search strategies such as
genetic algorithms and simulated annealing in an
effort to find globally optimal weight values (Sexton
& Dorsey, 2000).
To ensure successful generalization:
Extract a Test Set from the Training Data: Com-
monly 20% of the training data is reserved as a test
set. The neural network is only trained on 80% of
the data, and the degree to which it has learnt or
memorized the data is gauged by the measured
performance on the test set. When ample addi-
tional data is available, a third group of data known
as the validation set is used to evaluate the gener-
alization capabilities of the learnt model. For time
series prediction problems, the validation set is
usually taken as the most recently available data,
and provides the best indication of how the devel-
oped model will perform on future data. When
there is insufficient data to extract a test set and
leave enough training data for learning, cross-vali-
dation sets are used. This involves randomly ex-
tracting a test set, developing a neural network
models based on the remaining training data, and
repeating the process with several random divi-
sions of the data. The reported results are based on
the average performance of all randomly extracted
test sets. The most popular method is known as N
ten-fold cross-validation, and involves repeating
the approach for ten distinct subsets of the data.
Avoid Unnecessarily Large and Complex Ar-
chitectures: An architecture containing a large
number of hidden layers and hidden neurons re-
sults in more weights than a smaller architecture.
Since the weights correspond to the degrees of
freedom or number of parameters the model has
to fit the data, it is very easy for such large
architectures to overfit the training data. For the
sake of future generalization of the model, the
architecture should therefore be only as large as
is required to learn the data and achieve an accept-
able performance on all data sets (training, test,
and validation where available).
When these guidelines are observed, the chances of
developing a MFNN model that learns the training data
effectively and generalizes its learning on new data are
greatly improved. Most commercially available neural
network software packages include features to facilitate
adherence to these guidelines.
FUTURE TRENDS
Despite the successful application of neural networks to
a wide range of application areas, there is still much
research that continues to improve their functionality.
Specifically, research continues in the development of
hardware models (chips or specialized analog devices)
that enable neural networks to be implemented rapidly in
industrial contexts. Other research attempts to connect
neural networks back to their roots in neurophysiology,
and seeks to improve the biological plausibility of the
models. On-line learning of neural network models that
are more effective in situations when the data is dynami-
cally changing, will also become increasingly important.
A useful discussion of the future trends of neural net-
works can be found at a virtual workshop discussion:
http://www.ai.univie.ac.at/neuronet/workshop/.
CONCLUSION
Over the last decade or so, we have witnessed neural
networks come of age. The idea of learning to solve
complex pattern recognition problems using an intelli-
gent data-driven approach is no longer simply an inter-
esting challenge for academic researchers. Neural net-
works have proven themselves to be a valuable tool
across a wide range of application areas. As a critical
867
TEAM LinG

component of most data mining systems, they are also
changing the way organizations view the relationship
between their data and their business strategy.
The multilayered feedforward neural network (MFNN)
has been presented as the most common neural network
employing supervised learning to model the relationships
between inputs and outputs. This dominant neural net-
work model finds application across a broad range of
prediction and classification problems. A series of critical
guidelines have been provided to facilitate the successful
application of these neural network models.
REFERENCES
Andrews, R., Diederich, J., & Tickle, A. (1995). A survey
and critique of techniques for extracting rules from trained
artificial neural networks. Knowledge Based Systems, 8,
373-389.
Fogelman, S. F., & Gallinari, P. (Eds.). (1998). Industrial
applications of neural networks. Singapore: World Sci-
entific Press.
Haykin, S. (1994). Neural networks: A comprehensive
foundation. Englewood Cliffs, NJ: Macmillan Publishing
Company.
Ibnkahla, M. (2000). Applications of neural networks to
digital communications - a survey. Signal Processing,
80(7), 1185-1215.
Kinsella, J. A. (1992). Comparison and evaluation of
variants of the conjugate gradient methods for efficient
training in feed-forward neural networks with backward
error propagation. Neural Networks, 3(27).
Pyle, D. (1999). Data preparation for data mining. San
Francisco: Morgan Kaufmann Publishers.
Refenes, A. P. (Ed.). (1995). Neural networks in the
capital markets. Chichester: John Wiley & Sons Ltd.
Remus, W., & OConnor, M. (2001). Neural networks for
time series forecasting. In J. S. Armstrong (Ed.), Prin-
ciples of forecasting: A handbook for researchers and
practitioners. Norwell, MA: Kluwer Academic Publishers.
Rumelhart, D. E., & McClelland, J. L. (Eds.). (1986). Par-
allel distributed processing: Explorations in the micro-
structure of cognition. Cambridge, MA: MIT Press.
Sexton, R. S., & Dorsey, R. E. (2000). Reliable classification
using neural networks: A genetic algorithm and
backpropagation comparison. Decision Support Systems,
30, 11-22.
868
Neural Networks for Prediction and Classification
Smith, K. A., & Gupta, J. N. D. (Eds.). (2002). Neural
networks in business: Techniques and applications.
Hershey, Pennsylvania: Idea Group Publishing.
Weinstein, J. N., Myers, T., Casciari, J. J., Buolamwini, J.,
& Raghavan, K. (1994). Neural networks in the biomedical
sciences: A survey of 386 publications since the begin-
ning of 1991. Proceedings of the World Congress on
Neural Networks, 1 (pp. 121-126).
Werbos, P. J. (1974). Beyond regression: New tools for
prediction and analysis in the behavioral sciences. Cam-
bridge, MA: Harvard University, Ph.D. dissertation.
Wong, B. K., Jiang, L., & Lam, J. (2000). A bibliography of
neural network business application research: 1994 - 1998.
Computers and Operations Research, 27(11), 1045-1076.
Zhang, Q., Patuwo, B. E., & Hu, M. Y. (1998). Forecasting
with artificial neural networks: The state of the art. Inter-
national Journal of Forecasting, 14, 35-62.
KEY TERMS
Architecture: The configuration of the network of
neurons into layers of neurons is referred to as the neural
architecture. To specify the architecture involves declar-
ing the number of input variables, hidden layers, hidden
neurons, and output neurons.
Backpropagation: The name of the most common
learning algorithm for MFNNs. It involves modifying the
weights of the MFNN in such a way that the error (differ-
ence between the MFNN output and the training data
desired output) is minimized over time. The error at the
hidden neurons is approximated by propagating the out-
put error backwards, hence the name backpropagation.
Epoch: A complete presentation of all training data to
the MFNN is called one epoch. Typically a MFNN will
need to be trained for many thousands of epochs before
the error is reduced to an acceptable level.
Generalization: The goal of neural network training is
to develop a model that generalizes its knowledge to
unseen data. Overtraining must be avoided for generali-
zation to occur.
Hidden Neurons: The name given to the layer of
neurons between the input variables and the output
neuron. If too many hidden neurons are used, the neural
network may be easily overtrained. If too few hidden
neurons are used, the neural network may be unable to
learn the training data.
TEAM LinG
Neural Networks for Prediction and Classification
Learning Algorithm: The method used to change the
weights so that the error is minimized. Training data is
repeatedly presented to the MFNN through the input
layer, the output of the MFNN is calculated and compared
to the desired output. Error information is used to deter-
mine which weights need to be modified, and by how
much. There are several parameters involved in the learn-
ing algorithm including learning rate, momentum factor,
initial weights, etc.
Neural Model: Specifying a neural network model
involves declaring the architecture and activation func- N
tion types.
Overtraining: When the MFNN performs significantly
better on the training data than an out-of-sample test data,
it is considered to have memorized the training data, and
be overtrained. This can be avoided by following the
guidelines presented above.
869
TEAM LinG
870
Off-Line Signature Recognition
Indrani Chakravarty
Indian Institute of Technology, India
Nilesh Mishra
Indian Institute of Technology, India
Mayank Vatsa
Indian Institute of Technology, India
Richa Singh
Indian Institute of Technology, India
P. Gupta
Indian Institute of Technology, India
INTRODUCTION
The most commonly used protection mechanisms today
are based on either what a person possesses (e.g. an ID
card) or what the person remembers (like passwords and
PIN numbers). However, there is always a risk of pass-
words being cracked by unauthenticated users and ID
cards being stolen, in addition to shortcomings like for-
gotten passwords and lost ID cards (Huang & Yan, 1997).
To avoid such inconveniences, one may opt for the new
methodology of Biometrics, which though expensive will
be almost infallible as it uses some unique physiological
and/or behavioral (Huang & Yan, 1997) characteristics
possessed by an individual for identity verification. Ex-
amples include signature, iris, face, and fingerprint recog-
nition based systems.
The most widespread and legally accepted biometric
among the ones mentioned, especially in the monetary
transactions related identity verification areas is carried
out through handwritten signatures, which belong to
behavioral biometrics (Huang & Yan,1997). This tech-
nique, referred to as signature verification, can be classi-
fied into two broad categories - online and off-line. While
online deals with both static (for example: number of black
pixels, length and height of the signature) and dynamic
features (such as acceleration and velocity of signing,
pen tilt, pressure applied) for verification, the latter ex-
tracts and utilizes only the static features (Ramesh and
Murty, 1999). Consequently, online is much more efficient
in terms of accuracy of detection as well as time than off-
line. But, since online methods are quite expensive to
implement, and also because many other applications still
require the use of off-line verification methods, the latter,
though less effective, is still used in many institutions.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
BACKGROUND
Starting from banks, signature verification is used in many
other financial exchanges, where an organizations main
concern is not only to give quality services to its custom-
ers, but also to protect their accounts from being illegally
manipulated by forgers.
Forgeries can be classified into four typesrandom,
simple, skilled and traced (Ammar, Fukumura & Yoshida,
1988; Drouhard, Sabourin, & Godbout, 1996). Generally
online signature verification methods display a higher
accuracy rate (closer to 99%) than off-line methods (90-
95%) in case of all the forgeries. This is because, in off-line
verification methods, the forger has to copy only the
shape (Jain & Griess, 2000) of the signature. On the other
hand, in case of online verification methods, since the
hardware used captures the dynamic features of the
signature as well, the forger has to not only copy the
shape of the signature, but also the temporal characteris-
tics (pen tilt, pressure applied, velocity of signing etc.) of
the person whose signature is to be forged. In addition,
he has to simultaneously hide his own inherent style of
writing the signature, thus making it extremely difficult to
deceive the device in case of online signature verification.
Despite greater accuracy, online signature recogni-
tion is not encountered generally in many parts of the
world compared to off-line signature recognition, be-
cause it cannot be used everywhere, especially where
signatures have to be written in ink, e.g. on cheques,
where only off-line methods will work. Moreover, it re-
quires some extra and special hardware (e.g. pressure
sensitive signature pads in online methods vs. optical
scanners in off-line methods), which are not only expen-
sive but also have a fixed and short life span.
TEAM LinG
Off-Line Signature Recognition
MAIN THRUST
In general, all the current off-line signature verification
systems can be divided into the following sub-modules:
Data Acquisition
Preprocessing and Noise Removal
Feature Extraction and Parameter Calculations
Learning and Verification (or Identification)
Data Acquisition
Off-line signatures do not consider the time related as-
pects of the signature such as velocity, acceleration and
pressure. Therefore, they are often termed as static
signatures, and are captured from the source (i.e. paper)
using a camera or a high resolution scanner, in compari-
son to online signatures (in which data is captured using
a digitizer or an instrumented pen generating signals)
(Tappert, Suen, & Wakahara, 1990; Wessels & Omlin,
2000), which do consider the time related or dynamic
aspects besides the static features.
Preprocessing
The preprocessing techniques that are generally per-
formed in off-line signature verification methods com-
prise of noise removal, smoothening, space standardiza-
tion and normalization, thinning or skeletonization, con-
verting a gray scale image to a binary image, extraction of
the high pressure region images, etc.
Figure 1. Modular structure of an off-line verification
system
Signature
Cropped
scanned Image
Preprocessing module
Data Base Analyzer / Controller module
Learning module Feature Extraction
module
Figure 2. Noise removal using median filter
(a) Gray scale
Noise Removal: Signature images, like any other
image may contain noises like extra dots or pixels O
(Ismail & Gad, 2000), which originally do not belong
to the signature, but get included in the image
because of possible hardware problems or the pres-
ence of background noises like dirt. To recognize
the signature correctly, these noise elements have
to be removed from the background in order to get
the accurate feature matrices in the feature extrac-
tion phase. A number of filters have been used as
preprocessors (Ismail & Gad, 2000) by researchers
to obtain the noise free image. Examples include the
mean filter, median filter, filter based on merging
overlapped run lengths in one rectangle (Ismail &
Gad, 2000) etc. Among all the filtering techniques
mentioned above, average and median filtering are
considered to be standard noise reduction and iso-
lated peak noise removal techniques(Huang & Yan,
1997). However, median filter is preferred more be-
cause of its ability to remove noises without blur-
ring the edges of the signature instance unlike the
mean filter.
Space Standardization and Normalization: In
Space standardization, the distance between the
horizontal components of the same signature is
standardized, by removing blank columns, so that
it does not interfere with the calculation of global
and local features of the signature image (Baltzakis
& Papamarkos, 2001; Qi & Hunt, 1994). In nor-
malization, the signature image is scaled to a stan-
dard size which is the average size of all training
samples, keeping the width to height ratio constant
(Baltzakis & Papamarkos, 2001; Ismail & Gad,
2000; Ramesh & Murty, 1999).
Extracting the Binary Image from Grayscale
Image: Using the Otsus method a threshold is
calculated to obtain a binary version of the
grayscale image (Ammar, Fukumura, & Yoshida,
1988; Ismail & Gad, 2000; Qi & Hunt, 1994). The
algorithm is as follows:
Output
1 R ( x, y ) > threshold ,
S ( x, y ) =
Verification module 0 R ( x, y ) < threshold ,
Figure 3. Converting grayscale image into binary image
(b) Noise free (a) Gray scale (b) Binary
871
TEAM LinG

where S(x,y) is the binary image and R(x,y) is the
grayscale image.
Smoothening: The noise-removed image may have
small connected components or small gaps, which
may need to be filled up. It is done using a binary
mask obtained by thresholding followed by morpho-
logical operations (which include both erosion and
dilation) (Huang & Yan, 1997; Ismail & Gad, 2000;
Ramesh & Murty, 1999).
Extracting the High Pressure Region Image:
Ammar et al. (Ammar, M., Fukumura, T., & Yoshida
Y., 1986) have used the high pressure region as one
of the prominent features for detecting skilled
forgeries. It is the area of the image where the
writer gives special emphasis reflected in terms of
higher ink density (more specifically, higher gray
level intensities than the threshold chosen). The
threshold is obtained as follows:
Th HPR = I min + 0.75(I max I min ),
where Imin and Imax are the minimum and maximum
grayscale intensity values.
Thinning or Skeletonization: This process is carried
out to obtain a single pixel thick image from the
binary signature image. Different researchers have
used various algorithms for this purpose (Ammar,
Fukumura, & Yoshida, 1988; Baltzakis & Papamarkos,
2001; Huang & Yan, 1997; Ismail & Gad, 2000).
Feature Extraction
Most of the features can be classified in two categories -
global and local features.
Global Features: Ismail and Gad (2000) have de-
scribed global features as characteristics, which
identify or describe the signature as a whole. They
are less responsive to small distortions and hence
Figure 4. Extracting the high pressure region image
(a) Gray scale (b) High pressure region
Figure 5. Extraction of thinned image
(a) Binary (b) Thinned
image image
872
Off-Line Signature Recognition
are less sensitive to noise as well. Examples include
width and height of individual signature compo-
nents, width to height ratio, total area of black
pixels in the binary and high pressure region (HPR)
images, horizontal and vertical projections of sig-
nature images, baseline, baseline shift, relative
position of global baseline and centre of gravity
with respect to width of the signature, number of
cross and edge points, circularity, central line,
corner curve and corner line features, slant, run
lengths of each scan of the components of the
signature, kurtosis (horizontal and vertical), skew-
ness, relative kurtosis, relative skewness, relative
horizontal and vertical projection measures, enve-
lopes, individual stroke segments. (Ammar,
Fukumura, & Yoshida, 1990; Bajaj & Chaudhury,
1997; Baltzakis & Papamarkos, 2001, Fang, et al.,
2003, Huang & Yan, 1997; Ismail & Gad, 2000; Qi &
Hunt, 1994, Ramesh & Murty, 1999; Yacoubi,
Bortolozzi, Justino, & Sabourin, 2000; Xiao &
Leedham, 2002)
Local Features: Local features are confined to a
limited portion of the signature, which is obtained
by dividing the signature image into grids or treat-
ing each individual signature component as a sepa-
rate entity (Ismail & Gad, 2000). In contrast to
global features, they are responsive to small distor-
tions like dirt, but are not influenced by other
regions of the signature. Hence, though extraction
of local features requires more computations, they
are much more precise. Many of the global features
have their local counterparts as well. Examples of
local features include width and height of indi-
vidual signature components, local gradients such
as area of black pixels in high pressure region and
binary images, horizontal and vertical projections,
number of cross and edge points, slant, relative
position of baseline, pixel distribution, envelopes
etc. of individual grids or components. (Ammar,
Fukumura, & Yoshida, 1990; Bajaj & Chaudhury,
1997; Baltzakis & Papamarkos, 2001; Huang & Yan,
1997; Ismail & Gad, 2000; Qi & Hunt, 1994; Ramesh
& Murty, 1999; Yacoubi, Bortolozzi, Justino, &
Sabourin, 2000)
Learning and Verification
The next stage after feature extraction involves learning.
Though learning is not mentioned separately for verifi-
cation as a separate sub module, it is the most vital part
of the verification system in order to verify the authen-
ticity of the signature. Usually five to forty features are
passed in as a feature matrix/vector to train the system.
In the training phase, 3-3000 signature instances are
TEAM LinG
Off-Line Signature Recognition

Table 1. Summary of some prominent off-line papers

Mode of
O
Author Database Feature Extraction Results
Verification
Statistical Global baseline, upper and
10 genuine
Ammar, method-- lower extensions, slant
signatures per
Fukumara and Euclidian features, local features (e.g. 90% verification rate
person from 20
Yoshida (1988) distance and local slants) and pressure
people
threshold feature
Signature outline, core
Neural network feature, ink distribution,
Based-- A total of 3528 high pressure region,
99.5 % for random
Huang and Yan Multilayer signatures directional frontiers feature,
forgery and 90 % for
(1997) perception based (including genuine area of feature pixels in
targeted forgeries
neural networks and forged) core, outline, high pressure
trained and used region, directional frontiers,
coarse, and in fine ink
Global geometric features:
Aspect ratio, width without
blanks, slant angle, vertical
Genetic center of gravity (COG) etc.
Algorithms used Moment Based features: 90 % under genuine,
650 signatures, 20
for obtaining horizontal and vertical 98 % under random
Ramesh and genuine and 23
genetically projection images, kurtosis forgery and 70-80 %
Murty (1999) forged signatures
optimized measures( horizontal and under skilled forgery
from 15 people
weights for vertical) case
weighted feature Envelope Based:
vector extracting the lower and
upper envelopes
and
Wavelet Based features
Central line features, corner
220 genuine and line features, central circle 95% recognition rate
Ismail and Gad
Fuzzy concepts 110 forged features, corner curve and 98% verification
(2000)
samples features, critical points rate
features

Yacoubi, 4000 signatures

Average Error(%)
Bortolozzi, from 100 people Caliber, proportion,
Hidden Markov :0.48 on 1st database,
Justino and (40 people for 1st behavior guideline, base
Model and 0.88 on the 2nd
Sabourin and 60 for 2nd behavior, spacing
database
(2000) database

Nonliinear
dynamic time Horizontal and Vertical Best result of 18.1%
warping applied projections for non linear average error rate
Fang, Leung, 1320 genuine
to horizontal and dynamic time warping (average of FAR and
Tang, Tse, signatures from 55
vertical method. Skeletonized FRR) for first
Kwok, and authors and 1320
projections. image with approximation method. The
Wong (2003) forgeries from 12
Elastic bunch of the skeleton by short Average error rate
authors
graph matching lines for elastic bunch was 23.4% in case of
to individual graph matching algorithm the second method
stroke segments

used. These signature instances include either the genu-
ine or both genuine and forged signatures depending on
the method. All the methods extract features in a manner
such that the signature cannot be constructed back from
them in the reverse order but have sufficient data to
capture the features required for verification. These fea-
tures are stored in various formats depending upon the
system. It could be either in the form of feature values,
weights of the neural network, conditional probability
values of a belief (Bayesian) network or as a covariance
matrix (Fang et al., 2003). Later, when a sample signature
is to be tested, its feature matrix/vector is calculated, and
is passed into the verification sub module of the system,
which identifies the signature to be either authentic or
unauthentic. So, unless the system is trained properly,
chances, though less, are that it may recognize an authen-
tic signature to be unauthentic (false rejection), and in
certain other cases, recognize the unauthentic signature
to be authentic (false acceptance). So, one has to be
extremely cautious at this stage of the system. There are
various methods for off-line verification systems that are
in use today. Some of them include:

873

TEAM LinG

Statistical methods (Ammar, M., Fukumura, T., &
Yoshida Y., 1988; Ismail & Gad, 2000)
Neural network based approach (Bajaj & Chaudhury,
1997; Baltzakis & Papamarkos, 2001; Drouhard,
Sabourin, & Godbout, 1996; Huang & Yan, 1997)
Genetic algorithms for calculating weights (Ramesh
& Murty, 1999)
Hidden Markov Model (HMM) based methods
(Yacoubi, Bortolozzi, Justino, & Sabourin, 2000)
Bayesian networks (Xiao & Leedham, 2002)
Nonliinear dynamic time warping (in spatial domain)
and Elastic bunch graph matching (Fang et al., 2003)
The problem of signature verification is that of divid-
ing a space into two different sets of genuine and forged
signatures. Both online and off-line approaches use fea-
tures to do this, but, the problem with this approach is that
even two signatures by the same person may not be the
same. The feature set must thus have sufficient interper-
sonal variability so that we can classify the input signa-
ture as genuine or forgery. In addition, it must also have
a low intrapersonal variability so that an authentic signa-
ture is accepted. Solving this problem using fuzzy sets
and neural networks has also been tried. Another problem
is that an increase in the dimensionality i.e. using more
features does not necessarily minimize(s) the error rate.
Thus, one has to be cautious while choosing the appro-
priate/optimal feature set.
FUTURE TRENDS
Most of the presently used off-line verification methods
claim a success rate of more than 95% for random forgeries
and above 90% in case of skilled forgeries. Although, a
95% verification rate seems high enough, it can be noticed
that, even if the accuracy rate is as high as 99 %, when we
scale it to the size of a million, even a 1% error rate turns
out to be a significant number. It is therefore necessary to
increase this accuracy rate as much as possible.
CONCLUSION
Performance of signature verification systems is mea-
sured from their false rejection rate (FRR or type I error)
(Huang & Yan, 1997) and false acceptance rate (FAR or
type II error) (Huang & Yan, 1997) curves. Average error
rate, which is the mean of the FRR and FAR values, is also
used at times. Instead of using the FAR and FRR values,
many researchers quote the 100 - average rate values
as the performance result. Values for various approaches
have been mentioned in the table above. It is very diffi-
874
Off-Line Signature Recognition
cult to compare the values of these error rates, as there is
no standard database either for off-line or for online
signature verification methods.
Although online verification methods are gaining
popularity day by day because of higher accuracy rates,
off-line signature verification methods are still consid-
ered to be indispensable, since they are easy to use and
have a wide range of applicability. Efforts must thus be
made to improve its efficiency to as close as that of online
verification methods.
REFERENCES
Ammar, M., Fukumura, T., & Yoshida Y. (1986). A new
effective approach for off-line verification of signature
by using pressure features. Proceedings 8th Interna-
tional Conference on Pattern Recognition, ICPR86 (pp.
566-569), Paris.
Ammar, M., Fukumura, T., & Yoshida, Y. (1988). Off-line
preprocessing and verification of signatures. Interna-
tional Journal of Pattern Recognition and Artificial
Intelligence 2(4), 589-602.
Ammar, M., Fukumura, T., & Yoshida, Y. (1990). Structural
description and classification of signature images. Pat-
tern Recognition 23(7), 697-710.
Bajaj, R., & Chaudhury, S. (1997). Signature verification
using multiple neural classifiers. Pattern Recognition
30(1), l-7.
Baltzakis, H., & Papamarkos, N. (2001). A new signature
verification technique based on a two-stage neural net-
work classifier. Engineering Applications of Artificial
Intelligence, 14, 95-103.
Drouhard, J. P., Sabourin, R., & Godbout, M. (1996). A
neural network approach to off-line signature verifica-
tion using directional pdf. Pattern Recognition 29(3),
415-424.
Fang, B., Leung, C. H., Tang, Y. Y., Tse, K. W., Kwok, P.
C. K., & Wong, Y. K. (2003). Off-line signature verification
by tracking of feature and stroke positions. Pattern Rec-
ognition 36, 91-101.
Huang, K., & Yan, H. (1997). Off-line signature verification
based on geometric feature extraction and neural network
classification. Pattern Recognition, 30(1), 9-17.
Ismail, M. A., & Gad, S. (2000). Off-line Arabic signature
recognition and verification. Pattern Recognition, 33,
1727-1740.
TEAM LinG
Off-Line Signature Recognition
Jain, A. K. & Griess, F. D., (2000). Online Signature
Verification. Project Report, Department of Computer
Science and Engineering, Michigan State University, USA.
Qi, Y., & Hunt, B. R. (1994). Signature verification using
global and grid features. Pattern Recognition, 27(12),
1621-1629.
Ramesh, V.E., & Murty, M. N. (1999). Off-line signature
verification using genetically optimized weighted fea-
tures. Pattern Recognition, 32(7), 217-233.
Tappert, C. C., Suen, C. Y., & Wakahara, T. (1990).The
State of the Art in Onliine Handwriting Recognition. IEEE
Transactions on Pattern Analysis and Machine Intelli-
gence, 12(8).
Wessels, T., & Omlin, C. W, (2000). A Hybrid approach for
Signature Verification. International Joint Conference
on Neural Networks.
Yacoubi, A. El., Bortolozzi, F., Justino, E. J. R., & Sabourin,
R. (2000). An off-line signature verification system using
HMM and graphometric features. 4th IAPR International
Workshop on Document Analysis Systems (pp. 211-222).
Xiao, X., & Leedham, G. (2002). Signature verification
using a modified Bayesian network. Pattern Recognition,
35, 983-995.
KEY TERMS
Area of Black Pixels: It is the total number of black
pixels in the binary image.
Biometric Authentication: The identification of indi-
viduals using their physiological and behavioral charac-
teristics.
Centre of Gravity: The centre of gravity of the image
is calculated as per the following equation:
n n
= (x.Pv [x ])/ Pv [x ],
m m
X = (y.Ph [y])/ Ph [y], and Y
y =1 y =1 x =1 x =1 O
where X, Y are the x and y coordinates of the centre of
gravity of the image, and Ph and Pv are the horizontal and
vertical projections respectively.
Cross and Edge Points: A cross point is an image pixel
in the thinned image having at least three eight neighbors,
while an edge point has just one eight neighbor in the
thinned image.
Envelopes: Envelopes are calculated as upper and
lower envelopes above and below the global baseline.
These are the connected pixels which form the external
points of the signature obtained by sampling the signa-
ture at regular intervals.
Global Baseline: It is the median value of pixel distri-
bution along the vertical projection.
Horizontal and Vertical Projections: Horizontal pro-
jection is the projection of the binary image along the
horizontal axis. Similarly, vertical projection is the projec-
tion of the binary image along the vertical axis. They can
be calculated as follows:
m
Ph [y] = black.pixel(x, y),Pv [x ] = black.pixel(x, y),
n
x =1 y =1
where m = width of the image and n = height of the
image.
Moment Measures: Skewness and kurtosis are mo-
ment measures calculated using the horizontal and verti-
cal projections and co-ordinates of centre of gravity of the
signature.
Slant: Either it is defined as the angle at which the
image has maximum horizontal projection value on rota-
tion or it is calculated using the total number of positive,
negative, horizontally or vertically slanted pixels.
875
TEAM LinG
876
Online Analytical Processing Systems
Rebecca Boon-Noi Tan
Monash University, Australia
INTRODUCTION
Since its origin in the 1970s research and development
into databases systems has evolved from simple file
storage and processing systems to complex relational
databases systems, which have provided a remarkable
contribution to the current trends or environments. Data-
bases are now such an integral part of day-to-day life that
often people are unaware of their use. For example, pur-
chasing goods from the local supermarket is likely to
involve access to a database. In order to retrieve the price
of the item, the application program will access the prod-
uct database. A database is a collection of related data and
the database management system (DBMS) is software
that manages and controls access to the database (Elmasri
& Navathe, 2004).
BACKGROUND
Data Warehouse
A data warehouse is a specialized type of database. More
specifically, a data warehouse is a repository (or archive)
of information gathered from multiple sources, stored
under a unified schema, at a single site (Silberschatz,
Korth, & Sudarshan, 2002, p. 843). Chaudhuri and Dayal
(1997) consider that a data warehouse should be sepa-
rately maintained from the organizations operational
database since the functional and performance require-
ments of online analytical processing (OLAP) supported
by data warehouses are quite different from those of the
online transaction processing (OLTP) traditionally sup-
ported by the operational database.
Figure 1. OLTP system
Online
Transaction
Processing
(OLTP)
system
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
OLAP Versus OLTP
Two reasons why traditional OLTP is not suitable for data
warehousing are presented: (a) Given that operational
databases are finely tuned to support known OLTP
workloads, trying to execute complex OLAP queries against
the operational databases would result in unacceptable
performance. Furthermore, decision support requires data
that might be missing from the operational databases; for
instance, understanding trends or making predictions
requires historical data, whereas operational databases
store only current data. (b) Decision support usually
requires consolidating data from many heterogeneous
sources: these might include external sources such as
stock market feeds, in addition to several operational
databases. The different sources might contain data of
varying quality, or use inconsistent representations,
codes and formats, which have to be reconciled.
Traditional Online Transaction
Processing (OLTP)
Traditional relational databases have been used primarily
to support OLTP systems. The transactions in an OLTP
system usually retrieve and update a small number of
records accessed typically on their primary keys. Opera-
tional databases tend to be hundreds of megabytes to
gigabytes in size and store only current data (Ramakrishnan
& Gehrke, 2003).
Figure 1 shows a simple overview of the OLTP system.
The operational database is managed by a conventional
relational DBMS. OLTP is designed for day-to-day opera-
tions. It provides a real-time response. Examples include
Internet banking and online shopping.
TEAM LinG
Online Analytical Processing Systems
Online Analytical Processing (OLAP)
OLAP is a term that describes a technology that uses a
multi-dimensional view of aggregate data to provide quick
access to strategic information for the purposes of ad-
vanced analysis (Ramakrishnan & Gehrke, 2003).
OLAP supports queries and data analysis on aggre-
gated databases built in data warehouses. It is a system
for collecting, managing, processing and presenting mul-
tidimensional data for analysis and management pur-
poses (Figure 2). There are two main implementation
methods to support OLAP applications: relational OLAP
(ROLAP) and multidimensional OLAP (MOLAP).
ROLAP
Relational online analytical processing (ROLAP) pro-
vides OLAP functionality by using relational databases
and familiar relational query tools to store and analyse
multidimensional data (Ramakrishnan & Gehrke, 2003).
Entity Relationship diagrams and normalization tech-
niques are popularly used for database design in OLTP
environments. However, the database designs recom-
mended by ER diagrams are inappropriate for decision
support systems where efficiency in querying and in
loading data (including incremental loads) are crucial. A
special schema known as a star schema is used in an OLAP
environment for performance reasons (Martyn, 2004).
This star schema usually consists of a single fact table
and a dimension table for each dimension (Figure 3).
MOLAP
Multidimensional online analytical processing (MOLAP)
extends OLAP functionality to multidimensional data-
base management systems (MDBMSs). A MDBMS uses
Figure 2. OLAP system
Online
Analytical
Processing
(OLAP)
System
special proprietary techniques to store data in matrix-like
n-dimensional arrays (Ramakrishnan & Gehrke, 2003). O
The multi-dimensional data cube is implemented by
the arrays with the dimensions forming the axes of the
cube (Sarawagi, 1997). Therefore, only the data value
corresponding to a data cell is stored as direct mapping.
MOLAP servers have excellent indexing properties due to
the fact that looking for a cell is simple array lookups rather
than associative lookups in tables. But unfortunately it
provides poor storage utilization, especially when the
data set is sparse.
In a multi-dimensional data model, the focal point is on
a collection of numeric measures. Each measure depends
on a set of dimensions. For instance, the measure attribute
is amt as shown in Figure 4. Sales information is being
arranged in a three-dimensional array of amt. Figure 4
shows that the array only shows the values for single L#
value where L# = L001, which presented as a slice or-
thogonal to the L# axis.
Cube-By Operator
In decision support database systems, aggregation is a
commonly used operation. As previously mentioned,
current SQL can be very inefficient. Thus, to effectively
support decision support queries in OLAP environment,
a new operator, Cube-by was proposed by (Gray,
Bosworth, Lyaman, & Pirahesh, 1996). It is an extension
of the relational operator Group-by. The Cube-by opera-
tor computes Group-by corresponding to all possible
combinations of attributes in the Cube-by clause.
In order to see how a data cube is formed, an example
is provided in Figure 5. It shows an example of data cube
formation through executing the cube statement at the top
left of the figure. Figure 5 presents two way of presenting
the aggregated data: (a) a data cube, and (b) a 3D data cube
in table form.
877
TEAM LinG

Figure 3. Star schema showing that location, product, date, and sales are represented as relations
Figure 4. SALES presented as a multidimensional dataset
MAIN THRUST
Processing Level in OLAP Systems
The processing level where the execution of the raw data
or aggregated data takes place before presenting the result
to the user is shown in Figure 6. The user is only interested
in the data in the report in a fast response. However, the
background of the processing level is needed in order to
provide the user with the result of the aggregated data in
an effective and efficient way.
The processing level has been broken down into two
distinct areas: back-end processing and front-end pro-
878
Online Analytical Processing Systems
cessing. Back-end processing basically deals with the
raw data, which is stored in either tables (ROLAP) or
arrays (MOLAP) and then process it into aggregated
data, which is presented to the user. Front-end process-
ing is computing the raw data and managing the pre-
computed aggregated data in either in 3D data cube or n-
dimensional table.
It is important to mention that front-end processing
is similar to back-end processing as it also deals with raw
data. It needs to construct or execute the raw data into the
data cube and then store the pre-computed aggregated
data in either the memory or disk. It is convenient for the
user if the pre-computed aggregated data is ready and is
TEAM LinG
Online Analytical Processing Systems

Figure 5. An example of data cube formation through executing the cube statement at the top left of the figure
O

Figure 6. Two interesting areas of the processing level in OLAP environment

Online
Analytical
Processing
(OLAP)
System

879

TEAM LinG

stored in the memory whenever the user requests for it.
The difference between the two processing levels is
that the front-end has the pre-computed aggregated data
in the memory which is ready for the user to use or analyze
it at any given time. On the other hand the back-end
processing computes the raw data directly whenever
there is a request from the user. This is why the front-end
processing is considered to present the aggregated data
faster or more efficiently than back-end processing. How-
ever, it is important to note that back-end processing and
front-end processing are basically one whole processing.
The reason behind this break down of the two processing
levels is because the problem can be clearly seen in back-
end processing and front-end processing. In the next
section, the problems associated with each areas and the
related work in improving the problems will be considered.
Back-End Processing
Back-end processing involves basically dealing with
the raw data, which is stored in either tables (ROLAP) or
arrays (MOLAP) as shown in Figure 7. The user queries
the raw data for decision-making purpose. The raw data is
then processed and computed into aggregated data, which
will be presented to the user for analyzing. Generally the
basic stage: extracting, is followed by two sub-stages:
indexing, and partitioning in back-end processing.
Extracting is the process of querying the raw data
either from tables (ROLAP) or arrays (MOLAP) and com-
puting it. The process of extracting is usually time-con-
suming. Firstly, the database (data warehouse) size is
extremely large and secondly the computation time is
equally high. However, the user is only interested in a fast
Figure 7. Stages in back-end processing
880
Online Analytical Processing Systems
response time for providing the resulted data. Another
consideration is that the analyst or manager using the
data warehouse may have time constraints. There are two
sub-stages or fundamental methods of handling the data:
(a) indexing and (b) partitioning in order to provide the
user with the resulted data in a reasonable time frame.
Indexing has existed in databases for many decades.
Its access structures have provided faster access to the
base data. For retrieval efficiency, index structures would
typically be defined especially in data warehouses or
ROLAP where the fact table is very large (Datta,
VanderMeer & Ramamritham, 2002). ONeil and Quass
(1997) have suggested a number of important indexing
schemes for data warehousing including bitmap index,
value-list index, projection index, data index. Data index is
similar to projection index but it exploits a positional
indexing strategy (Datta, VanderMeer, & Ramamritham,
2002). Interestingly MOLAP servers have better indexing
properties than ROLAP servers since they look for a cell
using simple array lookups rather than associative look-
ups in tables.
Partitioning of raw data is more complex and challeng-
ing in data warehousing as compared to that of relational
and object databases. This is due to the several choices
of partitioning of a star schema (Datta, VanderMeer, &
Ramamritham, 2002). The data fragmentation concept in
the context of distributed databases aims to reduce query
execution time and facilitate the parallel execution of
queries (Bellatreche, Karlapalem, Mohania, & Schneide,
2000).
In a data warehouse or ROLAP, either the dimension
tables or the fact table or even both can be fragmented.
Bellatreche et al. (2000) have proposed a methodology for
applying the fragmentation techniques in a data ware-
house star schema to reduce the total query execution
cost. The data fragmentation concept in the context of
distributed databases aims to reduce query execution time
and facilitates the parallel execution of queries.
Front-End Processing
Front-end processing is computing the raw data and
managing the pre-computed aggregated data in either in
3D data cube or n-dimensional table. The three basic
types of stages that are involved in Front-end processing
are shown in Figure 8. They are (i) constructing, (ii)
storing and (iii) querying. Constructing is basically the
computation process of a data cube by using cube-by
operator. Cube-by is an expensive approach, especially
when the number of Cube-by attributes and the database
size are large.
The difference between constructing and extracting
needs to be clearly defined. Constructing in the Front-end
processing is similar to the extracting in the Back-end
TEAM LinG
Online Analytical Processing Systems
Figure 8. Stages in front-end processing
processing as both of them are basically querying the raw
data. However, in this case, extracting concentrates on
how the fundamental methods can help in handling the
raw data in order to provide efficient retrieval. Construct-
ing in the Front-end processing concentrates on the cube-
by operator that involves the computation of raw data.
Storing is the process of putting the aggregated data
into either the n-dimension table of rows or the n-dimen-
sional arrays or data cube. There are two parts within the
storing process. One part is to store temporary raw data
in the memory for executing purpose and other part is to
store pre-computed aggregated data. Hence, there are two
problems related to the storage: (a) insufficient memory
space due to the loaded raw data and also in addition to
the incremental loads; (b) poor storage utilization the
array may not fit into memory especially when the data set
is sparse.
Querying is the process of extracting useful informa-
tion from the pre-computed data cube or n-dimensional
table for decision makers. It is important to take note that
Querying is also part of Constructing process. Querying
also makes use of cube-by operator that involves the
computation of raw data. ROLAP is able to support ad hoc
requests and allows unlimited access to dimensions un-
like MOLAP only allows limited access to predefined
dimensions. Despite the fact that it is able to support ad
hoc requests and access to dimensions, certain queries
might be difficult to fulfill the need of decision makers and
also to reduce the querying execution time when there is
n-dimensional query as the time factor is important to
decision makers.
To conclude, the three stages and their associated
problems in the OLAP environment have been outlined.
First, Cube-by is an expensive approach, especially when
O
the number of Cube-by attributes and the database size
are large. Second, storage has insufficient memory space
this is due to the loaded data and also in addition to the
incremental loads. Third, storage is not properly utilized
the array may not fit into memory especially when the
data set is sparse. Fourth, certain queries might be diffi-
cult to fulfill the need of decision makers. Fifth, the
execution querying time has to be reduced when there is
n-dimensional query as the time factor is important to
decision makers. However, it is important to consider that
there is scope for other possible problems to be identified.
FUTURE TRENDS
Problems have been identified in each of the three stages,
which have generated considerable attention from re-
searchers to find solutions. Several researchers have
proposed a number of algorithms to solve these cube-by
problems. Examples include:
Constructing
Cube-by operator is an expensive approach, especially
when the number of Cube-by attributes and the database
size are large.
Fast Computation Algorithms
There are algorithms aimed at fast computation of large
sparse data cube (Ross & Srivastava, 1997; Beyer &
Ramakrishnon, 1999). Ross and Srivastava (1997) have
taken into consideration the fact that real data is fre-
881
TEAM LinG

quently sparse. (Ross & Srivastava, 1997) partitioned
large relations into small fragments so that there was
always enough memory to fit in the fragments of large
relation. Whereas, Beyer and Ramakrishnan (1999) pro-
posed the bottom-up method to help reduce the penalty
associated with the sorting of many large views.
Parallel Processing System
The assumption of most of the fast computation algo-
rithms is that their algorithms can be applied into the
parallel processing system. Dehne, Eavis, Hambrusch,
and Rau-Chaplin. (2002) presented a general methodol-
ogy for the efficient parallelization of exiting data cube
construction algorithms. Their paper described two dif-
ferent partitioning strategies, one for top-down and one
for bottom-up cube algorithms. They provide a good
summary and comparison with other parallel data cube
computations (Ng, Wagner, & Yin., 2001; Yu & Lu, 2001).
In Ng et al. (2001), the approach considers the
parallelization of sort-based data cube construction,
whereas Yu and Lu (2001) focuses on the overlap between
multiple data cube computations in a sequential setting.
Storing
Two problems related to the storage: (a) insufficient
memory space; (b) poor storage utilization.
Data Compression
Another technique is related to data compression.
Lakshmanan, Pei, and Zhao (2003) have developed a
systematic approach to achieve efficacious data cube
construction and exploration by semantic summarization
and compression.
Figure 9. Working areas in the OLAP system
882
Online Analytical Processing Systems
Condensed Data Cube
Wang, Feng, Lu, and Yu (2002) have proposed a new
concept called a condensed data cube. This new ap-
proach reduces the size of data cube and hence its com-
putation time. They make use of single base tuple
compression to generate a condensed cube so it is
smaller in size.
Querying
It might be difficult for certain queries to fulfill both the
need of the decision makers and also the need to reduce
the querying execution time when there is n-dimensional
query as the time factor is important to decision makers.
Types of Queries
Other work has focused on specialized data structures for
fast processing of special types of queries. Lee, Ling and
Li (2000) have focused on range-max queries and have
proposed hierarchical compact cube to support the range-
max queries. The hierarchical structure stores not only the
maximum value of all the children sub-cubes, but also
stores one of the locations of the maximum values among
the children sub-cubes. On the other hand, Tan, Taniar,
and Lu (2004) focus on data cube queries in term of SQL
and have presented taxonomy of data cube queries. They
also provide a comparison with different type queries.
Parallel Processing Techniques
Taniar and Tan (2002) have proposed three parallel tech-
niques to improve the performance of the data cube
queries. However, the challenges in this area continue to
grow with the cube-by problems.
TEAM LinG
Online Analytical Processing Systems
CONCLUSION
In this overview, the OLAP systems in general have been
considered, followed by processing level in the OLAP
systems and lastly the related and future work in the
OLAP systems. A number of problems and solutions in
the OLAP environment have been presented. However,
consideration needs to be given to the possibility that
other problems maybe identified which in turn will present
new challenges for the researchers to address.
REFERENCES
Bellatreche, L., Karlapalem, K., Mohania M., & Schneider
M. (2000, September). What can partitioning do for your
data warehouses and data marts? International IDEAS
conference (pp. 437-445), Yokohoma, Japan.
Beyer, K.S., & Ramakrishnan, R. (1999, June). Bottom-up
computation of sparse and iceberg cubes. International
ACM SIGKDD conference (pp. 359-370), Philadelphia,
PA.
Chaudhuri, S., & Dayal, U. (1997). An overview of data
warehousing and OLAP technology. ACM SIGMOD
Record, 26, 65-74.
Datta, A., VanderMeer, D., & Ramamritham, K. (2002).
Parallel Star Join + DataIndexes: Efficient Query Process-
ing in Data Warehouses and OLAP. IEEE Transactions
on Knowledge & Data Engineering, 14(6), 1299-1316.
Dehne, F., Eavis, T., Hambrusch, S., & Rau-Chaplin, A.
(2002). Parallelizing the data cube. International Journal
of Distributed & Parallel Databases, 11, 181-201.
Elmasri, R., & Navathe, S.B. (2004). Fundamentals of
database systems. Boston, MA: Addison Wesley.
Gray, J., Bosworth, A., Lyaman, A. & Pirahesh, H. (1996,
June). Data cube: A relational aggregation operator gen-
eralizing GROUP-BY, CROSS-TAB, and SUB-TOTALS.
International ICDE Conference (pp. 152-159), New Or-
leans, Louisiana.
Lakshmanan, L.V.S., Pei, J., & Zhao, Y. (2003, September).
Efficacious data cube exploration by semantic
semmarization and compression. International VLDB
Conference (pp. 1125-1128), Berlin, Germany.
Lee, S.Y, Ling, T.W., & Li, H.G. (2000, September). Hierar-
chical compact cube for range-max queries. International
VLDB Conference (pp. 232-241), Cairo, Egypt.
Martyn, T. (2004). Reconsidering multi-dimensional
schemas. ACM SIGMOD Record, 83-88.
Ng, R.T., Wagner, A., & Yin, Y. (2001, May). Iceberg-cube
computation with pc clusters. International ACM O
SIGMOD Conference (pp. 25-36), Santa Barbara, Califor-
nia.
ONeil, P., & Graefe, G. (1995). Multi-table joins through
bit-mapped join indices. SIGMOD Record, 24(3), 8-11.
Ramakrishnan, R., & Gehrke, J. (2003). Database manage-
ment systems. NY: McGraw-Hill.
Ross, K.A., & Srivastava, D. (1997, August). Fast compu-
tation of sparse datacubes. International VLDB Confer-
ence (pp. 116-185), Athens, Greece.
Sarawagi, S. (1997). Indexing OLAP data. IEEE Data
Engineering Bulletin, 20(1), 36-43.
Silberschatz, A., Korth, H., & Sudarshan, S. (2002). Data-
base system concepts. NY: McGraw-Hill.
Taniar, D., & Tan, R.B.N. (2002, May). Parallel processing
of multi-join expansion-aggregate data cube query in high
performance database systems. International I-SPAN
Conference (pp. 51-58), Manila, Philippines.
Tan, R.B.N., Taniar, D., & Lu, G.J. (2004). A Taxonomy for
Data Cube Queries, International Journal for Computers
and Their Applications, 11(3), 171-185.
Wang, W., Feng, J.L., Lu, H.J., & Yu, J.X. (2002, February).
Condensed Cube: An effective approach to reducing data
cube size. International Data Engineering Conference
(pp. 155-165), San Jose, California.
Yu, J.X., & Lu, H.J. (2001, April). Multi-cube computation.
International DASFAA Conference (pp. 126-133), Hong
Kong, China.
KEY TERMS
Back-End Processing: Is dealing with the raw data,
which is stored in either tables (ROLAP) or arrays
(MOLAP).
Data Cube Operator: Computes Group-by correspond-
ing to all possible combinations of attributes in the Cube-
by clause.
Data Warehouse: A type of database A subject-
oriented, integrated, time-varying, non-volatile collec-
tion of data that issued primarily in organizational deci-
sion making (Elmasri & Navathe, 2004, p.900).
Front-End Processing: Is computing the raw data and
managing the pre-computed aggregated data in either in
3D data cube or n-dimensional table.
883
TEAM LinG
 Online Analytical Processing Systems

Multidimensional OLAP (MOLAP): Extends OLAP Relational OLAP (ROLAP): Provides OLAP func-
functionality to multidimensional database management tionality by using relational databases and familiar rela-
systems (MDBMSs). tional query tools to store and analyse multidimensional
data.
Online Analytical Processing (OLAP): Is a term used
to describe the analysis of complex data from data ware-
house (Elmasri & Navathe, 2004, p.900).

884

TEAM LinG

Online Signature Recognition
Indrani Chakravarty
Indian Institute of Technology, India
Nilesh Mishra
Indian Institute of Technology, India
Mayank Vatsa
Indian Institute of Technology, India
Richa Singh
Indian Institute of Technology, India
P. Gupta
Indian Institute of Technology, India
INTRODUCTION
Security is one of the major issues in todays world and
most of us have to deal with some sort of passwords in
our daily lives; but, these passwords have some prob-
lems of their own. If one picks an easy-to-remember
password, then it is most likely that somebody else may
guess it. On the other hand, if one chooses too difficult
a password, then he or she may have to write it some-
where (to avoid inconveniences due to forgotten pass-
words) which may again lead to security breaches. To
prevent passwords being hacked, users are usually ad-
vised to keep changing their passwords frequently and
are also asked not to keep them too trivial at the same
time. All these inconveniences led to the birth of the
biometric field. The verification of handwritten signa-
ture, which is a behavioral biometric, can be classified
into off-line and online signature verification methods.
Online signature verification, in general, gives a higher
verification rate than off-line verification methods, be-
cause of its use of both static and dynamic features of
the problem space in contrast to off-line which uses
only the static features. Despite greater accuracy, online
signature recognition is not that prevalent in compari-
son to other biometrics. The primary reasons are:
It cannot be used everywhere, especially where
signatures have to be written in ink; e.g. on cheques,
only off-line methods will work.
Unlike off-line verification methods, online meth-
ods require some extra and special hardware, e.g.
electronic tablets, pressure sensitive signature
pads, etc. For off-line verification method, on the
other hand, we can do the data acquisition with
optical scanners.
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
885
O
The hardware for online are expensive and have a
fixed and short life cycle.
In spite of all these inconveniences, the use online
methods is on the rise and in the near future, unless a
process requires particularly an off-line method to be
used, the former will tend to be more and more popular.
BACKGROUND
Online verification methods can have an accuracy rate
of as high as 99%. The reason behind is its use of both
static and dynamic (or temporal) features, in compari-
son to the off-line, which uses only the static features
(Ramesh & Murty, 1999). The major differences be-
tween off-line and online verification methods do not
lie with only the feature extraction phases and accuracy
rates, but also in the modes of data acquisition, prepro-
cessing and verification/recognition phases, though the
basic sequence of tasks in an online verification (or
recognition) procedure is exactly the same as that of the
off-line. The phases that are involved comprise of:
Data Acquisition
Preprocessing and Noise Removal
Feature Extraction and
Verification (or Identification)
However, online signatures are much more difficult
to forge than off-line signatures (reflected in terms of
higher accuracy rate in case of online verification meth-
ods), since online methods involve the dynamics of the
signature such as the pressure applied while writing, pen
tilt, the velocity with which the signature is done etc. In
TEAM LinG
 Online Signature Recognition

case of off-line, the forger has to copy only the shape
(Jain & Griess, 2000) of the signature. On the other
hand, in case of online, the hardware used captures the
dynamic features of the signature as well. It is extremely
difficult to deceive the device in case of dynamic fea-
tures, since the forger has to not only copy the charac-
teristics of the person whose signature is to be forged,
but also at the same time, he has to hide his own inherent
style of writing the signature. There are four types of
forgeries: random, simple, skilled and traced forgeries
(Ammar, Fukumura, & Yoshida, 1988; Drouhard,
Sabourin, & Godbout, 1996). In case of online signa-
tures, the system shows almost 100% accuracy for the
first two classes of forgeries and 99% in case of the
latter. But, again, a forger can also use a compromised
signature-capturing device to repeat a previously re-
corded signature signal. In such extreme cases, even
online verification methods may suffer from repetition
attacks when the signature-capturing device is not physi-
cally secure.
Data Acquisition
Data acquisition (of the dynamic features) in online
verification methods is generally carried out using spe-
cial devices called transducers or digitizers (Tappert,
Suen, & Wakahara, 1990, Wessels & Omlin, 2000), in
contrast to the use of high resolution scanners in case of
off-line. The commonly used instruments include the
electronic tablets (which consist of a grid to capture the
x and y coordinates of the pen tip movements), pressure
sensitive tablets, digitizers involving technologies such
as acoustic sensing in air medium, surface acoustic
waves, triangularization of reflected laser beams, and
optical sensing of a light pen to extract information
about the number of strokes, velocity of signing, direc-
tion of writing, pen tilt, pressure with which the signa-
ture is written etc.
Preprocessing

Preprocessing in online is much more difficult than in

MAIN THRUST
Although the basic sequence of tasks in online signature
verification is almost the same as that of off-line meth-
ods, the modes differ from each other especially in the
ways the data acquisition, preprocessing and feature
extraction are carried out. More specifically, the sub-
modules of online are much more difficult with respect
to off-line (Jain & Griess, 2000). Figure 1 gives a
generic structure of an online signature verification
system. The online verification system can be classified
into the following modules:
Data Acquisition,
Preprocessing,
Feature Extraction,
Learning and Verification.
off-line, because it involves both noise removal (which
can be done using hardware or software) (Plamondon &
Lorette, 1989) and segmentation in most of the cases.
The other preprocessing steps that can be performed are
signal amplifying, filtering, conditioning, digitizing,
resampling, signal truncation, normalization, etc. How-
ever, the most commonly used include:
External Segmentation: Tappert, Suen and
Wakahara (1990) define external segmentation as
the process by which the characters or words of a
signature are isolated before the recognition is
carried out.
Resampling: This process is basically done to
ensure uniform smoothing to get rid of the redun-
dant information, as well as to preserve the re-
quired information for verification by comparing

Figure 1. Modular structure of a generic online verification system

Input device
/ interface

Preprocessing module

Data Base Analyzer / Controller module Output

Learning module Feature Extraction Verification module

module

886

TEAM LinG
Online Signature Recognition
the spatial data of two signatures. According to Jain
and Griess (2000), here, the distance between two
critical points is measured, and if the total distance
exceeds a threshold called the resampling length(
which is calculated by dividing the distance by the
number of sample points for that segment), then a
new point is created by using the gradient between
the two points,
Noise Reduction: Noise is nothing but irrelevant
data, usually in the form of extra dots or pixels in
images (in case of off-line verification methods)
(Ismail & Gad, 2000), which do not belong to the
signature, but are included in the image (in case of
off-line) or in the signal (in case of online), be-
cause of possible hardware problems (Tappert, Suen,
& Wakahara, 1990) or presence of background
noises like dirt or by faulty hand movements
(Tappert, Suen, & Wakahara, 1990) while signing.
The general techniques used are:
Smoothening: in which a point is averaged with
its neighbors. (Tappert, Sue, & Wakahara, 1990)
Filtering: Online thinning (Tappert, Suen, &
Wakahara, 1990) is not the same as off-line
thinning (Baltzakis & Papamarkos, 2001), al-
though, both decrease the number of irrelevant
dots or points. According to Tappert, Suen and
Wakahara (1990), it can be carried out either by
forcing a minimum distance between adjacent
points or by forcing a minimum change in the
direction of the tangent to the drawing for con-
secutive points.
Wild Point Correction: This removes noises
(points) caused by hardware problems (Tappert,
Suen, & Wakahara, 1990).
Dehooking: Dehooking on the other hand re-
moves hook like features from the signature,
which usually occur at the start or end of the
signature (Tappert, Suen, & Wakahara, 1990).
Dot Reduction: The dot size is reduced to
single points. (Tappert, Suen, & Wakahara, 1990)
Normalization: Off-line normalization often lim-
its itself to scaling the image to a standard size and/
or removing blank columns and rows. In case of
online, normalization includes deskewing, baseline
drift correction (orienting the signature to hori-
zontal), size normalization and stroke length nor-
malization (Ramesh & Murty, 1999; Tappert, Suen,
& Wakahara, 1990; Wessels & Omlin, 2000).
Feature Extraction
O
Online signature extracts both the static and the dy-
namic features. Some of the static and dynamic fea-
tures have been listed below.
Static Features: Although both static and dy-
namic information are available to the online
verification system, in most of the cases, the
static information is discarded, as dynamic fea-
tures are quite rich and they alone give high
accuracy rates. Some of the static features used
by online method are:
Width and height of the signature parts (Ismail
& Gad, 2000)
Width to height ratio (Ismail & Gad, 2000)
Number of cross and edge points (Baltzakis &
Papamarkos, 2001)
Run lengths of each scan of the components of
the signature (Xiao & Leedham, 2002)
Kurtosis (horizontal and vertical), skewness,
relative kurtosis, relative skewness, relative
horizontal and vertical projection measures
(Bajaj & Chaudhury, 1997; Ramesh & Murty,
1999)
Envelopes: upper and lower envelope fea-
tures (Bajaj & Chaudhury, 1997; Ramesh &
Murty, 1999)
Alphabet Specific Features: Like presence
of ascenders, descenders, cusps, closures, dots
etc. (Tappert, Suen, & Wakahara, 1990)
Dynamic Features: Though online methods uti-
lize some of the static features, they give more
emphasis to the dynamic features, since these
features are more difficult to imitate.
The most widely used dynamic features include:
Position (ux(t) and uy(t) ): e.g. the pen tip
position when the tip is in the air, and when it
is in the vicinity of the writing surface etc (Jain
& Griess, 2000; Plamondon & Lorette, 1989).
Pressure (u p (t)) (Jain & Griess, 2000;
Plamondon & Lorette, 1989)
Forces (uf(t), ufx(t), ufy(t)): Forces are com-
puted from the position and pressure coordi-
nates (Plamondon & Lorette, 1989).
Velocity (u v(t), uvx(t), uvy(t)): It can be de-
rived from position coordinates (Jain & Griess,
2000; Plamondon & Lorette, 1989).
887
TEAM LinG

Absolute and relative speed between two criti-
cal points (Jain, Griess, & Connell, 2002)
Acceleration (ua(t), u ax(t), uay(t)): Accelera-
tion can be derived from velocity or position
coordinates. It can also be computed using an
accelerometric pen (Jain & Griess, 2000;
Plamondon & Lorette, 1989).
Parameters: A number of parameters like num-
ber of peaks, starting direction of the signature,
number of pen lifts, means and standard devia-
tions, number of maxima and minima for each
segment, proportions, signature path length,
path tangent angles etc. are also calculated apart
from the above mentioned functions to increase
the dimensionality (Gupta, 1997; Plamondon &
Lorette, 1989; Wessels & Omlin, 2000). More-
over, all these features can be of both global and
local in nature.
Verification and Learning
For online, examples of comparison methods include
use of:
Corner Point and Point to Point Matching algo-
rithms (Zhang, Pratikakis, Cornelis, & Nyssen,
2000)
Similarity measurement on logarithmic spectrum
(Lee, Wu, & Jou, 1998)
Extreme Points Warping (EPW) (Feng & Wah,
2003)
String Matching and Common threshold (Jain,
Griess, & Connell, 2002)
Split and Merging, (Lee, Wu, & Jou, 1997)
Histogram classifier with global and local
likeliness coefficients (Plamondon & Lorette,
1989)
Clustering analysis ( Lorette, 1984)
Dynamic programming based methods; matching
with Mahalanobis pseudo-distance (Sato &
Kogure,1982)
Hidden Markov Model based methods (McCabe,
2000; Kosmala & Rigoll, 1998)
Table 1 gives a summary of some prominent works in
the online signature verification field.
Performance Evaluation
Performance evaluation of the output (which is to ac-
cept or reject the signature) is done using false rejec-
tion rate (FRR or type I error) and false acceptance rate
(FAR or type II error) (Huang & Yan, 1997). The values
888
Online Signature Recognition
for error rates of different approaches have been in-
cluded in the comparison table above. Equal error rate
(ERR) which is calculated using FRR and FAR is also
used for measuring the accuracy of the systems.
GENERAL PROBLEMS
The feature set in online has to be taken very carefully,
since it must have sufficient interpersonal variability so
that the input signature can be classified as genuine or
forgery. In addition, it must also have a low intra per-
sonal variability so that an authentic signature is ac-
cepted. Therefore, one has to be extremely cautious,
while choosing the feature set, as increase in dimen-
sionality does not necessarily mean an increase in effi-
ciency of a system.
FUTURE TRENDS
Biometrics is gradually replacing the conventional pass-
word and ID based devices, since it is both convenient
and safer than the earlier methods. Today, it is not
difficult at all to come across a fingerprint scanner or an
online signature pad. Nevertheless, it still requires a lot
of research to be done to make the system infallible,
because even an accuracy rate of 99% can cause failure
of the system when scaled to the size of a million.
So, it will not be strange if we have ATMs in near
future granting access only after face recognition, fin-
gerprint scan and verification of signature via embedded
devices, since a multimodal system will have a lesser
chance of failure than a system using a single biometric
or a password/ID based device. Currently online and off-
line signature verification systems are two disjoint
approaches. Efforts must be also made to integrate the
two approaches enabling us to exploit the higher accu-
racy of online verification method and greater applica-
bility of off-line method.
CONCLUSION
Most of the online methods claim near about 99%
accuracy for signature verification. These systems are
gaining popularity day by day and a number of products
are currently available in the market. However, online
verification is facing stiff competition from fingerprint
verification system which is both more portable and has
higher accuracy. In addition, the problem of maintaining
a balance between FAR and FRR has to be maintained.
Theoretically, both FAR and FRR are inversely related
to each other. That is, if we keep tighter thresholds to
TEAM LinG
Online Signature Recognition

Table 1. Summery of some prominent online papers Gupta, J., & McCabe, A. (1997). A review of dynamic
handwritten signature verification. Technical Article, O
Author Mode Of Verification Database Feature Extraction
Results
(Error rates) James Cook University, Australia.
coordinates to
86.5 % accuracy rate for
Wu, Lee and
Jou (1997)
Split and merging
200 genuine and
246 forged
represent the
signature and the
genuine and 97.2 % for
forged
Huang, K., & Yan, H. (1997). Off-line signature verification
27 people
velocity
based on geometric feature extraction and neural network
Features based on
Wu, Lee and
Similarity
measurement on
each 10 signs,
560 genuine and
coefficients of the FRR=1.4 % and classification. Pattern Recognition, 30(1), 9-17.
Jou (1998) logarithmic FAR=2.8 %
logarithmic spectrum 650 forged for
spectrum
testing
Ismail, M.A., & Gad, S. (2000). Off-line Arabic signature
0.1 % mismatch in
Zhang,
Pratikakis,
Corner point matching
corner points
segments in case of recognition and verification. Pattern Recognition, 33,
algorithm and Point to 188 signatures, corner point algorithm
Cornelis and
Nyssen
point matching from 19 people
extracted based on
velocity information
and 0.4 % mismatch in 1727-1740.
algorithm case of point to point
(2000)
matching algorithm
Jain, A. K., & Griess, F. D. (2000). Online signature
Number of strokes,
co-ordinate distance verification. Project Report, Department of Computer Sci-
between two points,
angle with respect to ence and Engineering, Michigan State University, USA.
Jain, Griess 1232 signatures x and y axis,
String matching and Type I: 2.8%
and Connell of 102 curvature, distance
common threshold Type II: 1.6%
(2002) individual from centre of
gravity, grey value
Jain, A. K., Griess, F. D., & Connell, S. D. (2002). Online
in 9X9
neighborhood and
signature verification. Pattern Recognition, 35(12), 2963-
velocity features 2972.
25 users EER (Euclidean)
Extreme Points
Feng and
Warping ( both
contributed
30 genuine
x and y trajectories
used, apart from
=
25.4 % Kosmala, A., & Rigoll, G. (1998). A systematic comparison
Euclidean distance and
Wah (2003)
Correlation
signatures
and 10 forged
torque and center of
mass.
EER
(correlation) = between online and off-line methods for signature verifi-
Coefficients used)
signatures 27.7 %
cation using hidden markov models. 14th International
Conference on Pattern Recognition (pp. 1755-1757).
decrease FAR, inevitably we increase the FRR by reject-
ing some genuine signatures. Further research is thus Lee, S. Y., Wu, Q. Z., & Jou, I. C. (1997). Online signature
still required to overcome this barrier. verification based on split and merge matching mecha-
nism. Pattern Recognition Letters, 18, 665-673
Lee, S. Y., Wu, Q. Z., & Jou, I. C. (1998). Online
REFERENCES signature verification based on logarithmic spectrum.
Pattern Recognition, 31(12), 1865-1871
Ammar, M., Fukumura, T., & Yoshida Y. (1988). Off-
line preprocessing and verification of signatures. Inter- Lorette, G. (1984). Online handwritten signature recogni-
national Journal of Pattern Recognition and Artifi- tion based on data analysis and clustering. Proceedings
cial Intelligence, 2(4), 589-602. of 7th International Conference on Pattern Recognition,
Vol. 2 (pp. 1284-1287).
Ammar, M., Fukumura, T., & Yoshida Y. (1990). Structural
description and classification of signature images. Pat- McCabe, A. (2000). Hidden markov modeling with simple
tern Recognition, 23(7), 697-710. directional features for effective and efficient handwrit-
ing verification. Proceedings of the Sixth Pacific Rim
Bajaj, R., & Chaudhury, S. (1997). Signature verification International Conference on Artificial Intelligence.
using multiple neural classifiers. Pattern Recognition,
30(1), l-7. Plamondon R., & Lorette, G. (1989). Automatic signa-
ture verification and writer identification the state of
Baltzakis, H., & Papamarkos, N. (2001). A new signature the art. Pattern Recognition, 22(2), 107-131.
verification technique based on a two-stage neural net-
work classifier. Engineering Applications of Artificial Ramesh, V.E., & Murty, M. N. (1999). Off-line signa-
Intelligence, 14, 95-103. ture verification using genetically optimized weighted
features. Pattern Recognition, 32(7), 217-233.
Drouhard, J. P., Sabourin, R., & Godbout, M. (1996). A
neural network approach to off-line signature verifica- Sato, Y., & Kogure, K. (1982). Online signature verifi-
tion using directional pdf. Pattern Recognition, 29(3), cation based on shape, motion and handwriting pressure.
415-424. Proceedings of 6th International Conference on Pat-
tern Recognition, Vol. 2 (pp. 823-826).
Feng, H., & Wah, C. C. (2003). Online signature verifi-
cation using a new extreme points warping technique. Tappert, C. C., Suen, C. Y., & Wakahara, T. (1990). The
Pattern Recognition Letters 24(16), 2943-2951. state of the art in on-line handwriting recognition. IEEE

889

TEAM LinG

Transactions on Pattern Analysis and Machine Intelli-
gence, 12(8).
Wessels, T., & Omlin, C. W. (2000). A hybrid approach for
signature verification. International Joint Conference
on Neural Networks.
Xiao, X., & Leedham, G. (2002). Signature verification
using a modified bayesian network. Pattern Recognition,
35, 983-995.
Zhang, K., Pratikakis, I., Cornelis, J., & Nyssen, E. (2000).
Using landmarks in establishing a point to point corre-
spondence between signatures. Pattern Analysis and
Applications, 3, 69-75.
KEY TERMS
Equal Error Rate: The error rate when the propor-
tions of FAR and FRR are equal. The accuracy of the
biometric system is inversely proportional to the value
of EER.
False Acceptance Rate: Rate of acceptance of a
forged signature as a genuine signature by a handwritten
signature verification system.
False Rejection Rate: Rate of rejection of a genu-
ine signature as a forged signature by a handwritten
signature verification system.
890
Online Signature Recognition
Global Features: The features are extracted using the
complete signature image or signal as a single entity.
Local Features: The geometric information of the
signature is extracted in terms of features after dividing
the signature image or signal into grids and sections.
Online Signature Recognition: The signature is
captured through a digitizer or an instrumented pen and
both geometric and temporal information are recorded
and later used in the recognition process.
Random Forgery: Random forgery is one in which
the forged signature has a totally different semantic
meaning and overall shape in comparison to the genuine
signature.
Simple Forgery: Simple forgery is one in which the
semantics of the signature are the same as that of the
genuine signature, but the overall shape differs to a great
extent, since the forger has no idea about how the
signature is done.
Skilled Forgery: In skilled forgery, the forger has a
prior knowledge about how the signature is written and
practices it well, before the final attempt of duplicating it.
Traced Forgery: For traced forgery, a signature
instance or its photocopy is used as a reference and tried
to be forged.
TEAM LinG

Organizational Data Mining
Hamid R. Nemati
The University of North Carolina at Greensboro, USA
Christopher D. Barko
The University of North Carolina at Greensboro, USA
INTRODUCTION
Data mining is now largely recognized as a business
imperative and considered essential for enabling the
execution of successful organizational strategies. The
adoption rate of data mining by enterprises is growing
quickly, due to noteworthy industry results in applica-
tions such as credit assessment, risk management, mar-
ket segmentation, and the ever-increasing volumes of
corporate data available for analysis. The quantity of
data being captured is staggeringdata experts esti-
mate that in 2002, the world generated five exabytes of
information. This amount of data is more than all the
words ever spoken by human beings (Hardy, 2004). The
rate of growth is just as astoundingthe amount of data
produced in 2002 was up 68% from just two years earlier.
The size of the typical business database has grown a
hundred-fold during the past five years as a result of
Internet commerce, ever-expanding computer systems,
and mandated recordkeeping by government regulations
(Hardy, 2004). Following this trend, a recent survey of
corporations across 23 countries revealed that the larg-
est transactional database almost doubled in size to 18 TB
(terabytes), while the largest decision-support database
grew to almost 30 TB (Reddy, 2004).
However, in spite of this enormous growth in enter-
prise databases, research from IBM reveals that organi-
zations use less than 1% of their data for analysis
(Brown, 2002). In a similar study, a leading business
intelligence firm surveyed executives at 450 companies
and discovered that 90% of these organizations rely on
gut instinct rather than hard facts for most of their
decisions, because they lack the necessary information
when they need it (Brown, 2002). In cases where suffi-
cient business information is available, those organiza-
tions only are able to utilize less than 7% of it (The
Economist, 2001). This is the fundamental irony of the
Information Age we live inorganizations possess enor-
mous amounts of business information yet have so little
real business knowledge.
In the past, companies have struggled to make deci-
sions because of lack of data. But in the current environ-
ment, more and more organizations are struggling to
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
891
O
overcome information paralysisthere is so much data
available that it is difficult to determine what is relevant
and how to extract meaningful knowledge. Organiza-
tions today routinely collect and manage terabytes of data
in their databases, thereby making information paralysis
a key challenge in enterprise decision making. The gen-
eration and management of business data lose much of
their potential organizational value unless important con-
clusions can be extracted from them quickly enough to
influence decision making, while the business opportu-
nity is still present. Managers must understand rapidly
and thoroughly the factors driving their business in order
to sustain a competitive advantage. Organizational speed
and agility, supported by fact-based decision making, are
critical to ensure an organization remains at least one
step ahead of its competitors.
BACKGROUND
The manner in which organizations execute this intri-
cate decision-making process is critical to their well-
being and industry competitiveness. Those organiza-
tions making swift, fact-based decisions by optimally
leveraging their data resources will outperform those
organizations that do not. A robust technology that
facilitates this process of optimal decision making is
known as Organizational Data Mining (ODM). ODM is
defined as leveraging data mining tools and technolo-
gies in order to enhance the decision-making process by
transforming data into valuable and actionable knowl-
edge to gain a competitive advantage (Nemati & Barko,
2001). ODM eliminates the guesswork that permeates
so much of corporate decision making. By adopting
ODM, an organizations managers and employees are
able to act sooner rather than later, be proactive rather
than reactive, and know rather than guess. ODM technol-
ogy has helped many organizations to optimize internal
resource allocations while better understanding and
responding to the needs of their customers.
ODM spans a wide array of technologies, including,
but not limited to, e-business intelligence, On-Line
Analytical Processing (OLAP), optimization, Customer
TEAM LinG

Relationship Management (CRM), electronic CRM (e-
CRM), Executive Information Systems (EIS), digital
dashboards, and enterprise information portals. ODM
enables organizations to answer questions about the past
(what has happened?), the present (what is happening?),
and the future (what might happen?). Armed with this
capability, organizations can generate valuable knowl-
edge from their data, which, in turn, enhances enterprise
decisions. This decision-enhancing technology offers
many advantages in operations (faster product develop-
ment, optimal supply chain management), marketing
(higher profitability and increased customer loyalty
through more effective marketing campaigns), finance
(optimal portfolio management, financial analytics),
and strategy implementation (Business Performance
Management [BPM] and the Balanced Scorecard).
Over the last three decades, the organizational role
of information technology has evolved from efficiently
processing large amounts of batch transactions to pro-
viding information in support of tactical and strategic
decision-making activities. This evolution, from auto-
mating expensive manual systems to providing strategic
organizational value, led to the birth of Decision Sup-
port Systems (DSS), such as data warehousing and data
mining. The organizational need to combine data from
multiple stand-alone systems (e.g., financial, manufac-
turing, and distribution) grew as corporations began to
acknowledge the power of combining these data sources
for reporting. This spurred the growth of data warehous-
ing, where multiple data sources were stored in a format
that supported advanced data analysis.
The slowness in adoption of ODM techniques in the
1990s was partly due to an organizational and cultural
resistance. Business management always has been re-
luctant to trust something it does not fully understand.
Until recently, most businesses were managed by in-
stinct, intuition, and gut feeling. The transition over the
past 20 years to a method of managing by the numbers is
both the result of technology advances as well as a
generational shift in the business world, as younger
managers arrive with information technology training
and experience.
ODM Research
Given the scarcity of past research in ODM along with
its growing acceptance and importance in organizations,
we conducted empirical research during the past several
years that explored the utilization of ODM in organiza-
tions along with project implementation factors critical
for success. We surveyed ODM professionals from
multiple industries in both domestic and international
organizations. Our initial research examined the ODM
industry status and best practices, identified both tech-
892
Organizational Data Mining
nical and business issues related to ODM projects, and
elaborated on how organizations are benefiting through
enhanced enterprise decision making (Nemati & Barko,
2001). The results of our research suggest that ODM
can improve the quality and accuracy of decisions for
any organization that is willing to make the investment.
After exploring the status and utilization of ODM in
organizations, we decided to focus subsequent research
on how organizations implement ODM projects and on
the factors critical to its success. To that end, we
developed a new ODM Implementation Framework based
on data, technology, organizations, and the Iron Triangle
(Nemati & Barko, 2003). Our research demonstrated
that selected organizational data mining project factors,
when modeled under this new framework, have a signifi-
cant influence on the successful implementation of
ODM projects.
Given the promise of strengthening customer rela-
tionships and enhancing profits, CRM technology and
associated research are gaining greater acceptance within
organizations. However, findings from recent studies
suggest that organizations generally fail to support their
CRM efforts with complete data (Brohman et al., 2003).
As further investigation, our latest research has focused
on a specific ODM technology known as Electronic
Customer Relationship Management (e-CRM) and its
data integration role within organizations. Consequently,
we developed a new e-CRM Value Framework to better
examine the significance of integrating data from all
customer touch-points with the goal of improving cus-
tomer relationships and creating additional value for the
firm. Our research findings suggest that, despite the
cost and complexity, data integration for e-CRM projects
contributes to a better understanding of the customer
and leads to higher return on investment (ROI), a greater
number of benefits, improved user satisfaction, and a
higher probability of attaining a competitive advantage
(Nemati, Barko & Moosa, 2003).
MAIN THRUST
Data mining is the process of discovering and interpret-
ing previously unknown patterns in databases. It is a
powerful technology that converts data into information
and potentially actionable knowledge. However, there
are many obstacles to the broad inclusion of data mining
in organizations. Obtaining new knowledge in an organi-
zational vacuum does not facilitate optimal decision
making in a business setting. Simply incorporating data
mining into the enterprise mix without considering non-
technical issues is usually a recipe for failure. Busi-
nesses must give careful thought when weaving data
mining into their organizations fabric. The unique orga-
TEAM LinG
Organizational Data Mining
nizational challenge of understanding and leveraging
ODM to engineer actionable knowledge requires assimi-
lating insights from a variety of organizational and tech-
nical fields and developing a comprehensive framework
that supports an organizations quest for a sustainable
competitive advantage. These multi-disciplinary fields
include data mining, business strategy, organizational
learning and behavior, organizational culture, organiza-
tional politics, business ethics and privacy, knowledge
management, information sciences, and decision sup-
port systems. These fundamental elements of ODM can
be categorized into three main groups: Artificial Intelli-
gence (AI), Information Technology (IT), and Organiza-
tional Theory (OT). Our research and industry experi-
ence suggest that successfully leveraging ODM requires
integrating insights from all three categories in an orga-
nizational setting typically characterized by complexity
and uncertainty. This is the essence and uniqueness of
ODM. Obtaining maximum value from ODM involves a
cross-department team effort that includes statisticians/
data miners, software engineers, business analysts, line-
of-business managers, subject-matter experts, and upper
management support.
Organizational Theory and ODM
Organizations are concerned primarily with studying
how operating efficiencies and profitability can be
achieved through the effective management of custom-
ers, suppliers, partners, and employees. To achieve these
goals, research in Organizational Theory (OT) suggests
that organizations use data in three vital knowledge cre-
ation activities. This organizational knowledge creation
and management is a learned ability that only can be
achieved via an organized and deliberate methodology.
This methodology is a foundation for successfully lever-
aging ODM within the organization. The three knowl-
edge creation activities (Choo, 1997) are:
Sense Making: The ability to interpret and under-
stand information about the environment and events
happening both inside and outside the organization.
Knowledge Making: The ability to create new
knowledge by combining the expertise of members
to learn and innovate.
Decision Making: The ability to process and ana-
lyze information and knowledge in order to select
and implement the appropriate course of action.
First, organizations use data to make sense of changes
and developments in the external environmentsa pro-
cess called sense making. This is a vital activity wherein
managers discern the most significant changes, interpret
their meaning, and develop appropriate responses. Sec-
ond, organizations create, organize, and process data to
generate new knowledge through organizational learn- O
ing. This knowledge creation activity enables the orga-
nization to develop new capabilities, design new prod-
ucts and services, enhance existing offerings, and im-
prove organizational processes. Third, organizations
search for and evaluate data in order to make decisions.
This data is critical, since all organizational actions are
initiated by decisions and all decisions are commit-
ments to actions, the consequences of which will, in
turn, lead to the creation of new data. Adopting an OT
methodology enables an enterprise to enhance the
knowledge engineering and management process.
In another OT study, researchers and academic
scholars have observed that there is no direct correla-
tion between information technology (IT) investments
and organizational performance. Research has con-
firmed that identical IT investments in two different
companies may give a competitive advantage to one
company but not the other. Therefore, a key factor for
the competitive advantage in an organization is not the
IT investment but the effective utilization of informa-
tion as it relates to organizational performance
(Brynjolfsson & Hitt, 1996). This finding emphasizes
the necessity of integrating OT practices with robust
information technology and artificial intelligence tech-
niques in successfully leveraging ODM.
ODM Practices at Leading Companies
A 2002 Strategic Decision-Making study conducted by
Hackett Best Practices determined that world-class
companies have adopted ODM technologies at more
than twice the rate of average companies (Hoblitzell,
2002). ODM technologies provide these world-class
organizations greater opportunities to understand their
business and make informed decisions. ODM also en-
ables world-class organizations to leverage their inter-
nal resources more efficiently and more effectively
than their average counterparts, who have not fully
embraced ODM.
Many of todays leading organizations credit their
success to the development of an integrated, enter-
prise-level ODM system. As part of an effective CRM
strategy, customer retention is now widely viewed by
organizations as a significant marketing strategy in
creating a competitive advantage. Research suggests
that as little as a 5% increase in retention can mean as
much as a 95% boost in profit, and repeat customers
generate over twice as much gross income as new
customers (Winer, 2001). In addition, many business
executives today have replaced their cost reduction
strategies with a customer retention strategyit costs
approximately five to 10 times more to acquire new
893
TEAM LinG

customers than to retain established customers (Pan &
Lee, 2003).
An excellent example of a successful CRM strategy
is Harrahs Entertainment, which has saved over $20
million per year since implementing its Total Rewards
CRM program. This ODM system has given Harrahs a
better understanding of its customers and has enabled
the company to create targeted marketing campaigns
that almost doubled the profit per customer and deliv-
ered same-store sales growth of 14% after only the first
year. In another notable case, Travelocity.com, an
Internet-based travel agency, implemented an ODM sys-
tem and improved total bookings and earnings by 100%
in 2000. Gross profit margins improved 150%, and
booker conversion rates rose 8.9%, the highest in the
online travel services industry.
In another significant study, executives from 24
leading companies in customer-knowledge management,
including FedEx, Frito-Lay, Harley-Davidson, Procter
& Gamble, and 3M, all realized that in order to succeed,
they must go beyond simply collecting customer data
and must translate it into meaningful knowledge about
existing and potential customers (Davenport, Harris &
Kohli, 2001). This study revealed that several objec-
tives were common to all of the leading companies, and
these objectives can be facilitated by ODM. A few of
these objectives are segmenting the customer base,
prioritizing customers, understanding online customer
behavior, engendering customer loyalty, and increasing
cross-selling opportunities.
FUTURE TRENDS
The number of ODM projects is projected to grow more
than 300% in the next decade (Linden, 1999). As the
collection, organization, and storage of data rapidly
increase, ODM will be the only means of extracting
timely and relevant knowledge from large corporate
databases. The growing mountains of business data,
coupled with recent advances in Organizational Theory
and technological innovations, provide organizations
with a framework to effectively use their data to gain a
competitive advantage. An organizations future suc-
cess will depend largely on whether or not they adopt
and leverage this ODM framework. ODM will continue
to expand and mature as the corporate demand for one-
to-one marketing, CRM, e-CRM, Web personalization,
and related interactive media increases.
We believe that organizations are slowly moving
from the Information Age to the Knowledge Age, where
decision makers will leverage ODM for optimal busi-
ness performance. Organizations are complex entities
comprised of employees, managers, politics, culture,
hierarchies, teams, processes, customers, partners, sup-
894
Organizational Data Mining
pliers, and shareholders. The never-ending challenge is
to successfully integrate data-mining technologies
within organizations in order to enhance decision mak-
ing with the objective of optimally allocating scarce
enterprise resources. This is not an easy task, as many
information technology professionals, consultants, and
managers can attest. The media can oversimplify the
effort, but successfully implementing ODM is not ac-
complished without political battles, project manage-
ment struggles, cultural shocks, business process
reengineering, personnel changes, short-term financial
and budgetary shortages, and overall disarray.
Recent ODM research has revealed a number of
industry predictions that are expected to be key ODM
issues in the coming years. Nemati & Barko (2001)
found that almost 80% of survey respondents expect
Web farming/mining and consumer privacy to be sig-
nificant issues. We also foresee the development of
widely accepted standards for ODM processes and tech-
niques to be an influential factor for knowledge seekers
in the 21st century. One attempt at ODM standardization
is the creation of the Cross Industry Standard Process
for Data Mining (CRISP-DM) project that developed an
industry and tool-neutral, data-mining process model to
solve business problems. Another attempt at industry
standardization is the work of the Data Mining Group in
developing and advocating the Predictive Model Markup
Language (PMML), which is an XML-based language
that provides a quick and easy way for companies to
define predictive models and share models between
compliant vendors applications. Last, Microsofts OLE
DB for Data Mining is a further attempt at industry
standardization and integration. This specification of-
fers a common interface for data mining that will enable
developers to embed data-mining capabilities into their
existing applications. One only has to consider
Microsofts industry-wide dominance of the office pro-
ductivity (Microsoft Office), software development
(Visual Basic and .Net), and database (SQL Server)
markets to envision the potential impact this could have
on the ODM market and its future direction.
CONCLUSION
Although many improvements have materialized over
the last decade, the knowledge gap in many organiza-
tions is still prevalent. Industry professionals have sug-
gested that many corporations could maintain current
revenues at half the current costs, if they optimized
their use of corporate data (Saarenvirta, 2001). Whether
this finding is true or not, it sheds light on an important
issue. Leading corporations in the next decade will
adopt and weave these ODM technologies into the fab-
ric of their organizations at the strategic, tactical, and
TEAM LinG
Organizational Data Mining
operational levels. Those enterprises that see the strate-
gic value of evolving into knowledge organizations by
leveraging ODM will benefit directly in the form of
improved profitability, increased efficiency, and sus-
tainable competitive advantage. Once the first organiza-
tion within an industry realizes a competitive advantage
through ODM, it is only a matter of time before one of
three events transpires: its industry competitors adopt
ODM, change industries, or vanish. By adopting ODM,
an organizations managers and employees are able to
act sooner rather than later, anticipate rather than react,
know rather than guess, and, ultimately, succeed rather
than fail.
REFERENCES
Brohman, M. et al. (2003). Data completeness: A key to
effective net-based customer service systems. Commu-
nications of the ACM, 46(6), 47-51.
Brown, E. (2002). Analyze this. Forbes, 169(8), 96-98.
Brynjolfsson, E., & Hitt, L. (1996, September 9). The
customer counts. InformationWeek. Retrieved from
www.informationweek.com/596/96mit.htm
Choo, C.W. (1997). The knowing organization: How
organizations use information to construct meaning,
create knowledge, and make decisions. Retrieved from
www.choo.fis.utoronto.ca/fis/ko/default.html
Davenport, T.H., Harris, J.G., & Kohli, A.K. (2001).
How do they know their customers so well? Sloan
Management Review, 42(2), 63-73.
Hardy, Q. (2004). Data of reckoning. Forbes, 173(10),
151-154.
Hoblitzell, T. (2002). Disconnects in todays BI sys-
tems. DM Review, 12(6), 56-59.
Linden, A. (1999). CIO update: Data mining applica-
tions of the next decade. Inside Gartner Group. Gartner
Inc.
Nemati, H.R., & Barko, C.D. (2001). Issues in organiza-
tional data mining: A survey of current practices. Jour-
nal of Data Warehousing, 6(1), 25-36.
Nemati, H.R., & Barko, C.D. (2003). Key factors for
achieving organizational data mining success. Indus-
trial Management & Data Systems, 103(4), 282-292.
Nemati, H.R., Barko, C.D., & Moosa, A. (2003). E-
CRM analytics: The role of data integration. Journal of
Electronic Commerce in Organizations, 1(3), 73-89.
Pan, S.L., & Lee, J.-N. (2003). Using E-CRM for a
unified view of the customer. Communications of the O
ACM, 46(4), 95-99.
Reddy, R. (2004). The reality of real time. Intelligent
Enterprise, 7(10), 40-41.
Saarenvirta, G. (2001). Operation data mining. DB2 Maga-
zine. Retrieved from http://www.db3mag.com/db_area/
archives/2001/q2/saarenvirta.html
The slow progress of fast wires. (2001). The Econo-
mist, 358(8209), 57-59.
Winer, R.S. (2001). A framework for customer rela-
tionship management. California Management Review,
43(4), 89-106.
TERMS
CRISP-DM (Cross Industry Standard Process
for Data Mining): An industry and tool-neutral data-
mining process model developed by members from
diverse industry sectors and research institutes.
CRM (Customer Relationship Management): The
methodologies, software, and Internet capabilities that
help a company manage customer relationships in an
efficient and organized manner.
Information Paralysis: A condition where too much
data causes difficulty in determining relevancy and ex-
tracting meaningful information and knowledge.
ODM (Organizational Data Mining): The process
of leveraging data-mining tools and technologies in an
organizational setting in order to enhance the decision-
making process by transforming data into valuable and
actionable knowledge in order to gain a competitive
advantage.
OLAP (Online Analytical Processing): A data
mining technology that uses software tools to interac-
tively and simultaneously analyze different dimensions
of multi-dimensional data.
OT (Organizational Theory): A body of research
that focuses on studying how organizations create and
use information in three strategic arenassense mak-
ing, knowledge making, and decision making.
PMML (Predictive Model Markup Language):
An XML-based language that provides a method for
companies to define predictive models and share mod-
els between compliant vendors applications.
895
TEAM LinG
896
Path Mining in Web Processes Using Profiles
Jorge Cardoso
University of Madeira, Portugal
INTRODUCTION
Business process management systems (BPMSs) (Smith
& Fingar, 2003) provide a fundamental infrastructure to
define and manage business processes, Web processes,
and workflows. When Web processes and workflows
are installed and executed, the management system gen-
erates data describing the activities being carried out
and is stored in a log. This log of data can be used to
discover and extract knowledge about the execution of
processes. One piece of important and useful informa-
tion that can be discovered is related to the prediction of
the path that will be followed during the execution of a
process. I call this type of discovery path mining. Path
mining is vital to algorithms that estimate the quality of
service of a process, because they require the predic-
tion of paths. In this work, I present and describe how
process path mining can be achieved by using data-
mining techniques.
BACKGROUND
BPMSs, such as workflow management systems (WfMS)
(Cardoso, Bostrom, & Sheth, 2004) are systems ca-
pable of both generating and collecting considerable
amounts of data describing the execution of business
processes, such as Web processes. This data is stored in
a process log systems, which are vast data archives that
are seldom visited. Yet, the data generated from the
execution of processes are rich with concealed infor-
mation that can be used for making intelligent business
decisions.
One important and useful piece of knowledge to
discover and extract from process logs is the implicit
rules that govern path mining.
In Web processes for e-commerce, suppliers and
customers define a contract between the two parties,
specifying quality of service (QoS) items, such as prod-
ucts or services to be delivered, deadlines, quality of
products, and cost of services. The management of QoS
metrics directly impacts the success of organizations
participating in e-commerce. A Web process, which
typically can have a graphlike representation, includes a
number of linearly independent control paths. Depend-
ing on the path followed during the execution of a Web
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
process, the QoS may be substantially different. If you
can predict with a certain degree of confidence the path
that will be followed at run time, you can significantly
increase the precision of QoS estimation algorithms for
Web processes.
Because the large amounts of data stored in process
logs exceeds understanding, I describe the use of data-
mining techniques to carry out path mining from the data
stored in log systems. This approach uses classification
algorithms to conveniently extract patterns represent-
ing knowledge related to paths. My work is novel be-
cause no previous work has targeted the path mining of
Web processes and workflows. The literature includes
only work on process and workflow mining (Agrawal,
Gunopulos, & Leymann, 1998; Herbst & Karagiannis,
1998; Weijters & van der Aalst, 2001).
Process mining allows the discovery of workflow
models from a workflow log containing information
about workflow processes executed. Luo, Sheth, Kochut,
and Arpinar (2003) present an architecture and the
implementation of a sophisticated exception-handling
mechanism supported by a case-based reasoning (CBR)
engine.
MAIN THRUST
The material presented in this section emphasizes the
use of data-mining techniques for uncovering interest-
ing process patterns hidden in large process logs. The
method contained in the next section is more suitable
for administrative and production processes compared
to Ad-hoc and collaborative processes, because they are
more repetitive and predictable.
Web Process Scenario
A major bank has realized that to be competitive and
efficient it must adopt a new and modern information
system infrastructure. Therefore, a first step was taken
in that direction with the adoption of a workflow man-
agement system to support its business processes. All
the services available to customers are stored and ex-
ecuted under the supervision of the workflow system.
One of the services supplied by the bank is the loan
process depicted in Figure 1.
TEAM LinG
Path Mining in Web Processes Using Profiles
A Web process is composed of Web services and
transitions. Web services are represented by circles,
and transitions are represented by arrows. Transitions
express dependencies between Web services. A Web
service with more than one outgoing transition can be
classified as an and-split or xor-split. And-split Web
services enable all their outgoing transitions after com-
pleting their execution. Xor-split Web services enable
only one outgoing transition after completing their
execution. And-split Web services are represented with
, and xor-split Web services are represented with .
A Web service with more than one incoming transition
can be classified as an and-join or xor-join. And-join
Web services start their execution when all their incom-
ing transitions are enabled. Xor-join Web services are
executed as soon as one of the incoming transitions is
enabled. As with and-split and xor-split Web services,
and-join and xor-join Web services are represented with
the symbols and , respectively. database record.
The Web process of this scenario is composed of 14
Web services. The Fill Loan Request Web service al-
lows clients to request a loan from the bank. In this step,
the client is asked to fill out an electronic form with
personal information and data describing the condition
of the loan being requested.
The second Web service, Check Loan Type, deter-
mines the type of loan a client has requested and, based
on the type, forwards the request to one of three Web
services: Check Home Loan, Check Educational Loan,
or Check Car Loan.
Educational loans are not handled and managed auto-
matically. After an educational loan application is sub-
mitted and checked, a notification is immediately sent
informing the client that he or she has to contact the
bank personally.
A loan request can be either accepted (Approve
Home Loan and Approve Car Loan) or rejected (Reject
Home Loan and Reject Car Loan). In the case of a home
loan, however, the loan can also be approved condition-
Figure 1. The loan process
ally. The Web service Approve Home Loan Conditionally,
as the name suggests, approves a home loan under a set P
of conditions.
The following formula is used to determine if a loan
is approved or rejected.
MP = (L*R*(1+R/12)12*NY)/(-12+12*(1+R12)12*NY)
(1)
MP=Monthly payment,L=Loan amount,R=Interest
rate,NY=Number of years
When the result of a loan application is known, it is
e-mailed to the client. Three Web services are respon-
sible for notifying the client: Notify Home Loan Client,
Notify Education Loan Client, and Notify Car Loan
Client. Finally, the Archive Application Web service
creates a report and stores the loan application data in a
Web Process Log
During the execution of Web processes (such as the one
presented in Figure 1), events and messages generated
by the enactment system are stored in a Web process
log. These data stores provide an adequate format on
which path mining can be performed. The data includes
real-time information describing the execution and be-
havior of Web processes, Web services, instances, tran-
sitions, and other elements such as runtime QoS metrics.
Table 1 illustrates an example of a modern Web process
log.
To perform path mining, current Web process logs
need to be extended to store information indicating the
values and the type of the input parameters passed to
Web services and the output parameters received from
Web services. Table 2 shows an extended Web process
log that accommodates input/output values of Web ser-
vices parameters generated at run time. Each Parameter/
value entry has a type, parameter name, and value (e.g.,
string loan-type=car-loan).
Additionally, the Web process log needs to include
path information describing the Web services that have
been executed during the enactment of a Web process.
This information can be easily stored in the log. For
example, an extra field can be added to the log system to
contain the information indicating the path followed.
The path needs only to be associated to the entry corre-
sponding to the last service of a process to be executed.
For example, in the Web process log illustrated in Table
2, the service NotifyUser is the last service of a Web
process. The log has been extended in such a way that the
NotifyUser record contains information about the path
that was followed during the Web process execution.
897
TEAM LinG
 Path Mining in Web Processes Using Profiles

Table 1. Web process log

Date Web process Process Web service Service Cost Durati
instance instance on
6:45 03-03-04 LoanApplication LA04 RejectCarLoan RCL03 $1.2 13 min
6:51 03-03-04 TravelRequest TR08 FillRequestTravel FRT03 $1.1 14 min
6:59 03-03-04 TravelRequest TR09 NotifyUser NU07 $1.4 24 hrs
7:01 03-03-04 InsuranceClaim IC02 SubmitClaim SC06 $1.2 05 min

Table 2. Extended Web process log

Process Web service Service Parameter/value Path

instance instance
LA04 RejectCarLoan RCL03 int LoanNum=14357;
string loan-type=car-loan
LA04 NotifyCLoanClient NLC07 string e-mail=jf@uma.pt
LA05 CheckLoanReques CLR05 double income=12000;
t string Name=Eibe Frank;
TR09 NotifyUser NU07 String e-mail=jf@uma.pt; FillForm->CheckForm->
String tel=35129170023 Approve->Sign->Report

Web Process Profile
When beginning work on path mining, it is necessary to
elaborate a profile for each Web process. A profile
provides the input to machine learning and is character-
ized by its values on a fixed, predefined set of attributes.
The attributes correspond to the Web service input/
output parameters that have been stored previously in the
Web process log. Path mining will be performed on
these attributes.
A profile contains two types of attributes, numeric
and nominal. Numeric attributes measure numbers, ei-
ther real or integer-valued. For example, Web services
inputs or outputs parameters that are of type byte, deci-
mal, int, short, or double will be placed in the profile and
classified as numeric. In Table 2, the parameters
LoanNum, income, BudgetCode, income, and tel will be
classified as numeric in the profile.
Nominal attributes take on values within a finite set
of possibilities. Nominal quantities have values that are
distinct symbols. For example, the parameter loan-type
from the loan application and present in Table 2 is
nominal because it can take the finite set of values:
home-loan, education-loan, and car-loan. In my approach,
string and Boolean data type manipulated by Web ser-
vices are considered to be nominal attributes.
Profile Classification
The attributes present in a profile trigger the execution
of a specific set of Web services. Therefore, for each
profile previously constructed, I associate an additional
attribute, the path attribute, indicating the path followed
when the attributes of the profile have been assigned to
specific values. The path attribute is a target class.
Classification algorithms classify samples or instances
into target classes.
After the profiles and a path attribute value for each
profile have been determined, I can use data-mining
methods to establish a relationship between the pro-
files and the paths followed at run time. One method
appropriate to deal with my problem is the use of
classification.
In classification, a learning schema takes a set of
classified profiles, from which it is expected to learn a
way of classifying unseen profiles. Because the path of
each training profile is provided, my methodology uses
supervised learning.
EXPERIMENTS
In this section, I present the results of applying my
algorithm to a synthetic loan dataset. To generate a
synthetic dataset, I start with the process presented in
the introductory scenario and, using this as a process
model graph, log a set of process instance executions.
The data are lists of event records stored in a Web
process log consisting of process names, instance iden-
tification, Web services names, variable names, and so
forth. Table 3 shows the additional data that have been
stored in the Web process log. The information in-
cludes the Web service variable values that are logged
by the system and the path that has been followed during
the execution of instances. Each entry corresponds to
an instance execution.

898

TEAM LinG
Path Mining in Web Processes Using Profiles

Table 3. Additional data stored in the Web process log

Income Loan_Type Loan_ Loan_ Name SSN Path

P
amount years
1361.0 Home-Loan 129982.0 33 Bernard-Boar 10015415 FR>CLT>CHL>AHL>NHC>CA
Unknown Education-Loan Unknown Unknown John-Miller 15572979 FR>CLT>CEL>CA
1475.0 Car-Loan 15002.0 9 Eibe-Frank 10169316 FR>CLT>CCL>ACL>NCC>CA

Web process profiles provide the input to machine Each experiment has involved data from 1,000 Web
learning and are characterized by a set of six attributes: process executions and a variable number of attributes
income, loan_type, loan_amount, loan_years, name, and (ranging from two to six). I have conducted 34 experi-
SSN. The profiles for the loan process contain two types ments, analyzing a total of 34,000 records containing data
of attributes: numeric and nominal. The attributes income, from Web process instance executions. Figure 2 shows
loan_amount, loan_years, and SSN are numeric, whereas the results that I have obtained.
the attributes loan_type and name are nominal. As an The path-mining technique developed has achieved
example of a nominal attribute, loan_type can take the encouraging results. When three or more attributes are
finite set of values home-loan, education-loan, and car- involved in the prediction, the system is able to predict
loan. These attributes correspond to the Web service correctly the path followed for more than 75% of the
input/output parameters that have been stored previ- process instances. This accuracy improves when four
ously in the Web process log presented in Table 3. attributes are involved in the prediction; in this case,
Each profile is associated with a class indicating the more than 82% of the paths are correctly predicted.
path that has been followed during the execution of a When five attributes are involved, I obtain a level of
process when the attributes of the profile have been prediction that reaches a high of 93.4%. Involving all six
assigned specific values. The last column of Table 3 attributes in the prediction gives excellent results: 88.9%
shows the class named path. The profiles and path at- of the paths are correctly predicted. When a small
tributes will be used to establish a relationship between number of attributes are involved in the prediction, the
the profiles and the paths followed at runtime. The results are not as good. For example, when only two
profiles and the class path have been extracted from the attributes are selected, I obtain predictions that range
Web process log. from 25.9% to 86.7%.
After profiles are constructed and associated with
paths, these data are combined and formatted to be
analyzed using Weka (2004), a set of software for FUTURE TRENDS
machine learning and data mining. The data is automati-
cally formatted using the ARFF format. I have used the Currently, organizations use BPMSs, such as WfMS, to
J.48 algorithm, which is Wekas implementation of the define, enact, and manage a wide range of distinct appli-
C4.5 (Hand, Mannila, & Smyth, 2001) decision tree cations (Q-Link Technologies, 2002), such as insur-
learner to classify profiles. C4.5 decision tree learner ance claims, bank loans, bioinformatic experiments
is one of the most well-known decision tree algorithms in (Hall, Miller, Arnold, Kochut, Sheth, & Weise, 2003),
the data-mining community. Weka system and its data health-care procedures (Anyanwu, Sheth, Cardoso,
format (ARFF) is also one of the most well-known data- Miller, & Kochut, 2003), and telecommunication services
mining systems in academia. (Luo, Sheth, Kochut, & Arpinar, 2003).
In the future, I expect to see a wider spectrum of
Figure 2. Experimental results applications managing processes in organizations. Ac-
cording to the Aberdeen Groups estimates, spending in
100.00% the business process management software sector (which
% Correctly Predicted Paths

80.00%
includes workflow systems) reached $2.26 billion in
2001 (Cowley, 2002).
60.00%
The concept of path mining can be used effectively in
40.00% many business applications for example, to estimate
20.00%
the QoS of Web processes and workflows (Cardoso,
Miller, Sheth, Arnold, & Kochut, 2004) because the
0.00%
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 5 5 5 5 5 5 6
estimation requires the prediction of paths. Organizations
Number of Attributes operating in modern markets, such as e-commerce activi-

899

TEAM LinG

ties and distributed Web services interactions, require
QoS management. Appropriate quality control leads to
the creation of quality products and services; these, in
turn, fulfill customer expectations and achieve customer
satisfaction (Cardoso, Sheth, & Miller, 2002).
CONCLUSION
BPMSs, Web processes, workflows, and workflow sys-
tems represent fundamental technological infrastruc-
tures that efficiently define, manage, and support busi-
ness processes. The data generated from the execution
and management of Web processes can be used to
discover and extract knowledge about the process ex-
ecutions and structure.
I have shown that one important area of Web pro-
cesses to analyze is path mining. I have demonstrated
how path mining can be achieved by using data-mining
techniques, namely classification, to extract path knowl-
edge from Web process logs. From my experiments, I
can conclude that classification methods are a good
solution to perform path mining on administrative and
production Web processes.
REFERENCES
Agrawal, R., Gunopulos, D., & Leymann, F. (1998).
Mining process models from workflow logs. Proceed-
ings of the Sixth International Conference on Extend-
ing Database Technology, Spain.
Anyanwu, K., Sheth, A., Cardoso, J., Miller, J. A., &
Kochut, K. J. (2003). Healthcare enterprise process
development and integration. Journal of Research and
Practice in Information Technology, 35(2), 83-98.
Cardoso, J., Bostrom, R. P., & Sheth, A. (2004). Workflow
management systems and ERP systems: Differences, com-
monalities, and applications. Information Technology
and Management Journal, 5(3-4), 319-338.
Cardoso, J., Miller, J., Sheth, A., Arnold, J., & Kochut, K.
(2004). Quality of service for workflows and Web service
processes. Journal of Web Semantics: Science, Services
and Agents on the World Wide Web, 1(3), 281-308.
Cardoso, J., Sheth, A., & Miller, J. (2002). Workflow
quality of service. Proceedings of the International Con-
ference on Enterprise Integration and Modeling Tech-
nology and International Enterprise Modeling Confer-
ence, Spain.
900
Path Mining in Web Processes Using Profiles
Cowley, S. (2002, September 23). Study: BPM market
primed for growth. Available from the InfoWorld Web
site, http://www.infoworld.com
Hall, R. D., Miller, J. A., Arnold, J., Kochut, K. J., Sheth,
A. P., & Weise, M. J. (2003). Using workflow to build an
information management system for a geographically
distributed genome sequence initiative. In R. A. Prade &
H. J. Bohnert (Eds.), Genomics of plants and fungi (pp.
359-371). New York: Marcel Dekker.
Hand, D. J., Mannila, H., & Smyth, P. (2001). Principles of
data mining. Bradford Book.
Herbst, J., & Karagiannis, D. (1998). Integrating machine
learning and workflow management to support acquisi-
tion and adaption of workflow models. Proceedings of the
Ninth International Workshop on Database and Expert
Systems Applications.
Luo, Z., Sheth, A., Kochut, K., & Arpinar, B. (2003).
Exception handling for conflict resolution in cross-
organizational workflows. Distributed and Parallel
Databases, 12(3), 271-306.
Q-Link Technologies. (2002). BPM2002: Market mile-
stone report. Retrieved from http://www.qlinktech.com.
Smith, H., & Fingar, P. (2003). Business process man-
agement (BPM): The third wave. Meghan-Kiffer Press.
Weijters, T., & van der Aalst, W. M. P. (2001). Process
mining: Discovering workflow models from event-based
data. Proceedings of the 13th Belgium-Netherlands
Conference on Artificial Intelligence.
Weka. (2004). Weka [Computer software.] Retrieved from
http://www.cs.waikato.ac.nz/ml/weka/
KEY TERMS
Business Process: A set of one or more linked
activities that collectively realize a business objective
or goal, normally within the context of an organizational
structure.
Business Process Management System (BPMS):
Provides an organization with the ability to collectively
define and model its business processes, deploy these
processes as applications that are integrated with its
existing software systems, and then provide managers
with the visibility to monitor, analyze, control, and
improve the execution of those processes.
TEAM LinG
Path Mining in Web Processes Using Profiles
Process Definition: The representation of a business
process in a form that supports automated manipulation
or enactment by a workflow management system.
Web Process: A set of Web services that carry out a
specific goal.
Web Process Data Log: Records and stores events
and messages generated by the enactment system during
the execution of Web processes.
Web Service: Describes a standardized way of inte-
grating Web-based applications by using open standards
over an Internet protocol.
Workflow: The automation of a business process, in
whole or part, during which documents, information, or P
tasks are passed from one participant to another for
action, according to a set of procedural rules.
Workflow Management System: A system that de-
fines, creates, and manages the execution of workflows
through the use of software, which is able to interpret
the process definition, interact with participants, and,
where required, invoke the use of tools and applications.
901
TEAM LinG
902
Pattern Synthesis for Large-Scale Pattern
Recognition
P. Viswanath
Indian Institute of Science, India
M. Narasimha Murty
Indian Institute of Science, India
Shalabh Bhatnagar
Indian Institute of Science, India
INTRODUCTION
Two major problems in applying any pattern recognition
technique for large and high-dimensional data are (a)
high computational requirements and (b) curse of di-
mensionality (Duda, Hart, & Stork, 2000). Algorithmic
improvements and approximate methods can solve the
first problem, whereas feature selection (Guyon &
Elisseeff, 2003), feature extraction (Terabe, Washio,
Motoda, Katai, & Sawaragi, 2002), and bootstrapping
techniques (Efron, 1979; Hamamoto, Uchimura, &
Tomita, 1997) can tackle the second problem. We pro-
pose a novel and unified solution for these problems by
deriving a compact and generalized abstraction of the
data. By this term, we mean a compact representation of
the given patterns from which one can retrieve not only
the original patterns but also some artificial patterns.
The compactness of the abstraction reduces the compu-
tational requirements, and its generalization reduces
the curse of dimensionality effect. Pattern synthesis
techniques accompanied with compact representations
attempt to derive compact and generalized abstractions
of the data. These techniques are applied with nearest
neighbor classifier (NNC), which is a popular nonpara-
metric classifier used in many fields, including data
mining, since its conception in the early 1950s
(Dasarathy, 2002).
BACKGROUND
Pattern synthesis techniques, compact representations
and its application with NNC are based on more estab-
lished fields:
Pattern Recognition: Statistical techniques,
parametric and nonparametric methods, classifier
design, nearest neighbor classification, curse of
dimensionality, similarity measures, feature selec-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
tion, feature extraction, prototype selection, and
clustering techniques.
Data Structures and Algorithms: Computational
requirements, compact storage structures, effi-
cient nearest neighbor search techniques, approxi-
mate search methods, algorithmic paradigms, and
divide-and-conquer approaches.
Database Management: Relational operators,
projection, cartesian product, data structures, data
management, queries, and indexing techniques.
MAIN THRUST
Pattern synthesis, compact representations followed by
its application with NNC, are described in this section.
Pattern Synthesis
Generation of artificial new patterns by using the given
set of patterns is called pattern synthesis. There are
two broad ways of doing pattern synthesis: model-
based pattern synthesis and instance-based pattern
synthesis.
In model-based pattern synthesis, a model (such as
the Hidden Markov model) or description (such as
probability distribution) of the data is derived first and
is then used to generate new patterns. This method can
be used to generate as many patterns as needed, but it has
two drawbacks. First, any model depends on the under-
lying assumptions; hence, the synthetic patterns gener-
ated can be erroneous. Second, deriving the model might
be computationally expensive. Another argument against
this method is that if pattern classification is the pur-
pose, then the model itself can be used without generat-
ing any patterns at all!
Instance-based pattern synthesis, on the other hand,
uses the given training patterns and some of the proper-
ties of the data. It can generate only a finite number of
TEAM LinG
Pattern Synthesis for Large-Scale Pattern Recognition
new patterns. Computationally, this method can be less
expensive than deriving a model. It is especially useful
for nonparametric methods, such as NNC- and Parzen-
window-based density estimation (Duda et al., 2000),
which directly use the training instances. Further, this
method can also result in reduction of the computa-
tional requirements.
This article presents two instance-based pattern syn-
thesis techniques called overlap-based pattern syn-
thesis and partition-based pattern synthesis and their
corresponding compact representations.
Overlap-based Pattern Synthesis
Let F be the set of features (or attributes). There may
exist a three-block partition of F, say, {A, B, C}, with the
following properties. For a given class, there is a depen-
dency (probabilistic) among features in A U B. Simi-
larly, features in B U C have a dependency. However,
features in A (or C ) can affect those in C (or A ) only
through features in B. That is, to state it more formally,
A and C are statistically independent, given B. Suppose
that this is the case and you are given two patterns, X =
(a1, b, c1) and Y = (a2, b, c2), such that a1 is a feature-
vector that can be assigned to the features in A, b to the
features in B, and c1 to the features in C. Similarly, a2, b,
and c2 are feature-vectors that can be assigned to fea-
tures in A, B, and C, respectively. Our argument, then, is
that the two patterns, (a 1, b, c 2) and (a2, b, c 1), are also
valid patterns in the same class or category as X and Y. If
these two new patterns are not already in the class of
patterns, it is only because of the finite nature of the set.
We call this generation of additional patterns an over-
lap-based pattern synthesis, because this kind of syn-
thesis is possible only if the two given patterns have the
same feature-values for features in B. In the given ex-
ample, feature-vector b is common between X and Y and
therefore is called the overlap. This method is suitable
only with discrete valued features (can also be of sym-
bolic or categorical types). If more than one such parti-
tion exists, then the synthesis technique is applied se-
quentially with respect to the partitions in some order.
One simple example to illustrate this concept is as
follows. Consider a supermarket sales database where
two records, (bread, milk, sugar) and (coffee, milk,
biscuits), are given. Assume that a known dependency
exists between (a) bread and milk, (b) milk and sugar, (c)
coffee and milk, and (d) milk and biscuits. The two new
Figure 1. A compact representation
records that can then be synthesized are (bread, milk,
biscuits) and (coffee, milk, sugar). Here, milk is the P
overlap. A compact representation in this case is shown
in Figure 1, where a path from left to right denotes a data
item or pattern. So you get four patterns total (two
original and two synthetic patterns) from the graph
shown in Figure 1. Association rules derived from asso-
ciation rule mining (Han & Kamber, 2000) can be used
to find these kinds of dependencies. Generalization of
this concept and its compact representation for large
datasets are described in the paragraphs that follow.
If the set of features, F, can be arranged in an order
such that F = {f1, f 2 , ..., fd } is an ordered set, with fk being
the k th feature and all possible three-block partitions
can be represented as Pi = {A i, B i, Ci } such that Ai = (f 1,
..., fa ), B i = (f a+1, ..., fb ), and Ci = (fb+1, ..., fd ), then the
compact representation called overlap pattern graph
is described with the help of an example.
Overlap Pattern Graph (OLP-graph)
Let F = (f 1, f2, f 3 , f4 , f5 ). Let two partitions satisfying the
conditional independence requirement be P1 = { {f1}, { f 2
, f3 }, { f4 , f5 }} and P2 = { {f 1, f2 } , {f3 , f4 }, { f5 }}. Let three
given patterns be (a,b,c,d,e), (p,b,c,q,r), and (u,v,c,q,w),
respectively. Because (b,c) is common between the1st
and 2nd patterns, two synthetic patterns that can be
generated are (a,b,c,q,r) and (p,b,c,d,e). Likewise, three
other synthetic patterns that can be generated are
(p,b,c,d,e), (p,b,c,q,w), and (a,b,c,q,w). (Note that the
last synthetic pattern is derived from two earlier syn-
thetic patterns.) A compact representation called over-
lap pattern graph (OLP-graph) for the entire set (in-
cluding both given and synthetic patterns) is shown in
Figure 2, where a path from left to right represents a
pattern. The graph is constructed by inserting the given
patterns, whereas the patterns that can be extracted out
of the graph form the entire synthetic set consisting of
both original and synthetic patterns. Thus, from the
graph in Figure 2, a total of eight patterns can be ex-
tracted, five of which are new synthetic patterns.
OLP-graph can be constructed by scanning the given
dataset only once and is independent of the order in
which the given patterns are considered. An approxi-
mate method for finding partitions, a method for con-
Figure 2. OLP-graph
903
TEAM LinG

struction of OLP-graph, and its application to NNC are
described in Viswanath, Murty, and Bhatnagar (2003).
For large datasets, this representation drastically re-
duces the space requirement.
Partition-based Pattern Synthesis
Let P = {A1, A2,...,Ak} be a k block partition of F such
that A1, A2,..., and Ak are statistically independent sub-
sets. Let Y be the given dataset and ProjA1(Y) be the
projection of Y onto features in the subset A1. Then the
cartesian product ProjA1(Y) X ProjA2(Y) X X ProjAk(Y)
is the synthetic set generated using this method.
The partiton P satisfying the above requirement can
be obtained from domain knowledge or from association
rule mining. Approximate partitioning methods using
mutual information or pair-wise correlation between the
features can also be used to get suitable partitions and are
experimentally demonstrated to work well with NNC in
Viswanath, Murty, and Bhatnagar (2004).
Partitioned Pattern Count Tree (PPC-
tree)
This compact representation, suitable for storing the
partition-based synthetic patterns, is based on Pattern
Count Tree (PC-tree) (Ananthanarayana, Murty, &
Subramanian, 2001), which is a prefix-tree-like struc-
ture. PPC-tree is a generalization of PC-tree such that a
PC-tree is built for each of the projected patterns, that is,
ProjAi(Y) for i = 1 to k. PPC-tree is a more compact
structure than PC-tree, which can be built by scanning the
dataset only once and is independent of the order in
which the given patterns are considered. The construc-
tion method and properties of PPC-tree can be found in
(Viswanath et al., 2004). As an example, consider four given
patterns, (a,b,c,d,e,f), (a,u,c,d,v,w), (a,b,p,d,e,f), and
(a,b,c,d,v,w), and a two-block partition, where the first
block consists of the first three features, and the second
block consists of the remaining three features. The corre-
sponding PPC-tree is shown in Figure 3, where each node
contains a feature-value and an integer called count,
which gives the number of patterns sharing that node. The
figure shows two PC-trees, one corresponding to each
Figure 3. PPC-tree
904
Pattern Synthesis for Large-Scale Pattern Recognition
block. A path in the PC-tree for block 1 (from root to leaf)
concatenated with a path in the PC-tree for block 2 gives
a pattern that can be extracted. So a total of six patterns
can be extracted from the structures shown in Figure 3.
An Application of Synthetic Patterns
with Nearest Neighbor Classifier
Classification and prediction methods are among vari-
ous elements of data mining and knowledge discovery,
including association rule mining, clustering, link analy-
sis, rule induction, and so forth (Wang, 2003). The
nearest neighbor classifier (NNC) is a very popular
nonparametric classifer (Dasarathy, 2002). It is widely
used in various fields because of its simplicity and
good performance. Its conceptual equivalents in other
fields (such as artificial intelligence) are instance-
based learning, lazy learning, memory-based reason-
ing, case-based reasoning, and so forth. Theoretically,
with an infinite number of samples, its error rate is
upper-bounded by twice the error of Bayes classifier
(Duda et al., 2000). NNC is a lazy classifier in the sense
that no general model (like a decision tree) is built until
a new sample needs to be classified. So NNC can easily
adapt to situations where the dataset changes frequently,
but computational requirements and curse of dimension-
ality are the two major issues that need to be addressed
in order to make the classifier suitable for data mining
applications (Dasarathy, 2002). Prototype selection
(Susheela Devi & Murty, 2002), feature selection (Guyon
& Elisseeff, 2003), feature extraction (Terabe et al., 2002),
compact representations of the datasets (Khan, Ding, &
Perrizo, 2002), and bootstrapping (Efron, 1979; Hamamoto
et al., 1997) are some of the remedies to tackle these
problems. But all these techniques need to be followed
one after the other; that is, they cannot be combined
together. Pattern synthesis techniques and compact
representations described in this article provide a unified
solution.
Efficient implementations of NNC that can di-
rectly work with OLP-graph are given in (Viswanath et al.,
2003). These implementations use dynamic programming
techniques to reuse the partial distance computations
and thus reduce the classification time requirement.
Partition-based pattern synthesis is presented in
(Viswanath et al., 2004), where an efficient NNC imple-
mentation with constant classification time is presented.
These methods are, in general, based on the divide-and-
conquer approach and are efficient to work directly with
the compact representation. Thus, the computational
requirements are reduced. Because the total number of
patterns considered (i.e., the size of the synthetic set) is
much larger than those in the given training set, the effect
of curse of dimensionality is reduced.
TEAM LinG
Pattern Synthesis for Large-Scale Pattern Recognition
FUTURE TRENDS
Integration of various data mining tasks, such as associa-
tion rule mining, feature selection, feature extraction,
classification, and so forth, by using compact and gener-
alized abstractions is a very promising direction. Apart
from NNC, other applications using synthetic patterns
and corresponding compact representations, including
decision tree induction, rule deduction, clustering, and so
forth, are also interesting areas that can give valuable
results.
CONCLUSION
Pattern synthesis is a novel technique that can enhance
the generalization ability of lazy learning methods such
as NNC by reducing the curse of dimensionality effect;
also, the corresponding compact representations can
reduce the computational requirements. In essence, it
derives a compact and generalized abstraction of the
given dataset. Overlap-based pattern synthesis and par-
tition-based pattern synthesis are two instance-based
pattern synthesis methods where OLP-graph and PPC-
tree are respective compact storage structures.
REFERENCES
Ananthanarayana, V. S., Murty, M. N., & Subramanian, D.
K. (2001). An incremental data mining algorithm for com-
pact realization of prototypes. Pattern Recognition, 34,
2249-2251.
Dasarathy, B. V. (2002). Data mining tasks and methods:
Classification: Nearest-neighbor approaches. In Hand-
book of data mining and knowledge discovery (pp. 288-
298). New York: Oxford University Press.
Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern
classification (2nd ed.) Wiley.
Efron, B. (1979). Bootstrap methods: Another look at
the jackknife. Annual Statistics, 7, 1-26.
Hamamoto, Y., Uchimura, S., & Tomita, S. (1997). A boot-
strap technique for nearest neighbor classifier design.
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 19(1), 73-79.
Han, J., & Kamber, M. (2000). Data mining: Concepts and
techniques. Morgan Kaufmann.
Guyon, I., & Elisseeff, A.(2003). An introduction to vari-
able and feature selection. Journal of Machine Learning P
Research, 3, 1157-1182.
Khan, M., Ding, Q., & Perrizo, W. (2002). K-nearest neighbor
classification on spacial data streams using P-trees. Proceed-
ings of the Pacific-Asia Conference on Knowledge Discovery
and Data Mining (pp. 517-528), Taiwan.
Susheela Devi, V., & Murty, M. N. (2002). An incremental
prototype set building technique. Pattern Recognition
Journal, 35, 505-513.
Terabe, M., Washio, T., Motoda, H., Katai, O., & Sawaragi,
T. (2002). Attribute generation based on association rules.
Knowledge and Information Systems, 4(3), 329-349.
Viswanath, P., Murty, M. N., & Bhatnagar, S. (2003).
Overlap pattern synthesis with an efficient nearest neigh-
bor classifier (Indian Institute of Science Tech. Rep. 1/
2003). Retrieved from http://keshava.csa.iisc.ernet.in/
techreports/2003/overlap.pdf
Viswanath, P., Murty, M. N., & Bhatnagar, S. (2004).
Fusion of multiple approximate nearest neighbor clas-
sifiers for fast and efficient classification [Electronic
version]. Information Fusion, 5(4), 239-250.
Wang, J. (2003). Data mining: Opportunities and chal-
lenges. Hershey, PA: Idea Group Publishing.
KEY TERMS
Bootstrapping: Generating artificial patterns from
the given original patterns. (This does not mean that the
artificial set is larger in size than the original set; also,
artificial patterns need not be distinct from the original
patterns.)
Curse of Dimensionality Effect: When the number
of samples needed to estimate a function (or model)
grows exponentially with the dimensionality of the data.
Compact and Generalized Abstraction of the
Training Set: A compact representation built by using
the training set from which not only the original patterns
but also some new synthetic patterns can be derived.
Prototype Selection: The process of selecting a
few representative samples from the given training set
suitable for the given task.
Training Set: The set of patterns whose class labels
are known and that are used by the classifier in classify-
ing a given pattern.
905
TEAM LinG
906
Physical Data Warehousing Design
Ladjel Bellatreche
LISI/ENSMA, France
Mukesh Mohania
IBM India Research Lab, India
INTRODUCTION
Recently, organizations have increasingly emphasized
applications in which current and historical data are
analyzed and explored comprehensively, identifying use-
ful trends and creating summaries of the data in order to
support high-level decision making. Every organization
keeps accumulating data from different functional units,
so that they can be analyzed (after integration), and
important decisions can be made from the analytical
results. Conceptually, a data warehouse is extremely
simple. As popularized by Inmon (1992), it is a subject-
oriented, integrated, time-invariant, non-updatable col-
lection of data used to support management decision-
making processes and business intelligence. A data
warehouse is a repository into which are placed all data
relevant to the management of an organization and from
which emerge the information and knowledge needed to
effectively manage the organization. This management
can be done using data-mining techniques, comparisons
of historical data, and trend analysis. For such analysis,
it is vital that (1) data should be accurate, complete,
consistent, well defined, and time-stamped for informa-
tional purposes; and (2) data should follow business rules
and satisfy integrity constraints. Designing a data ware-
house is a lengthy, time-consuming, and iterative pro-
cess. Due to the interactive nature of a data warehouse
application, having fast query response time is a critical
performance goal. Therefore, the physical design of a
warehouse gets the lions part of research done in the data
warehousing area. Several techniques have been devel-
oped to meet the performance requirement of such an
application, including materialized views, indexing tech-
niques, partitioning and parallel processing, and so forth.
Next, we briefly outline the architecture of a data ware-
housing system.
BACKGROUND
The conceptual architecture of a data warehousing sys-
tem is shown in Figure 1. Data of a warehouse is extracted
from operational databases (relational, object-oriented,
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
or relational-object) and external sources (legacy data,
other files formats) that may be distributed, autonomous,
and heterogeneous. Before integrating this data into a
warehouse, it should be cleaned to minimize errors and to
fill in missing information, when possible, and trans-
formed to reconcile semantic conflicts that can be found
in the various sources. The cleaned and transformed data
are integrated finally into a warehouse. Since the sources
are updated periodically, it is necessary to refresh the
warehouse. This component also is responsible for man-
aging the warehouse data, creating indices on data tables,
partitioning data, and updating meta-data. The ware-
house data contain the detail data, summary data, consoli-
dated data, and/or multi-dimensional data. The data typi-
cally are accessed and analyzed using tools, including
OLAP query engines, data mining algorithms, informa-
tion, visualization tools, statistical packages, and report
generators.
The meta-data generally is held in a separate reposi-
tory. The meta-data contain the informational data about
the creation, management, and usage of tools (e.g., ana-
lytical tools, report writers, spreadsheets and data-mining
tools) for analysis and informational purposes. Basically,
the OLAP server interprets client queries (the client inter-
acts with the front-end tools and passes these queries to
the OLAP server) and converts them into complex SQL
queries required to access the warehouse data. It also
might access the data warehouse. It serves as a bridge
between the users of the warehouse and the data con-
tained in it. The warehouse data also are accessed by the
OLAP server to present the data in a multi-dimensional
way to the front-end tools. Finally, the OLAP server
passes the multi-dimensional views of data to the front-
end tools, which format the data according to the clients
requirements.
The warehouse data are typically modeled multi-
dimensionally. The multi-dimensional data model has
been proved to be the most suitable for OLAP applica-
tions. OLAP tools provide an environment for decision
making and business modeling activities by supporting
ad-hoc queries. There are two ways to implement a multi-
dimensional data model: (1) by using the underlying
relational architecture (star schemas, snowflake schemas)
TEAM LinG
Physical Data Warehousing Design
Figure 1. A data warehousing architecture
- Relational - Select
- Transform
- Legacy
Meta - Clean
- Network Data
- Integrate
- Others - Refresh
Information Sources Warehouse Creation
and Management Component
to project a pseudo-multi-dimensional model (example
includes Informix Red Brick Warehouse); and (2) by using
true multi-dimensional data structures such as, arrays
(example includes Hyperion Essbase OLAP Server
Hyperion). The advantage of MOLAP architecture is that
it provides a direct multi-dimensional view of the data
whereas the ROLAP architecture is just a multi-dimen-
sional interface to relational data. On the other hand, the
ROLAP architecture has two major advantages: (i) it can
be used and easily integrated into other existing relational
database systems; and (ii) relational data can be stored
more efficiently than multi-dimensional data.
Data warehousing query operations include standard
SQL operations, such as selection, projection, and join. In
addition, it supports various extensions to aggregate
functions, such as percentile functions (e.g., top 20th
percentile of all products), rank functions (e.g., top 10
products), mean, mode, and median. One of the important
extensions to the existing query language is to support
multiple group-by, by defining roll-up, drill-down, and
cube operators. Roll-up corresponds to doing further
group-by on the same data object. Note that roll-up
operator is order sensitive; that is, when it is defined in the
extended SQL, the order of columns (attributes) matters.
The function of a drill-down operation is the opposite of
roll-up.
OLTP vs. OLAP
Relational database systems (RDBMS) are designed to
record, retrieve, and manage large amounts of real-time
transaction data and to keep the organization running by
supporting daily business transactions (e.g., update trans-
actions). These systems generally are tuned for a large
community of users, and the user typically knows what is
needed and generally accesses a small number of rows in
a single transaction. Indeed, relational database systems
are suited for robust and efficient Online Transaction
Processing (OLTP) on operational data. Such OLTP appli-
P
- Analysis
Ware - Data Clients
ROLAP/ Mining
house MOLAP
- Report
Data
Generator
- Others
OLAP Server Front-end Tools
cations have driven the growth of the DBMS industry in
the past three decades and will doubtless continue to be
important. One of the main objectives of relational sys-
tems is to maximize transaction throughput and minimize
concurrency conflicts. However, these systems generally
have limited decision support functions and do not ex-
tract all the necessary information required for faster,
better, and intelligent decision making for the growth of
an organization. For example, it is hard for an RDBMS to
answer the following query: What are the supply patterns
for product ABC in New Delhi in 2003, and how were they
different from the year 2002? Therefore, it has become
important to support analytical processing capabilities in
organizations for (1) the efficient management of organi-
zations, (2) effective marketing strategies, and (3) efficient
and intelligent decision making. OLAP tools are well
suited for complex data analysis, such as multi-dimen-
sional data analysis and to assist in decision support
activities that access data from a separate repository
called a data warehouse, which selects data from many
operational legacies, and possibly heterogeneous data
sources. The following table summarizes the differences
between OLTP and OLAP.
MAIN THRUST
Decision-support systems demand speedy access to data,
no matter how complex the query. To satisfy this objec-
tive, many optimization techniques exist in the literature.
Most of these techniques are inherited from traditional
relational database systems. Among them are materialized
views (Bellatreche et al., 2000; Jixue et al., 2003; Mohania
et al., 2000; Sanjay et al., 2000, 2001), indexing methods
(Chaudhuri et al., 1999; Jgens et al., 2001; Stockinger et
al., 2002), data partitioning (Bellatreche et al., 2000, 2002,
2004; Gopalkrishnan et al., 2000; Kalnis et al., 2002; Oracle,
2000; Sanjay et al., 2004), and parallel processing (Datta
et al., 1998).
907
TEAM LinG
 Physical Data Warehousing Design

Table 1. Comparison between OLTP and OLAP applications

Criteria OLTP OLAP

User Clerk, IT Professional Decision maker

Function Day-to-day operations Decision support
BD Design Application-oriented Subject-oriented
Data Current Historical, consolidated
View Detailed, flat relation Summarized, multi-dimensional
Usage Structured, repetitive Ad hoc
Unit of Work Short, simple transaction Complex query
Access Read/Write Append mostly
Records Accessed Tens Millions
Users Thousands Tens
Size MB-GB GB-TB
Metric Transaction throughput Query throughput

Materialized Views
Materialized views are used to precompute and store
aggregated data, such as sum of sales. They also can be
used to precompute joins with or without aggregations.
So, materialized views are used to reduce the overhead
associated with expensive joins or aggregations for a large
or important class of queries. Two major problems related
to materializing the views are (1) the view-maintenance
problem, and (2) the view-selection problem. Data in the
warehouse can be seen as materialized views generated
from the underlying multiple data sources. Materialized
views are used to speed up query processing on large
amounts of data. These views need to be maintained in
response to updates in the source data. This often is done
using incremental techniques that access data from under-
lying sources. In a data-warehousing scenario, accessing
base relations can be difficult; sometimes data sources
may be unavailable, since these relations are distributed
across different sources. For these reasons, the issue of
self-maintainability of the view is an important issue in
data warehousing. The warehouse views can be made self-
maintainable by materializing some additional information,
called auxiliary relations, derived from the intermediate
results of the view computation. There are several algo-
rithms, such as counting algorithm and exact-change algo-
rithm, proposed in the literature for maintaining material-
ized views.
To answer the queries efficiently, a set of views that are
closely related to the queries is materialized at the data
warehouse. Note that not all possible views are material-
ized, as we are constrained by some resource like disk
space, computation time, or maintenance cost. Hence, we
need to select an appropriate set of views to materialize
under some resource constraint. The view selection prob-
lem (VSP) consists in selecting a set of materialized views
that satisfies the query response time under some resource
constraints. All studies showed that this problem is an NP-
hard. Most of the proposed algorithms for the VSP are
static. This is because each algorithm starts with a set of
frequently asked queries (a priori known) and then se-
lects a set of materialized views that minimize the query
response time under some constraint. The selected ma-
terialized views will be a benefit only for a query belong-
ing to the set of a priori known queries. The disadvantage
of this kind of algorithm is that it contradicts the dynamic
nature of decision support analysis. Especially for ad-
hoc queries, where the expert user is looking for interest-
ing trends in the data repository, the query pattern is
difficult to predict.
Indexing Techniques
Indexing has been the foundation of performance tuning
for databases for many years. It creates access struc-
tures that provide faster access to the base data relevant
to the restriction criteria of queries. The size of the index
structure should be manageable, so that benefits can be
accrued by traversing such a structure. The traditional
indexing strategies used in database systems do not
work well in data warehousing environments. Most OLTP
transactions typically access a small number of rows;
most OLTP queries are point queries. B-trees, which are
used in the most common relational database systems,
are geared toward such point queries. They are well
suited for accessing a small number of rows. An OLAP
query typically accesses a large number of records for
summarizing information. For example, an OLTP transac-
tion would typically query for a customer who booked a
flight on TWA 1234 on April 25, for instance; on the
other hand, an OLAP query would be more like give me
the number of customers who booked a flight on TWA
1234 in one month, for example. The second query would
access more records that are of a type of range queries.
B-tree indexing scheme is not suited to answer OLAP
queries efficiently. An index can be a single-column or
multi-column table (or view). An index either can be
clustered or non-clustered. An index can be defined on

908

TEAM LinG
Physical Data Warehousing Design
one table (or view) or many tables using a join index. In the
data warehouse context, when we talk about index, we
refer to two different things: (1) indexing techniques and
(2) the index selection problem. A number of indexing
strategies have been suggested for data warehouses:
value-list index, projection index, bitmap index, bit-sliced
index, data index, join index, and star join index.
Data Partitioning and Parallel
Processing
The data partitioning process decomposes large tables
(fact tables, materialized views, indexes) into multiple
(relatively) small tables by applying the selection opera-
tors. Consequently, the partitioning offers significant
improvements in availability, administration, and table
scan performance Oracle9i.
Two types of partitioning are possible to decompose
a table: vertical and horizontal. In the vertical fragmenta-
tion, each partition consists of a set of columns of the
original table. In the horizontal fragmentation, each par-
tition consists of a set of rows of the original table. Two
versions of horizontal fragmentation are available: pri-
mary horizontal fragmentation and derived horizontal
fragmentation. The primary horizontal partitioning (HP)
of a relation is performed using predicates that are defined
on that table. On the other hand, the derived partitioning
of a table results from predicates defined on another
relation. In a context of ROLAP, the data partitioning is
applied as follows (Bellatreche et al., 2002): it starts by
fragmenting dimension tables, and then, by using the
derived horizontal partitioning, it decomposes the fact
table into several fact fragments. Moreover, by partition-
ing data of ROLAP schema (star schema or snowflake
schema) among a set of processors, OLAP queries can be
executed in a parallel, potentially achieving a linear
speedup and thus significantly improving query response
time (Datta et al., 1998). Therefore, the data partitioning
and the parallel processing are two complementary tech-
niques to achieve the reduction of query processing cost
in data warehousing environments.
FUTURE TRENDS
It has been seen that many enterprises are moving toward
building the Operational Data Store (ODS) solutions for
real-time business analysis. The ODS gets data from one
or more Enterprise Resource Planning (ERP) systems and
keeps the most recent version of information for analysis
rather than the history of data. Since the Client Relation-
ship Management (CRM) offerings have evolved, there is
a need for active integration of CRM with the ODS for real- P
time consulting and marketing (i.e., how to integrate ODS
with CRM via messaging system for real-time business
analysis).
Another trend that has been seen recently is that many
enterprises are moving from data warehousing solutions
to information integration (II). II refers to a category of
middleware that lets applications access data as though
they were in a single database, whether or not they are. It
enables the integration of data and content sources to
provide real-time read and write access in order to trans-
form data for business analysis and data warehousing and
to data placement for performance, currency, and avail-
ability. That is, we envisage that there will more focus on
integrating the data and contents rather than only inte-
grating structured data, as done in the data warehousing.
CONCLUSION
The data warehousing design is quite different from those
of transactional database systems, commonly referred as
Online Transaction Processing (OLTP) systems. A data
warehouse tends to be extremely large, and the informa-
tion in a warehouse usually is analyzed in a multi-dimen-
sional way. The main objective of a data warehousing
design is to facilitate the efficient query processing and
maintenance of materialized views. For achieving this
objective, it is important that the relevant data is materi-
alized in the warehouse. Therefore, the problems of select-
ing materialized views and maintaining them are very
important and have been addressed in this article. To
further reduce the query processing cost, the data can be
partitioned. That is, partitioning helps in reducing the
irrelevant data access and eliminates costly joins. Further,
partitioning at a finer granularity can increase the data
access and processing cost. The third problem is index
selection. We found that judicious index selection does
reduce the cost of query processing, but we also showed
that indices on materialized views improve the perfor-
mance of queries even more. Since indices and material-
ized views compete for the same resource (storage), we
found that it is possible to apply heuristics to distribute
the storage space among materialized views and indices
so as to efficiently execute queries and maintain material-
ized views and indexes.
It has been seen that enterprises are moving toward
building the data warehousing and operational data store
solutions.
909
TEAM LinG

REFERENCES
Bellatreche, L. et al. (2000). What can partitioning do for
your data warehouses and data marts? Proceedings of the
International Database Engineering and Applications
Symposium (IDEAS), Yokohoma, Japan.
Bellatreche, L. et al. (2002). PartJoin: An efficient storage
and query execution for data warehouses. Proceedings of
the International Conference on Data Warehousing and
Knowledge Discovery (DAWAK), Aix-en Provence,
France.
Bellatreche, L. et al. (2004). Bringing together partitioning,
materialized views and indexes to optimize performance of
relational data warehouses. Proceedings of the Interna-
tional Conference on Data Warehousing and Knowl-
edge Discovery (DAWAK), Zaragoza, Spain.
Bellatreche, L., Karlapalem, K., & Schneider, M. (2000). On
efficient storage space distribution among materialized
views and indices in data warehousing environments.
Proceedings of the International Conference on Infor-
mation and Knowledge Management (ACM CIKM),
McLearn, VA, USA.
Chaudhuri, S., & Narasayya, V. (1999). Index merging.
Proceedings of the International Conference on Data
Engineering (ICDE), Sydney, Australia.
Datta, A., Moon, B., & Thomas, H.M. (2000). A case for
parallelism in data warehousing and olap. Proceedings of
the DEXA Workshop, London, UK.
Gopalkrishnan, V., Li, Q., & Karlapalem, K. (2000). Effi-
cient query processing with associated horizontal class
partitioning in an object relational data warehousing
environment. Proceedings of the International Work-
shop on Design and Management of Data Warehouses,
Stockholm, Sweden.
Hyperion. (2000). Hyperion Essbase OLAP Server. http:/
/www.hyperion.com/
Inmon, W.H. (1992). Building the data warehouse. John
Wiley.
Jixue, L., Millist, W., Vincent, M., & Mohania, K. (2003).
Maintaining views in object-relational databases. Knowl-
edge and Information Systems, 5(1), 50-82.
Jrgens, M., Lenz, H. (2001). Tree based indexes versus
bitmap indexes: A performance study. International Jour-
nal of Cooperative Information Systems (IJCIS), 10(3),
355-379.
Kalnis, P. et al. (2002). An adaptive peer-to-peer network
for distributed caching of OLAP results. Proceedings of
910
Physical Data Warehousing Design
the International Conference on Management of Data
(ACM SIGMOD), Wisconsin, USA.
Mohania, M., & Kambayashi, Y. (2000). Making aggregate
views self-maintainable. Data & Knowledge Engineer-
ing, 32(1), 87-109.
Oracle Corp. (2004). Oracle9i enterprise edition partition-
ing option. Retrieved from http://otn.oracle.com/prod-
ucts/oracle9i/datasheets/partitioning.html
Sanjay, A., Chaudhuri, S., & Narasayya, V.R. (2000).
Automated selection of materialized views and indexes in
Microsoft SQL server. Proceedings of the 26th Interna-
tional Conference on Very Large Data Bases
(VLDB2000), Cairo, Egypt.
Sanjay, A., Chaudhuri, S., & Narasayya, V. (2001). Mate-
rialized view and index selection tool for Microsoft SQP
server 2000. Proceedings of the International Confer-
ence on Management of Data (ACM SIGMOD), CA.
Sanjay, A., Narasayya, V., & Yang, B. (2004). Integrating
vertical and horizontal partitioning into automated physi-
cal database design. Proceedings of the International
Conference on Management of Data (ACM SIGMOD),
Paris, France.
Stockinger, K. (2000). Bitmap indices for speeding up
high-dimensional data analysis. Proceedings of the Inter-
national Conference Database and Expert Systems Ap-
plications (DEXA), London, UK.
KEY TERMS
Bitmap Index: Consists of a collection of bitmap vec-
tors, each of which is created to represent each distinct
value of the indexed column. A bit i in a bitmap vector,
representing value x, is set to 1, if the record i in the indexed
table contains x.
Cube: A multi-dimensional representation of data that
can be viewed from different perspectives.
Data Warehouse: An integrated decision support
database whose content is derived from the various
operational databases. A subject-oriented, integrated,
time-variant, non-updatable collection of data used in
support of management decision-making processes.
Dimension: A business perspective useful for analyz-
ing data. A dimension usually contains one or more
hierarchies that can be used to drill up or down to different
levels of detail.
TEAM LinG
Physical Data Warehousing Design
Dimension Table: A table containing the data for one
dimension within a star schema. The primary key is used
to link to the fact table, and each level in the dimension has
a corresponding field in the dimension table.
Fact Table: The central table in a star schema, contain-
ing the basic facts or measures of interest. Dimension
fields are also included (as foreign keys) to link to each
dimension table.
Horizontal Partitioning: Distributing the rows of a
table into several separate tables.
Join Index: Built by translating restrictions on the
column value of a dimension table to restrictions on a large
fact table. The index is implemented using one of two
representations: row id or bitmap, depending on the P
cardinality of the indexed column.
Legacy Data: Data that you already have and use.
Most often, this takes the form of records in an existing
database on a system in current use.
Measure: A numeric value stored in a fact table or
cube. Typical examples include sales value, sales volume,
price, stock, and headcount.
Star Schema: A simple database design in which
dimensional data are separated from fact or event data. A
dimensional model is another name for star schema.
911
TEAM LinG
912
Predicting Resource Usage for Capital Efficient
Marketing
D.R. Mani
Massachusetts Institute of Technology, USA, and Harvard University, USA
Andrew L. Betz
Progressive Insurance, USA
James H. Drew
Verizon Laboratories, USA
INTRODUCTION
A structural conflict exists in businesses that sell services
whose production costs are discontinuous and whose
consumption is continuous but variable. A classic ex-
ample is in businesses where capital-intensive infrastruc-
ture is necessary for provisioning service, but the capac-
ity resulting from capital outlay is not always fully and
efficiently utilized. Marketing departments focus on ini-
tiatives that increase infrastructure usage to improve
both customer retention and ongoing revenue. Engineer-
ing and operations departments focus on the cost of
service provision to improve the capital efficiency of rev-
enue dollars received. Consequently, a marketing initiative
to increase infrastructure usage may be resisted by engi-
neering, if its introduction would require great capital
expense to accommodate that increased usage. This con-
flict is exacerbated when a usage-enhancing initiative tends
to increase usage variability so that capital expenditures
are triggered with only small increases in total usage.
A data warehouse whose contents encompass both
these organizational functions has the potential to medi-
ate this conflict, and data mining can be the tool for this
mediation. Marketing databases typically have customer
data on rate plans, usage, and past responses to market-
ing promotions. Engineering databases generally record
infrastructure locations, usages, and capacities. Other
information often is available from both general domains
to allow for the aggregation, or clustering, of customer
types, rate plans, and marketing promotions, so that
marketing proposals and their consequences can be evalu-
ated systematically to aid in decision making. These
databases generally contain such voluminous or compli-
cated data that classical data analysis tools are inad-
equate. In this article, we look at a case study where data
mining is applied to predicting capital-intensive resource
or infrastructure usage, with the goal of guiding marketing
decisions to enable capital-efficient marketing. Although
the data mining models developed in this article do not
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
Figure 1. Marketing initiatives to boost average demand
can indirectly increase peak demand to beyond capacity.
Capital Expense vs. Resource Capacity
100
80
Resource Capacity
Capacity
60
Peak Demand
40
20 Avg Demand
0
0 1 2 3 4 5 6
Capital Outlay
provide conclusive positions on specific marketing initia-
tives, and their engineering consequences, the usage
revenues, and infrastructure performance predicted by
these models provide systematic, sound, and quantita-
tive input for making balanced and cost-effective busi-
ness decisions.
BACKGROUND
In this business context, applying data mining
(Abramowicz & Zurada, 2000; Berry & Linoff, 2004; Han
& Kamber, 2000) to capital efficient marketing is illus-
trated here by a study from wireless telephony 1 (Green,
2000), where marketing plans2 introduced to utilize excess
off-peak network capacity3 (see Figure 1) potentially could
result in requiring fresh capital outlays by indirectly
driving peak demand to levels beyond current capacity.
We specifically consider marketing initiatives (e.g.,
rate plans with free nights and weekends) that are aimed
at stimulating off-peak usage. Given a rate plan with a fixed
peak minute allowance, availability of extra off-peak min-
TEAM LinG
Predicting Resource Usage for Capital Efficient Marketing
Figure 2. Flowchart describing data sources and data mining operations used in predicting busy-hour impact of
marketing initiatives
utes could potentially increase peak usage. The quantifi-
cation of this effect is complicated by the corporate reality
of myriad rate plans and geographically extensive and
complicated peak usage patterns. In this study, we use
data mining methods to analyze customer, call detail, rate
plan, and cell-site location data to predict the effect of
marketing initiatives on busy-hour4 network utilization.
This will enable forecasting network cost of service for
marketing initiatives, thereby leading to optimization of
capital outlay.
MAIN THRUST
Ideally, the capital cost of a marketing initiative is ob-
tained by determining the existing capacity, the increased
capacity required under the new initiative, and then fac-
toring the cost of the additional capital; data for a study
like this would come from a corporate data warehouse
(Berson & Smith, 1997) that integrates data from relevant
sources. Unfortunately, such detailed cost data are not
available in most corporations and businesses. In fact, in
many situations, the connection between promotional
marketing initiatives and capital cost is not even recog-
nized. In this case study, we therefore need to assemble
relevant data from different and disparate sources in order
to predict the busy-hour impact of marketing initiatives.
Data
The parallelograms in the flowchart in Figure 2 indicate
essential data sources for linking marketing initiatives to
busy-hour usage. Customer data characterize the cus-
tomer by indicating a customers mobile phone number(s),
P
lifetime value5, access charge, subscribed rate plan, and
peak and off-peak minutes used. Rate plan data provide
details for a given rate plan, including monthly charges,
allowed peak, off-peak, weekend minutes of use, per-
minute charges for excess use, long distance, roaming
charges, and so forth. Call detail data, for every call placed
in a given time period, provide the originating and termi-
nating phone numbers (and, hence, originating and termi-
nating customers), cell sites used in handling the call, call
duration, and other call details. Cell site location data
indicate the geographic location of cell sites, capacity of
each cell site, and details about the physical and electro-
magnetic configuration of the radio towers.
Data Mining Process
Figure 2 provides an overview of the analysis and data-
mining process. The numbered processes are described in
more detail to illustrate how the various components are
integrated into an exploratory tool that allows marketers
and network engineers to evaluate the effect of proposed
initiatives.
1. Cell-Site Clustering: Clustering cell sites using
the geographic location (latitude, longitude) results
in cell site clusters that capture the underlying
population density, with cluster area generally in-
versely proportional to population. This is a natural
consequence of the fact that heavily populated
urban areas tend to have more cell towers to cover
the large call volumes and provide good signal
coverage. The flowchart for cell site clustering is
included in Figure 3 with results of k-means cluster-
ing (Hastie, Tibshirani & Friedman, 2001) for the San
913
TEAM LinG
 Predicting Resource Usage for Capital Efficient Marketing

Figure 3. Steps 1and 2 in the data mining process outlined Francisco area shown in Figure 4. The cell sites in
in Figure 2 the area are grouped into four clusters, each cluster
approximately circumscribed by an oval.
2. Predictive Modeling: The predictive modeling
stage, shown in Figure 3, merges customer and rate
plan data to provide the data attributes (features)
used for modeling. The target for modeling is ob-
tained by combining cell site clusters with network
usage information. Note that the sheer volume of
call detail data makes its summarization and merg-
ing with other customer and operational data a
daunting one. See Berry and Linoff (2000) for a
discussion. The resulting dataset has related cus-
tomer characteristics and rate plan details for every
customer, matched up with that customers actual
network usage and the cell sites providing service
for that customer. This data can then be used to
build a predictive model. Feature selection (Liu &
Motoda, 1998) is performed, based on correlation
to target, with grouped interactions taken into
account. The actual predictive model is based on
linear regression (Hand, Mannila & Smyth, 2001;
Hastie, Tibshirani & Friedman, 2001), which, for
this application, performs similar to neural network
models (Haykin, 1994). Note that the sheer number
of customer and rate plan characteristics requires
the variable reduction capabilities of a data-mining
solution.

For each of the four cell site clusters shown in Figure

4, Figure 5 shows the actual vs. the predicted busy-hour
usage for calls initiated during a specific month in the San

Figure 4. Clustering cell sites in the San Francisco area, based on geographic position. Each spot represents a cell
site, with the ovals showing approximate location of cell site clusters.

914

TEAM LinG
Predicting Resource Usage for Capital Efficient Marketing

Figure 5. Regression modeling results showing predicted plans into a small number of groups. Without such
vs. actual busy-hour usage for the cell site clusters shown clustering, the complex data combinations and analy- P
in Figure 4. ses needed to predict busy-hour usage will result in
very little predictive power. Clusters ameliorate the
situation by providing a small set of representative
Ac tu a l vs . P red ic ted B H U sa g e
Cell Cluster 1
Ac tu al vs . P re dic te d B H U sa g e
Cell Cluster 2
cases to use in the prediction.
10 10

Similar to cell site clustering, based on empirical explo-

Predicted BH Usage (min)

8 8

Predicted BH Usage (min)

y = 0.6615x + 0.3532

6
R2 = 0.672
6
y = 0.3562x + 0.5269
R2 = 0.3776
ration, we decide on using four clusters for both custom-
4 4
ers and rate plans. Results of the clustering for customers
2 2
and applicable rate plans in the San Francisco area are
shown in Figures 6 and 7.
0 0
0 2 4 6 8 10 0 2 4 6 8 10
Actual BH Usage (min) Actual BH Usage (min)

Ac tu a l v s. P red icted B H U s ag e A ctu a l vs . P red icte d B H U s ag e

Cell Cluster 3 Cell Cluster 4
5. Integration and Application of the Predictive Model:
10 10
Stringing all the pieces together, we can start with
Predicted BH Usage (min)

a proposed rate plan and an inwards projection6 of

Predicted BH Usage (min)

8 8

6 6

4
y = 0.3066x + 0.6334
R2 = 0.3148 4
y = 0.3457x + 0.4318 the expected number of customers who would sub-
R2 = 0.3591

2 2
scribe to the plan and predict the busy-hour usage
0
0 2 4 6 8 10
0 on targeted cell sites. The process is outlined in
Figure 8. Inputs labeled A, B, and C come from the
0 2 4 6 8 10
Actual BH Usage (min) Actual BH Usage (min)

respective flowcharts in Figures 2, 6, and 7.

Francisco region. With statistically significant R2 correla- Validation

tion values ranging from about 0.31 to 0.67, the models,
while not perfect, are quite good at predicting cluster- Directly evaluating and validating the analytic model
level usage. developed would require data summarizing the capital
cost of a marketing initiative. This requires determining
3. and 4. Customer and Rate Plan Clustering: Using k- the existing capacity, the increased capacity required
means clustering to cluster customers based on under the new initiative, and then factoring the cost of the
their attribute, and rate plans, based on their salient additional capital. Unfortunately, as mentioned earlier,
features, results in categorizing customers and rate such detailed cost data are generally unavailable. There-

Figure 6. Customer clustering flowchart and results of Figure 7. Rate plan clustering flowchart and clustering
customer clustering. The key attributes identifying the results. The driving factor that distinguishes rate plan
four clusters are access charge (ACC_CHRG), lifetime clusters include per-minute charges (intersystem, peak,
value (LTV_PCT), peak minutes of use (PEAK_MOU), and off-peak) and minutes of use.
and total calls (TOT_CALL).

915

TEAM LinG
 Predicting Resource Usage for Capital Efficient Marketing

Figure 8. Predicting busy-hour usage at a cell site for a given marketing initiative

fore, we validate the analytic model by estimating its ity could be useful, for example, when price elasticity data
impact and comparing the estimate with known param- suggest inward projection differences by cluster.
eters. Starting at the cluster level, we applied the predictive
We began this estimation by identifying the 10 most model to estimate the average busy-hour usage for each
frequent rate plans in the data warehouse. From this list, customer on each cell tower. These cluster-level predic-
we selected one rate plan for validation. Based on market- tions were disaggregated to the cellular-tower level by
ing area and geographic factors, we assume an equal assigning busy-hour minutes proportionate to total min-
inwards projection for each customer cluster. Of course, utes for each cell tower within the respective cellular-
our approach allows for, but does not require, different tower cluster. Following this, we then calculated the
inward projections for the customer clusters. This flexibil- actual busy-hour usage per customer of that rate plan
across the millions of call records. In Figure 9, a scatter
plot of actual busy-hour usage against predicted busy-
Figure 9. Scatter plot showing actual vs. predicted hour usage, with an individual cell tower now the unit of
busy-hour usage for each cell site, for a specific rate plan analysis, reveals an R2 correlation measure of 0.13.
The estimated model accuracy dropped from R2s in the
mid 0.30s for cluster-level data (Figure 5) to about 0.13
Actual vs. Predicted BH Usage for Each Cell Site when the data were disaggregated to the cellular tower
Rate Plan RDCDO
level (Figure 9). In spite of the relatively low R2 value, the
1 correlation is statistically significant, indicating that this
0.8
approach can make contributions to the capital estimates
Predicted BH Usage

y = 1.31x - 0.1175
R2 = 0.1257
of marketing initiatives. However, the model accuracy on
0.6 disaggregated data was certainly lower than the effects
0.4
observed at the cluster level. The reasons for this loss in
accuracy probably could be attributed to the fine-grained
0.2
disaggregation, the large variability among the cell sites
0 in terms of busy-hour usage, the proportionality assump-
0 0.1 0.2 0.3 0.4 0.5 0.6 tion made in disaggregating, and model sensitivity to
Actual BH Usage inward projections. These data point out both an oppor-

916

TEAM LinG
Predicting Resource Usage for Capital Efficient Marketing

Figure 10. Lifetime value (LTV) distributions for customers with below and above average usage and BH usage
indicates that high-LTV customers also impact the network with high BH usage P

LTV distribution for LTV distribution for

customers with less than customers with more than
average usage and BH average usage and BH
usage usage

tunity (that rate plans can be intelligently targeted to
specific locations with excess capacity) and a threat (that
high busy-hour volatility would lead engineering to be
cautious in allowing usage-increasing plans).
Business Insights
In the context of the analytic model, the data also can
reveal some interesting business insights. First, observe
the customer lifetime value density plots as a function of
strain that they place on the network. The left panel in
Figure 10 shows LTV density for customers with below-
average total usage and below-average busy-hour usage.
The panel on the right shows LTV density for customers
with above-average total usage and above-average busy-
hour usage. Although the predominant thinking in mar-
keting circles is that higher LTV is always better, the data
suggest this reasoning should be tempered by whether

Figure 11. Exploring customer clusters and rate plan clusters in the context of BH usage shows that customers in a
cluster who subscribe to a class of rate plans impact BH usage differently from customers in a different cluster

BH Impact by Customer & Rate Plan Cluster

B H Impact by C ustome r C luste r

0.25 0.25
Proportion of Heavy BH Users

0.2
0.2 0.2
Proportion of Heavy BH Users

0.15 0.15 0.15

0.1 0.1 0.1

0.05 0.05 0.05

0 0
0
1 2 3 4 1 4
Customer Cluster Customer Cluster
A
Rate Plan Cluster 2 Rate Plan Cluster 3 B

917

TEAM LinG
 Predicting Resource Usage for Capital Efficient Marketing
the added value in revenue offsets the disproportionate
strain on network resources. This is the basis for a
fundamental tension between marketing and engineering
functions in large businesses.
Looking at busy-hour impact by customer cluster and
rate plan cluster is also informative, as shown in Figure 11.
For example, if we define heavy BH users as customers
who are above average in total minutes as well as busy-
hour minutes, we can see main effect differences across
customer clusters (Figure 11a).
This is not entirely surprising, as we have already seen
that LTV was a major determinant of customer cluster and
heavy BH customers also skewed towards having higher
LTV. However, there was an unexpected crossover inter-
action of rate plan cluster by customer cluster when heavy
BH users were the target (Figure 11b). The implication is
that controlling for revenue, certain rate plan types are
more network-friendly, depending on the customer clus-
ter under study. Capital-conscious marketers in theory
could tailor their rate plans to minimize network impact by
tuning rate plans more precisely to customer segments.
FUTURE TRENDS
The proof of concept case study sketched here describes
an analytical effort to optimize capital costs of marketing
programs, while promoting customer satisfaction and
retention by utilizing the business insights gained to
tailor marketing programs to better match the needs and
requirements of customers (Lenskold, 2003). As database
systems incorporate data-mining engines (e.g., Oracle
data mining [Oracle, 2004]), future software and customer
relationship management applications will automatically
incorporate such an analysis to extract optimal recommen-
dations and business insights for maximizing business
return on investment and customer satisfaction, leading
to effective and profitable one-on-one customer relation-
ship management (Brown, 2000; Gilmore & Pine, 2000;
Greenberg, 2001).
CONCLUSION
We have made the general argument that a comprehensive
company data warehouse and broad sets of models ana-
lyzed with modern data mining techniques can resolve
tactical differences between different organizations within
the company and provide a systematic and sound basis
for business decision making. The role of data mining in
so doing has been illustrated here in mediating between
the marketing and engineering functions in a wireless
telephony company, where statistical models are used to
918
target specific customer segments with rate plan promo-
tions in order to increase overall usage (and, hence,
retention), while more efficiently using, but not exceeding
network capacity. A further practical advantage of this
data-mining approach is that all of the customer groups,
cell site locations, and rate plan promotions are simplified
through clustering in order to simplify their characteristic
representation facilitating productive discussion among
upper-management strategists.
We have sketched several important practical busi-
ness insights that come from this methodology. A basic
concept to be demonstrated to upper management is that
busy-hour usage, the main driver of equipment capacity
needs, varies greatly by customer segment and by general
cell site grouping (see Figure 9) and that usage can be
differently volatile over site groupings (see Figure 5).
These differences point out the potential need for target-
ing specific customer segments with specific rate plan
promotions in specific locations. One interesting example
is illustrated in Figure 11, where two customer segments
respond differently to each of two candidate rate plans,
one group decreasing its BH usage under one plan and the
other decreasing it under the other plan. This leads to the
possibility that certain rate plans should be offered to
specific customer groups in locations where there is little
excess equipment capacity, but others can be offered
where there is more slack in capacity.
Of course, there are many organizational, implementa-
tion, and deployment issues associated with this inte-
grated approach. All involved business functions must
accept the validity of the modeling and its results, and this
concurrence requires the active support of upper manage-
ment overseeing each function. Second, these models
should be repeatedly run to assure their relevance in a
dynamic business environment, as wireless telephony is
in our illustration. Third, provision should be made for
capturing the sales and engineering consequences of the
decisions made from this approach, to be built into future
models. Despite these organizational challenges, busi-
ness decision makers informed by these models hopefully
may resolve a fundamental business rivalry.
REFERENCES
Abramowicz, W., & Zurada, J. (2000). Knowledge discov-
ery for business information systems. Kluwer.
Berry, M.A., & Linoff, G.S. (2000). Mastering data min-
ing: The art and science of customer relationship man-
agement. Wiley.
Berry, M.A., & Linoff, G.S. (2004). Data mining tech-
niques: For marketing, sales, and customer relationship
management. Wiley.
TEAM LinG
Predicting Resource Usage for Capital Efficient Marketing
Berson, A., & Smith, S. (1997). Data warehousing, data
mining and OLAP. McGraw-Hill.
Brown, S.A. (2000). Customer relationship management:
Linking people, process, and technology. John Wiley.
Drew, J., Mani, D.R., Betz, A., & Datta, P. (2001). Targeting
customers with statistical and data-mining techniques.
Journal of Services Research, 3(3), 205-219.
Gilmore, J., & Pine, J. (2000). Markets of one. Harvard
Business School Press.
Green, J.H. (2000). The Irwin handbook of telecommuni-
cations. McGraw-Hill.
Greenberg, P. (2001). CRM at the speed of light: Captur-
ing and keeping customers in Internet real time. Mc-
Graw Hill.
Han, J., & Kamber, M. (2000). Data mining: Concepts and
techniques. Morgan Kaufmann.
Hand, D.J., Mannila, H., & Smyth, P. (2001). Principles of
data mining. Bradford Books.
Hastie, T., Tibshirani, R., & Friedman, J. (2001). The
elements of statistical learning: Data mining, inference,
and prediction. Springer.
Haykin, S. (1994). Neural networks: A comprehensive
foundation. Prentice-Hall.
Lenskold, J.D. (2003). Marketing ROI: The path to cam-
paign, customer, and corporate profitability. Mc-Graw
Hill.
Liu, H., & Motoda, H. (1998). Feature selection for knowl-
edge discovery and data mining. Kluwer.
Mani, D.R., Drew, J., Betz, A., & Datta, P. (1999). Statistics
and data mining techniques for lifetime value modeling.
Proceedings of the Fifth ACM SIGKDD Conference on
Knowledge Discovery and Data Mining.
Oracle. (2004). Oracle Corporation. Retrieved from http:/
/otn.oracle.com/products/bi/odm/index.html
KEY TERMS
Busy Hour: The hour at which a mobile telephone
network handles the maximum call traffic in a 24-hour
period. It is that hour during the day or night when the
product of the average number of incoming calls and
average call duration is at its maximum.
Capital-Efficient Marketing: Marketing initiatives
that explicitly take into account and optimize, if possible, P
the capital cost of provisioning service for the introduced
initiative or promotion.
Feature Selection: The process of identifying those
input attributes that contribute significantly to building
a predictive model for a specified output or target.
Inwards Projection: Estimating the expected number
of customers that would sign on to a marketing promotion,
based on customer characteristics and profiles.
k-Means Clustering: A clustering method that groups
items that are close together, based on a distance metric
like Euclidean distance, to form clusters. The members in
each of the clusters can be described succinctly using the
mean (or centroid) of the respective cluster.
Lifetime Value: A measure of the profit generating
potential, or value, of a customer; a composite of expected
tenure (how long the customer stays with the business
provider) and expected revenue (how much a customer
spends with the business provider).
Predictive Modeling: Use of statistical or data-mining
methods to relate attributes (input features) to targets
(outputs) using previously existing data for training in
such a manner that the target can be predicted for new data
based on the input attributes alone.
ENDNOTES
1
Wireless telephony here refers to both the engi-
neering and commercial aspects of mobile telecom-
munications by radio signal, also known as cellular
telephony.
2
In this article, the terms marketing plan and mar-
keting initiative refer to promotions and incen-
tives, generally mediated through a rate plana
mobile calling plan that specifies minute usage al-
lowances at peak and off-peak times, monthly
charge, cost of extra minutes, roaming and long-
distance charges, and so forth.
3
The capacity of a cellular telephone network is the
maximum number of simultaneous calls that can be
handled by the system at any given time. Times
during the day when call volumes are higher than
average are referred to as peak hours; off-peak
hours entail minimal call trafficlike nights and
weekends.
919
TEAM LinG
 Predicting Resource Usage for Capital Efficient Marketing
4
Busy hour (BH) is that hour during which the net-
work utilization is at its peak. It is that hour during
the day or night when the product of the average
number of incoming calls and average call duration
is at its maximum.
5
The well-established and widely used concept of
customer lifetime value (LTV) captures expected
revenue and expected tenure, preferably personal-
920
ized for each individual customer. (Drew, et al., 2001;
Mani et al., 1999).
6
Inwards projection is a term used to denote an
estimate of the number of customers that would
subscribe to a new (or enhanced) rate plan, with
details reasonably summarizing characteristics of
customers expected to sign on to such a plan.
TEAM LinG

Privacy and Confidentiality Issues in Data Mining
Ycel Saygin
Sabanci University, Turkey
INTRODUCTION
Data regarding people and their activities have been
collected over the years, which has become more perva-
sive with widespread usage of the Internet. Collected data
usually are stored in data warehouses, and powerful data
mining tools are used to turn it into competitive advan-
tage. Besides businesses, government agencies are among
the most ambitious data collectors, especially in regard to
the increase of safety threats coming from global terrorist
organizations. For example, CAPPS (Computer Assisted
Passenger Prescreening System) collects flight reserva-
tion information as well as commercial information about
passengers. This data, in turn, can be utilized by govern-
ment security agencies. Although CAPPS represents US
national data collection efforts, it also has an effect on
other countries. The following sign at the KLM ticket desk
in Amsterdam International Airport illustrates the inter-
national level of data collection efforts: Please note that
KLM Royal Dutch Airlines and other airlines are required
by new security laws in the US and several other countries
to give security customs and immigration authorities
access to passenger data. Accordingly, any information
we hold about you and your travel arrangements may be
disclosed to the concerning authorities of these countries
in your itinerary. This is a very striking example of how
the confidential data belonging to citizens of one country
could be handed over to authorities of some other country
via newly enforced security laws. In fact, some of the
largest airline companies in the US, including American,
United, and Northwest, turned over millions of passenger
records to the FBI, according to the New York Times
(Schwartz & Maynard, 2004).
Aggressive data collection efforts by government
agencies and enterprises have raised a lot of concerns
among people about their privacy. In fact, the Total
Information Awareness (TIA) project, which aims to build
a centralized database that will store the credit card
transactions, e-mails, Web site visits, and flight details of
Americans was not funded by Congress due to privacy
concerns. Although the privacy of individuals is pro-
tected by regulations, such regulations may not be enough
to ensure privacy against new technologies such as data
mining. Therefore, important issues need to be consid-
ered by the data collectors and the data owners. First of
all, the data itself may contain confidential information
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
921
P
that needs to be secured. More importantly, the privacy
of the data owners needs to be taken into consideration,
depending on how the data will be used. Privacy risks
increase even more when we consider powerful data-
mining tools that can be used to extract confidential
information. We will try to address the privacy and con-
fidentiality issues that relate to data-mining techniques
throughout this article.
BACKGROUND
Historically, researchers have been more interested in the
security aspect of data storage and transfer. Data security
is still an active area of research in relation to new data
storage and transfer models, such as XML. Also, new
media of data transfer, such as wireless, introduce new
challenges.
The most important work on data security that relates
to the privacy and confidentiality issues in data analysis
is statistical disclosure control and data security in statis-
tical databases (Adam & Wortmann, 1989). The basic idea
of a statistical database is to let users obtain statistical or
aggregate information from a database, such as the aver-
age income and maximum income in a given department.
This is done while limiting the disclosure of confidential
information, such as the salary of a particular person. The
concept of k-anonymity, which ensures that the obtained
values refer to at least k individuals (Sweeney, 2002), was
proposed as a measure for the degree of confidentiality in
aggregate data. In this way, identification of individuals
can be blocked up to a certain degree. However, succes-
sive query results over the database may pose a risk to
security, since a combination of intelligently issued que-
ries may infer confidential data. The general problem of
adversaries being able to obtain sensitive data from non-
sensitive data is called the inference problem (Farkas &
Jajodia, 2003).
MAIN THRUST
Data mining is considered to be a tool for analyzing large
collections of data. As a result, government and industry
seek to exploit its potential by collecting extensive infor-
mation about people and their activities. Coupled with its
TEAM LinG

power to extract previously unknown knowledge, data
mining also has become one of the main targets of privacy
advocates. In reference to data mining, there are two main
concerns regarding privacy and confidentiality:
(1) Protecting the confidentiality of data and privacy of
individuals against data mining methods.
(2) Enabling the data mining algorithms to be used over
a database without revealing private and confiden-
tial information.
We should note that, although both of the tracks seem
to be similar to each other, their purposes are different. In
the first case, the data may not be confidential, but data-
mining tools could be used to infer confidential informa-
tion. Therefore, some sanitization techniques are needed
to modify the database, so that privacy concerns are
alleviated. In the second track, the data are considered
confidential and are perturbed before they are given to a
third party. Both of the approaches are necessary in order
to achieve full privacy of individuals and will be discussed
in detail in the following subsections.
Privacy Preserving Data Mining
In the year 2000, Agrawal and Srikant, in their seminal paper
published in ACM SIGMOD proceedings, coined the term
privacy preserving data mining. Based on this work,
privacy-preserving data mining can be defined as a tech-
nology that enables data-mining algorithms to work on
encrypted or perturbed data (Agrawal & Srikant, 2000). We
think of it as mining without seeing the actual data. The
authors have pointed out that the collected data may be
outsourced to third parties to perform the actual data
mining. However, before the data could be handed over to
third parties, the confidential values in the database, such
as the salary of employees, needs to be perturbed. The
research results are for specific data-mining techniques;
namely, decision tree construction for classification. The
main idea is to perturb the confidential data values in a way
that the original data distribution could be reconstructed
but not the original data values. In this way, the classifica-
tion techniques that rely on the distribution of the data to
construct a model can still work within a certain error margin
that the authors approve.
Another interesting work on privacy-preserving data
mining was conducted in the US at Purdue University
(Kantarcioglu & Clifton, 2002). This work was in the
context of association rules in a distributed environment.
The authors considered a case in which there were mul-
tiple companies that had their own confidential local
databases that they did not want to share with others.
When the datasets of the individual companies have the
same schema, then this means that the data are distributed
922
Privacy and Confidentiality Issues in Data Mining
horizontally. If the schema of the local databases are
complementary to each other, then this means that the
data are vertically distributed. The Purdue research group
considered both horizontal and vertical data distribution
for privacy-preserving association rule mining. In both
cases, the individual association rules together with their
statistical properties were assumed to be confidential.
Also, secure multi-part computation techniques were
employed, based on specialized encryption practices, to
make sure that the confidential association rules were
circulated among the participating companies in encrypted
form. The resulting global association rules can be ob-
tained in a private manner without each company knowing
which rule belongs to which local database.
Both of the seminal works on privacy-preserving data
mining (Agrawal & Srikant, 2000; Kantarcioglu & Clifton,
2002) have shaped the research in this area. They show
how data mining can be done on private data in a central-
ized and distributed environment. Although a specific
data-mining model was the target in both of these papers,
these authors initiated ideas that could be applied to other
data-mining models.
Privacy and Confidentiality Against
Data Mining
Data mining and data warehousing technology have a
great impact on data analysis and business decision
making, which motivated the collection of data on practi-
cally everything, ranging from customer purchases to
navigation patterns of Web site visitors. As a result of
data collection efforts, privacy advocates in the US are
now raising their voices against the misuse of data, even
if it was collected for security purposes. The following
quote from the New York Times article demonstrates the
point of privacy against data mining (Markoff, 2002):
Pentagon has released a study that recommends the
government to pursue specific technologies as potential
safeguards against the misuse of data-mining systems
similar to those now being considered by the government
to track civilian activities electronically in the United
States and abroad.
This shows us that even the most aggressive data
collectors in the US are aware of the fact that the data-
mining tools could be misused, and we need a mechanism
to protect the confidentiality and privacy of people.
The initial work by Chang and Moskovitz (2000) points
out the possible threats imposed by data mining tools. In
this work, the authors argue that even if some of the data
values are deleted (or hidden) in a database, there is still
a threat of the hidden data being recovered. They show
that a classification model could be constructed using
TEAM LinG
Privacy and Confidentiality Issues in Data Mining
part of the released data as a training set, which can be used
further to predict the hidden data values. Chang and
Moskovitz (2000) propose a feedback mechanism that will
try to construct a prediction model and go back to the
database to update it with the aim of blocking the predic-
tion of confidential data values.
Another aspect of privacy and confidentiality against
data mining is that data-mining results, such as the pat-
terns in a database, could be confidential. For example, a
database could be released with the thought that the
confidential values are hidden and cannot be queried.
However, there may be patterns in the database that are
confidential. This issue was first pointed out by OLeary
(1991). Clifton and Marks (1996) further elaborate on the
issue of patterns being confidential by specific examples
of data-mining models, such as association rules. The
association rules, which are very popular in retail market-
ing, may contain confidential information that should not
be disclosed. In Verykios, et al. (2004), a way of identifying
confidential association rules and sanitizing the database
to limit the disclosure of the association rules are dis-
cussed. Furthermore, the work illustrates ways in which
heuristics for updating the database can reduce the signifi-
cance of the association rules in terms of their support and
confidence. The main idea is to change the transactions in
the database that contribute to the support and confi-
dence of the sensitive (or private) association rules in a
way that the support and/or confidence of the rules de-
crease with a limited effect on the non-sensitive rules.
FUTURE TRENDS
Privacy and confidentiality issues in data mining will be
more and more crucial as data collection efforts increase
and the type of data collected becomes more diverse. A
typical example of this is the usage of RFID tags and mobile
phones, which reveal our sensitive location information.
As more data become available and more tools are imple-
mented to search this data, the privacy risks will increase
even more. Consider the World Wide Web, which is a huge
data repository with very powerful search engines work-
ing on top of it. An adversary can find the phone number
of a person from some source and use the search engine
Google to obtain the address of that person. Then, by
applying the address to a tool such as Mapquest, door-to-
door driving directions to the corresponding persons
home could be found easily. This is a very simple example
of how the privacy of a person could be in danger by
integrating data from a couple of sources, together with a
search engine over a data repository. Data integration over
multiple data repositories will be one of the main chal-
lenges in privacy and confidentiality of data against data
mining.
The New York Times article cited previously high-
lights another important aspect of privacy issues in data P
mining by saying, Perhaps the strongest protection
against abuse of information systems is Strong Audit
mechanisms we need to watch the watchers (Markoff,
2002). This confirms that data-mining tools also should
be monitored, and users access to data via data-mining
tools should be controlled. Although there is some initial
work on this issue (Oliveira, Zaiane & Saygin, 2004), how
this could be achieved is still an open problem waiting to
be addressed, since there is no available access control
mechanism specifically developed for data-mining tools
similar to the one employed in database systems.
Privacy-preserving data-mining techniques have been
proposed, but they may not fully preserve the privacy, as
pointed out in Agrawal and Aggarwal (2001) and
Evfimievski, Gehrke, and Srikant (2003). Therefore, pri-
vacy metrics and benchmarks are needed to assess the
privacy threats and the effectiveness of the proposed
privacy-preserving data-mining techniques or the pri-
vacy breaches introduced by data-mining techniques.
CONCLUSION
Data mining has found a lot of applications in the indus-
try and government, due to its success in combining
machine learning, statistics, and database fields with the
aim of turning heaps of data into valuable knowledge.
Widespread usage of the Internet, especially for e-com-
merce and other services, has led to the collection and
storage of more data at a lower cost. Also, large data
repositories about individuals and their activities, coupled
with powerful data mining tools, have increased fears
about privacy. Therefore, data mining researchers have
felt the urge to address the privacy and confidentiality
issues. Although privacy-preserving data mining is still
in its infancy, some promising results have been achieved
as the outcomes of initial studies. For example, data
perturbation techniques enable data-mining models to
be built on private data, and encryption techniques allow
multiple parties to mine their databases as if their data
were stored in a central database. The threat of powerful
data-mining tools revealing confidential information also
has been addressed. The initial results in this area have
shown that confidential patterns in a database can be
concealed by specific hiding techniques. For the ad-
vance of the data-mining technology, privacy issues
need to be investigated more, and the current problems,
such as privacy leaks in privacy-preserving data-mining
algorithms, and the scalability issues need to be re-
solved. In summary, data-mining technology is needed
to make our lives easier and to increase our safety
standards, but, at the same time, privacy standards should
923
TEAM LinG

be established and enforced on data collectors in order to
protect the privacy of data owners against the misuse of
the collected data.
REFERENCES
Adam, N.R., & Wortmann, J.C. (1989). Security-control
methods for statistical databases: A comparative study.
ACM Computing Surveys, 21(4), 515-556.
Agrawal, D., & Aggarwal, C. (2001). On the design and
quantification of privacy preserving data mining algo-
rithms. Proceedings of ACM Symposium on Principles of
Database Systems, Santa Barbara, California.
Agrawal, R., & Srikant, R. (2000). Privacy preserving data
mining. Proceedings of SIGMOD Conference, Dallas,
Texas.
Clifton, C., & Marks, D. (1996). Security and privacy
implications of data mining. Proceedings of ACM Work-
shop on Data Mining and Knowledge Discovery, Montreal
Canada.
Evfimievski, A.V., Gehrke, J., & Srikant, R. (2003). Limiting
privacy breaches in privacy preserving data mining. Pro-
ceedings of ACM Symposium on Principles of Database
Systems, San Diego, California.
Farkas, C., & Jajodia, S. (2003). The inference problem: A
survey. ACM SIGKDD Explorations, 4(2), 6-12.
Kantarcioglu, M., & Clifton, C. (2002). Privacy-preserving
distributed mining of association rules on horizontally
partitioned data. Proceedings of The ACM SIGMOD
Workshop on Research Issues on Data Mining and Knowl-
edge Discovery, Madison, Wisconsin.
Markoff, J. (2002, December 19). Study seeks technology
safeguards for privacy. New York Times (p. 18).
OLeary, D.E. (1991). Knowledge discovery as a threat to
database security. In G. Piatetsky-Shapiro, & W.J. Frawley
(Eds.), Knowledge discovery in databases (pp. 507-516).
AAI/MIT Press.
Oliveira, S., Zaiane, O., & Saygin, Y. (2004). Secure asso-
ciation rule sharing. Proceedings of the 8th Pacific-Asia
924
Privacy and Confidentiality Issues in Data Mining
Conference on Knowledge Discovery and Data Mining,
Sydney, Australia.
Saygin, Y., Verykios, V.S., & Clifton, C. (2001). Using
unknowns to prevent the discovery of association rules.
SIGMOD Record, 30(4), 45-54.
Schwartz, J., & Micheline, M. (2004, May 1). Airlines gave
F.B.I. millions of records on travelers after 9/11. New York
Times.
Sweeney, L. (2003). k-anonymity: A model for protecting
privacy. International Journal on Uncertainty, Fuzziness
and Knowledge-Based Systems, 10(5), 557-570.
Verykios, V.S., Elmagarmid, A., Bertino, E., Saygin, Y., &
Dasseni, E. (2004). Association rule hiding. IEEE Trans-
actions on Knowledge and Data Engineering, 16(4),
434-447.
KEY TERMS
Data Perturbation: Modifying the data so that origi-
nal confidential data values cannot be recovered.
Distributed Data Mining: Performing the data-mining
task on data sources distributed in different sites.
K-Anonymity: A privacy metric, which ensures an
individuals information cannot be distinguished from at
least k-1 other people when a data source is disclosed.
Privacy Against Data Mining: Preserving the privacy
of individuals against data-mining tools when disclosed
data contain private information that could be extracted
by data-mining tools.
Privacy Preserving Data Mining: Performing the data-
mining task on private data sources (centralized or distributed).
Secure Multiparty Computation: Computing the re-
sult of an operation (i.e., sum, min, max) on private data
(e.g., finding the richest person among a group of people
without revealing the wealth of the individuals).
Statistical Database: Type of database system that is
designed to support statistical operations while prevent-
ing operations that could lead to the association of
individuals with confidential data.
TEAM LinG

Privacy Protection in Association Rule Mining
Neha Jha
Indian Institute of Technology, Kharagpur, India
Shamik Sural
Indian Institute of Technology, Kharagpur, India
INTRODUCTION
Data mining technology has emerged as a means for
identifying patterns and trends from large sets of data.
Mining encompasses various algorithms, such as discov-
ery of association rules, clustering, classification and
prediction. While classification and prediction techniques
are used to extract models describing important data
classes or to predict future data trends, clustering is the
process of grouping a set of physical or abstract objects
into classes of similar objects.
Alternatively, association rule mining searches for
interesting relationships among items in a given data set.
The discovery of association rules from a huge volume of
data is useful in selective marketing, decision analysis
and business management. A popular area of application
is the market basket analysis, which studies the buying
habits of customers by searching for sets of items that are
frequently purchased together (Han & Kamber, 2003).
Let I = {i1, i 2, , in} be a set of items and D be a set of
transactions, where each transaction T belonging to D is
an itemset such that T is a subset of I. A transaction T
contains an itemset A if A is a subset of T. An itemset A
with k items is called a k-itemset. An association rule is an
implication of the form A => B where A and B are subsets
of I and A B is null. The rule A => B is said to have a
support s in the database D if s% of the transactions in D
contain AUB. The rule is said to have a confidence c if c%
of the transactions in D that contain A also contain B. The
problem of mining association rules is to find all rules
whose support and confidence are higher than a specified
minimum support and confidence. An example associa-
tion rule could be: Driving at night and speed > 90 mph
results in a road accident, where at least 70% of drivers
meet the three criteria and at least 80% of those meeting
the driving time and speed criteria actually have an acci-
dent.
While data mining has its roots in the traditional fields
of machine learning and statistics, the sheer volume of
data today poses a serious problem. Traditional methods
typically make the assumption that the data is memory
resident. This assumption is no longer tenable and imple-
mentation of data mining ideas in high-performance par-
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
925
P
allel and distributed computing environments has be-
come crucial for successful business operations. The
problem is not simply that the data is distributed, but that
it must remain distributed. For instance, transmitting large
quantities of data to a central site may not be feasible or
it may be easier to combine results than to combine the
sources. This has led to the development of a number of
distributed data mining algorithms (Ashrafi et al., 2002).
However, most of the distributed algorithms were initially
developed from the point of view of efficiency, not secu-
rity. Privacy concerns arise when organizations are will-
ing to share data mining results but not the data. The data
could be distributed among several custodians, none of
who are allowed to transfer their data to another site. The
transaction information may also be specific to individual
users who do not appreciate the disclosure of individual
record values. Consider another situation. A multina-
tional company wants to mine its own data spread over
many countries for globally valid results while national
laws prevent trans-border data sharing.
Thus, a need for developing privacy preserving dis-
tributed data mining algorithms has been felt in recent
years. This keeps the information about each site secure,
and at the same time determines globally valid results for
taking effective business decisions. There are many vari-
ants of these algorithms depending on how the data is
distributed, what type of data mining we wish to do, and
what restrictions are placed on sharing of information.
This paper gives an overview of the different approaches
to privacy protection and information security in distrib-
uted association rule mining. The following two sections
describe the traditional privacy preserving methods along
with the pioneering work done in recent years. Finally, we
discuss the open research issues and future directions of
work.
BACKGROUND
A direct approach to data mining over multiple sources
would be to run existing data mining tools at each site
independently and then combine the final results. While
it is simple to implement, this approach often fails to give
TEAM LinG

globally valid results since a rule that is valid in one or
more of the individual locations need not be valid over the
entire data set.
Efforts have been made to develop methods that
perform local operations at each site to produce interme-
diate results, which can then be used to obtain the final
result in a secure manner. For example, it can be easily
shown that if a rule has support > m% globally, it must
have support > m% on at least one of the individual sites.
This result can be applied to the distributed case with
horizontally partitioned data (all sites have the same
schema but each site has information on different enti-
ties). A distributed algorithm for this would work by
requesting each site to send all rules with support at least
m. For each rule returned, the sites are then asked to send
the count of their items that support the rule, and the total
count of all items at the site. Using these values, the global
support of each rule can be computed with the assurance
that all rules with support at least m have been found.
This method provides a certain level of information
security since the basic data is not shared. However, the
problem becomes more difficult if we want to protect not
only the individual items at each site, but also how much
each site supports a given rule. The above method reveals
this information, which may be considered to be a breach
of security depending on the sensitivity of any given
application. Theoretical studies in the field of secure
computation started in the late 1980s. In recent years, the
focus has shifted more to the application field (Maurer,
2003). The challenge is to apply the available theoretical
results in solving intricate real-world problems. Du &
Atallah (2001) review and suggest a number of open
secure computation problems including applications in
the field of computational geometry, statistical analysis
and data mining. They also suggest a method for solving
secure multi-party computational geometry problems and
secure computation of dot products in separate pieces of
work (Atallah & Du, 2001; Ioannidis et al., 2002).
In all the above-mentioned work, the secure computa-
tion problem has been treated with an approach to provid-
ing absolute zero knowledge whereas the corporations
may not always be willing to bear the cost of zero informa-
tion leakage as long as they can keep the information
shared within known bounds. In the next section, we
discuss some of the important approaches to privacy
preserving data mining with an emphasis on the algo-
rithms developed for association rule mining.
MAIN THRUST
In this section, we describe the various levels of privacy
protection possible while mining data and the corre-
sponding algorithms to achieve the same. Identification
926
Privacy Protection in Association Rule Mining
of privacy concerns in data mining has led to a wide range
of proposals in the past few years. The solutions can be
broadly categorized as those belonging to the classes of
data obfuscation, data summarization and data separa-
tion. The goal of data obfuscation is to hide the data to be
protected. This is achieved by perturbing the data before
delivering it to the data miner by either randomly modify-
ing the data, swapping the values between the records or
performing controlled modification of data to hide the
secrets. Cryptographic techniques are often employed to
encrypt the source data, perform intermediate operations
on the encrypted data and then decrypt the values to get
back the final result with each site not knowing anything
but the global rule. Summarization, on the other hand,
attempts to make available innocuous summaries of the
data and therefore only the needed facts are exposed.
Data separation ensures that only the trusted parties can
see the data by making all operations and analysis to be
performed either by the owner/creator of the data or by
trusted third parties.
One application of data perturbation technique is
decision tree based classification to protect individual
privacy by adding random values from a normal/Gaussian
distribution of mean 0 to the actual data values (Agrawal
& Srikant, 2000). Bayes rule for density functions is then
used to reconstruct the distribution. The approach is
quite elegant since it provides a method for approximating
the original data distribution and not the original data
values by using the distorted data and information on the
random data distribution. Similar data perturbation tech-
niques can be applied to the mining of Boolean associa-
tion rules also (Rizvi & Haritsa, 2002). It is assumed that
the tuples in the database are fixed length sequences of
0s and 1s. A typical example is the market basket appli-
cation where the columns represent the items sold by a
supermarket, and each row describes, through a sequence
of 1s and 0s, the purchases made by a particular cus-
tomer (1 indicates a purchase and 0 indicates no pur-
chase). One interesting feature of this work is a flexible
definition of privacy; for example, the ability to correctly
guess a value of 1 from the perturbed data can be
considered a greater threat to privacy than correctly
learning a 0.
For many applications such as market basket, it is
reasonable to expect that the customers would want more
privacy for the items they buy compared to the items they
do not. There are primarily two ways of handling this
requirement. In one method, the data is changed or per-
turbed to a certain extent to hide the exact information that
can be extracted from the original data. In another ap-
proach, data is encrypted before running the data mining
algorithms on it. While data perturbation techniques
usually result in a transformation that leads to loss of
information and the exact result cannot be determined,
TEAM LinG
Privacy Protection in Association Rule Mining
cryptographic protocols try to achieve zero knowledge
transfer using lossless transformation. Cryptographic tech-
niques have been developed for the ID3 classification
algorithm with two parties having horizontally partitioned
data (Lindell & Pinkas, 2000). While the approach is inter-
esting, it is not very efficient in mining rules from very large
databases. Also, completely secure multi-party computa-
tion may not actually be required in practical applications.
Although there may be many different data mining
techniques, they often perform similar computations at
various stages. For example, counting the number of items
in a subset of the data shows up in both association rule
mining and learning decision trees. There exist four popu-
lar methods for privacy-preserving computations that can
be used to support different forms of data mining. These
are: secure sum, set union, set intersection size and scalar
product. Though all methods are not truly secure (in some,
information other than the results is revealed), but they do
have provable bounds on the information released. In
addition, they are efficient as the communication and
computation cost is not significantly increased through
addition of the privacy-preserving component.
At this stage, it is important to understand that not all
the association rules have equal need for protection. One
has to analyze the sensitivity of the various association
rules mined from a database (Saygin et al., 2002). From the
large amount of data made available to the public through
various sources, a malicious user may be able to extract
association rules that were meant to be protected. Reduc-
ing the support or the confidence can hide the sensitive
rules. Evfimievski et al. (2002) investigate the problem of
applying randomization techniques in association rule
mining. One of their most important contributions is the
application of privacy preserving techniques in rule min-
ing from categorical data. They also provide a formal
definition for privacy breaches. It is quite challenging to
extend this work by combining randomization techniques
with cryptographic techniques to make the scheme more
robust.
Kantarcioglu & Clifton (2002) propose a cryptographic
technique for mining association rules in a horizontally
partitioned database. It assumes that the transactions are
distributed among n sites. The global support count of an
item set is the sum of all the local support counts. An
itemset A is globally supported if the global support count
of A is greater than s% of the total transaction database
size. The global confidence of a rule A => B can be given
as (AUB).support/(A).support and a k-itemset is called a
globally large k-itemset if it is globally supported. Quasi-
commutative hash functions are used for secure computa-
tion of set unions that determine globally frequent candi-
date itemsets from locally frequent candidate itemsets.
The globally frequent k-itemsets are determined from the
candidate itemsets with a secure protocol.
Since the goal is to determine if the support exceeds
a threshold rather than to learn the exact support of a rule, P
the secure sum computation is slightly altered and, in-
stead of sending the computed values to each other, the
sites perform secure comparison among themselves. If
the goal were to have a totally secure method, the union
step would have to be eliminated. However, using the
secure union method gives higher efficiency with prov-
ably controlled disclosure of some minor information
(e.g., the number of duplicate items and the candidate
sets). The validity of even this disclosed information can
be reduced by noise addition as each site can add some
fake large itemsets to its actual locally large itemsets. In
the pruning phase, the fake items can be eliminated.
In contrast to the above method, computing associa-
tion rules in vertically partitioned data is even more
challenging (Vaidya & Clifton, 2002). Here the items are
partitioned and each itemset is split between sites. Most
steps of the traditional a priori algorithm can be done
locally at each of the sites (Agarwal & Srikant, 1994). The
crucial step involves finding the support count of an
itemset. If the support count of an itemset is securely
computed, it can be checked if the support is greater than
the threshold to determine whether the itemset is fre-
quent. Consider the entire transaction database to be a
Boolean matrix where 1 represents the presence of an item
(column) in a transaction (row), while 0 correspondingly
represents an absence. Then the support count of an
itemset is the scalar product of the vectors representing
the sub-itemsets with both parties. An algorithm to
compute the scalar product securely is sufficient for
secure computation of the support count.
Most of the above protocols assume a semi-honest
model, where the parties involved will honestly follow
the protocol but can later try to infer additional informa-
tion from whatever data they receive through the proto-
col. One result of this is that parties are not allowed to
give spurious input to the protocol. If a party is allowed
to give spurious input, they can probe to determine the
value of a specific item at other parties. For example, if a
party gives the input (0, ..., 0, 1, 0, , 0), the result of the
scalar product (1 or 0) tells the malicious party if the other
party has the transaction corresponding to the 1. Attacks
of this type can be termed as probing attacks. All of the
protocols currently suggested in the literature are sus-
ceptible to such probing attacks. Better techniques,
which work even in the malicious model, are needed to
guard against this.
FUTURE DIRECTIONS
The ongoing work in the field has suggested several
interesting directions for future research. Existing ap-
927
TEAM LinG

proaches should be extended to cover new classes of
privacy breaches and to ascertain the theoretical limits on
discoverability for a given level of privacy (Agrawal &
Aggarwal, 2001). Potential research needs to concentrate
on combining the randomization and cryptographic pro-
tocols to get the strengths of both without the weak-
nesses of either. Privacy estimation formulas used in data
perturbation techniques can be refined so as to include
the effects of using the mining output to re-interrogate the
distorted database. Extending association rule mining of
vertically partitioned data to multiple parties is an impor-
tant research topic in itself, especially considering collu-
sion between the parties.
Much of the current work on vertically partitioned
data is limited to Boolean association rule mining. Cat-
egorical attributes and quantitative association rule min-
ing are significantly more complex problems. Most impor-
tantly, a wide range of data mining algorithms exists such
as classification, clustering and sequence detection, and
the effect of privacy constraints on these algorithms is
also an interesting area of future research (Verykios et al.,
2004). The final goal of researchers is to develop methods
enabling any data mining operation that can be done at a
single site to be done across various sources, while
respecting the privacy policies of the sources.
CONCLUSION
Several algorithms have been proposed to address the
conflicting goals of supporting privacy and accuracy
while mining association rules on large databases. Once
this field of research is matured, a toolkit of privacy-
preserving distributed computation techniques needs to
be built that can be assembled to solve specific real-world
problems (Clifton et al., 2003). Current techniques address
the problem of performing one secure computation and
using that result to perform the next computation reveals
intermediate information that may not be part of the final
results. Controlled disclosure is guaranteed by evaluat-
ing whether the real results together with the extra infor-
mation violate privacy constraints. This approach how-
ever, becomes more difficult with iterative techniques as
intermediate results from several iterations may reveal a
lot of information. Proving that this does not violate
privacy is a difficult problem.
REFERENCES
Agrawal, D., & Aggarwal, C.C. (2001). On the design and
quantification of privacy preserving data mining algo-
rithms. In Twentieth ACM SIGACT-SIGMOD-SIGART
928
Privacy Protection in Association Rule Mining
Symposium on Principles of Database Systems (pp. 247-
255). Santa Barbara, California.
Agrawal, R., & Srikant, R. (2000). Privacy-preserving data
mining. In ACM SIGMOD Conference on Management of
Data (pp. 439-450). Dallas, TX, USA.
Ashrafi, M.Z., Taniar, D., & Smith, K.A. (2002). A data
mining architecture for distributed environments. Inno-
vative Internet Computing Systems, 27-38.
Atallah, M.J., & Du, W. (2001). Secure multi-party compu-
tational geometry. In Seventh International Workshop on
Algorithms and Data Structures (pp. 165-179). Provi-
dence, Rhode Island, USA.
Clifton, C., Kantarcioglu, M., Vaidya, J., Lin, X., & Zhu, M.
(2003). Tools for privacy preserving distributed data
mining. ACM SIGKDD Explorations, 4(2), 28-34.
Du, W., & Atallah, M.J. (2001). Secure multi-party compu-
tation problems and their applications: A review and open
problems. In New Security Paradigms Workshop (pp. 11-
20). Cloudcroft, New Mexico, USA.
Evfimievski, A., Srikant, R., Agrawal, R., & Gehrke, G.
(2002). Privacy preserving mining of association rules. In
ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (pp. 217-228). Edmonton,
Canada28.
Han, J., & Kamber, M. (2003). Data mining: Concepts and
techniques. San Francisco: Morgan Kaufmann Publish-
ers.
Ioannidis, I., Grama, A., & Atallah, M. (2002). A secure
protocol for computing dot-products in clustered and
distributed environments. In International Conference
on Parallel Processing (pp. 279-285).
Kantarcioglu, M., & Clifton, C. (2002). Privacy-preserving
distributed mining of association rules on horizontally
partitioned data. In ACM SIGMOD Workshop on Re-
search Issues on Data Mining and Knowledge Discov-
ery.
Lindell, Y., & Pinkas, B. (2000). Privacy preserving data
mining. Advances in Cryptology (pp. 36-54).
Maurer, U. (2003). Secure multi-party computation made
simple. In Third Conference on Security in Communica-
tion Networks (SCN02) (pp. 14-28). Lecture Notes in
Computer Science (Vol. 2576). Berlin: Springer-Verlag.
Rizvi, S.J., & Haritsa, J.R. (2002). Maintaining data privacy
in association rule mining. In Twenty-Eighth Interna-
tional Conference on Very Large Data Bases (pp. 682-
689).
TEAM LinG
Privacy Protection in Association Rule Mining
Saygin, Y., Verykios, V.S., & Elmagarmid, A.K. (2002).
Privacy preserving association rule mining. In Twelfth
International Workshop on Research Issues in Data
Engineering (pp. 151-158).
Vaidya, J., & Clifton, C. (2002). Privacy preserving asso-
ciation rule mining in vertically partitioned data. In Eighth
ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (pp. 639-644). Edmonton,
Alberta, Canada.
Verykios, V.S., Bertino, E., Fovino, I.N., Provenza, L.P.,
Saygin, Y., & Theodoridis. (2004). State-of-the-art in pri-
vacy preserving data mining. ACM SIGMOD Record, 33
(1), 50-57.
KEY TERMS
Association Rule: A relation between the occurrences
of a set of items with another set of items in a large data
set.
Cryptographic Data Mining Techniques: Methods
that encrypt individual data before running data mining
algorithms so that the final result is also available in an
encrypted form.
Distributed Data Mining: Mining information from a
very large set of data spread across multiple locations P
without transferring the data to a central location.
Horizontally Partitioned Data: A distributed archi-
tecture in which the all the sites share the same database
schema but have information about different entities. The
union of all the rows across all the sites forms the complete
database.
Itemset: A set of one or more items that are purchased
together in a single transaction.
Privacy Preserving Data Mining: Algorithms and
methods that are used to mine rules from a distributed set
of data in which the sites reveal detailed information as
less as possible.
Secure Multi-Party Computation: Computation of an
overall result based on the data from a number of users in
which only the final result is known to the individual user
at the end of computation and nothing else.
Vertically Partitioned Data: A distributed architec-
ture in which the different sites store different attributes
of the data. The union of all these attributes or columns
together forms the complete database.
929
TEAM LinG
930
Profit Mining
Senqiang Zhou
Simon Fraser University, Canada
Ke Wang
Simon Fraser University, Canada
INTRODUCTION
A major obstacle in data mining applications is the gap
between the statistic-based pattern extraction and the
value-based decision-making. Profit mining aims to
reduce this gap. In profit mining, given a set of past
transactions and pre-determined target items, we like to
build a model for recommending target items and pro-
motion strategies to new customers, with the goal of
maximizing profit. Though this problem is studied in the
context of retailing environment, the concept and tech-
niques are applicable to other applications under a gen-
eral notion of utility. In this short article, we review
existing techniques and briefly describe the profit mining
approach recently proposed by the authors. The reader is
referred to (Wang, Zhou & Han, 2002) for the details.
BACKGROUND
It is a very complicated issue whether a customer buys
a recommended item. Consideration includes items
stocked, prices or promotions, competitors offers,
recommendation by friends or customers, psychologi-
cal issues, conveniences, etc. For on-line retailing, it
also depends on security consideration. It is unrealistic
to model all such factors in a single system. In this
article, we focus on one type of information available in
most retailing applications, namely past transactions.
The belief is that shopping behaviors in the past may
shed some light on what customers like. We try to use
patterns of such behaviors to recommend items and
prices.
Consider an on-line store that is promoting a set of
target items. At the cashier counter, the store likes to
recommend one target and a promotion strategy (such
as a price) to the customer based on non-target items
purchased. The challenge is determining an item inter-
esting to the customer at a price affordable to the
customer and profitable to the store. We call this prob-
lem profit mining (Wang, Zhou & Han, 2002).
Most statistics-based rule mining, such as associa-
tion rules (Agrawal, Imilienski & Swami, 1993; Agrawal
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of IGI is prohibited.
& Srikant, 1994), considers a rule as interesting if it
passes certain statistical tests such as support/confi-
dence. To an enterprise, however, it remains unclear
how such rules can be used to maximize a given business
object. For example, knowing Perfume Lipstick
and Perfume Diamond, a store manager still cannot
tell which of Lipstick and Diamond, and what price
should be recommended to a customer who buys Per-
fume. Simply recommending the most profitable item,
say Diamond, or the most likely item, say Lipstick, does
not maximize the profit because there is often an in-
verse correlation between the likelihood to buy and the
dollar amount to spend. This inverse correlation re-
flects the general trend that the more dollar amount is
involved, the more cautious the buyer is when making a
purchase decision.
MAIN THRUST OF THE CHAPTER
Related Work
Profit maximization is different from the hit maximi-
zation as in classic classification because each hit may
generate different profit. Several approaches existed to
make classification cost-sensitive. (Domingos, 1999)
proposed a general method that can serve as a wrapper to
make a traditional classifies cost-sensitive. (Zadrozny
& Elkan, 2001) extended the error metric by allowing
the cost to be example dependent. (Margineantu &
Dietterich, 2000) gave two bootstrap methods to esti-
mate the average cost of a classifier. (Pednault, Abe &
Zadrozny, 2002) introduced a method to make sequential
cost-sensitive decisions, and the goal is to maximize the
total benefit over a period of time. These approaches
assume a given error metric for each type of
misclassification, which is not available in profit mining.
Profit mining is related in motivation to action-
ability (or utility) of patterns: a pattern is interesting in
the sense that the user can act upon it to her advantage
(Silberschatz & Tuzhilin, 1996). (Kleinberg,
Papadimitriou & Raghavan, 1998) gave a framework for
evaluating data mining operations in terms of utility in
TEAM LinG
Profit Mining
decision-making. These works, however, did not pro-
pose concrete solutions to the actionability problem.
Recently, there were several works applying associa-
tion rules to address business related problems. (Brijs,
Swinnen, Avanhoof & Wets, 1999; Wong, Fu & Wang,
2003; Wang & Su, 2002) studied the problem of select-
ing a given number of items for stocking. The goal is to
maximize the profit generated by selected items or
customers. These works present one important step
beyond association rue mining, i.e., addressing the issue
of converting a set of individual rules into a single
actionable model for recommending actions in a given
scenario.
There were several attempts to generalize associa-
tion rules to capture more semantics, e.g., (Lin, Yao &
Louie, 2002; Yao, Hamilton & Butz, 2004; Chan, Yang
& Shen, 2003). Instead of a uniform weight associated
with each occurrence of an item, these works associate
a general weight with an item and mine all itemsets that
pass some threshold on the aggregated weight of items
in an itemset. Like association rule mining, these works
did not address the issue of converting a set of rules or
itemsets into a model for recommending actions.
Collaborative filtering (Resnick & Varian, 1997)
makes recommendation by aggregating the opinions
(such as rating about movies) of several advisors who
share the taste with the customer. Built on this technol-
ogy, many large commerce web sites help their custom-
ers to find products. For example, Amazon.com uses
Book Matcher to recommend books to customers;
Moviefinder.com recommends movies to customers
using We Predict recommender system. For more
examples, please refer to (Schafer, Konstan & Riedl,
1999). The goal is to maximize the hit rate of recom-
mendation. For items of varied profit, maximizing profit
is quite different from maximizing hit rate. Also, col-
laborative filtering relies on carefully selected item
endorsements for similarity computation, and a good
set of advisors to offer opinions. Such data are not
easy to obtain. The ability of recommending prices, in
addition to items, is another major difference between
profit mining and other recommender systems.
Another application where data mining is heavily
used for business targets is direct marketing. See (Ling
& Li, 1998; Masand & Shapiro, 1996; Wang, Zhou,
Yeung & Yang, 2002), for example. The problem is to
identify buyers using data collected from previous cam-
paigns, where the product to be promoted is usually
fixed and the best guess is about who are likely to buy.
The profit mining, on the other hand, is to guess the best
item and price for a given customer. Interestingly, these
two problems are closely related to each other. We can
model the direct marketing problem as profit mining
problem by including customer demographic data as
part of her transactions and including a special target
item NULL representing no recommendation. Now, P
each recommendation of a non-NULL item (and price)
corresponds to identifying a buyer of the item. This
modeling is more general than the traditional direct
marketing in that it can identify buyers for more than
one type of item and promotion strategies.
Profit Mining
We solve the profit mining by extracting patterns from
a set of past transactions. A transaction consists of a
collection of sales of the form (item, price). A simple
price can be substituted by a promotion strategy, such
as buy one get one free or X quantity for Y dollars,
that provides sufficient information for derive the price.
The transactions were collected over some period of
times and there could be several prices even for the
same item if sales occurred at different times. Given a
collection of transactions, we find recommendation
rules of the form {s 1, , sk}<I,P>, where I is a target
item and P is a price of I, and each si is a pair of non-target
item and price. An example is (Perfume,
price=$20) (Lipstick, price=$10). This recommen-
dation rule can be used to recommend Lipstick at the
price of $10 to a customer who bought Perfume at the
price of $20. If the recommendation leads to a sale of
Lipstick of quantity Q, it generates (10-C)*Q profit,
where C is the cost of Lipstick.
Several practical considerations would make rec-
ommendation rules more useful. First, items on the
left-hand side in si can be item categories instead to
capture category-related patterns. Second, a customer
may have paid a higher price if a lower price was not
available at the shopping time. We can incorporate the
domain knowledge that paying a higher price implies the
willingness of paying a lower price (for exactly the
same item) to search for stronger rules at lower prices.
This can be done through multi-level association mining
(Srikant and Agrawal, 1995; Han and Fu, 1995), by
modeling a lower price as a more general category than
a higher price. For example, the sale {<chicken, $3.8>}
in a transaction would match any of the following more
general sales in a rule: <chicken, $3.8>, <chicken,
$3.5>, <chicken, $3.0>, chicken, meat, food. Note that
the last three sales are generalized by climbing up the
category hierarchy and dropping the price.
A key issue is how to make a set of individual rules
work as a single recommender. Our approach is ranking
rules the recommendation profit. The recommendation
profit of a rule r is defined as the average profit of the
target item in r among all transactions that match r. Note
that the rank by average profit implicitly takes into
account of both confidence and profit because a high
931
TEAM LinG

average profit implies that both confidence and profit are
high. Given a new customer, we pick up the highest
ranked matching rule to make recommendation.
Before making recommendation, however, over-
fitting rules that work only for observed transactions,
but not for new customers, should be pruned because our
goal is to maximize profit on new customers. The idea is
as follows. Instead of ranking rules by observed profit,
we rank rules by projected profit, which is based on the
estimated error of a rule adapted for pruning classifiers
(Quinlan, 1993). Intuitively, the estimated error will
increase for a rule that matches a small number of
transactions. Therefore, over-fitting rules tend to have a
larger estimated error, which translates into a lower
projected profit, and a lower rank.
For a detailed exposure and experiments on real life
and synthetic data sets, the reader is referred to (Wang,
Zhou & Han, 2002).
FUTURE TRENDS
The profit mining proposed is only the first, but impor-
tant, step in addressing the ultimate goal of data mining.
To make profit mining more practical, several issues
need further study. First, it is quite likely that the recom-
mended item tends to be the item that the customer will
buy independently of the recommendation. Obviously,
such items need not be recommended, and recommenda-
tion should focus on those items that the customer may
buy if informed, but may not otherwise. Recommending
such items likely brings in additional profit. Second, the
current model maximizes only the profit of one-shot
selling effect, therefore, a sale in a large quantity is
favored. In reality, a customer may regularly shop the
same store over a period of time, in which case a sale in
a large quantity will affect the shopping frequency of a
customer, thus, profit. In this case, the goal is maximiz-
ing the profit for reoccurring customers over a period of
time. Another interesting direction is to incorporate the
feedback whether a certain recommendation is rejected
and accepted to improve future recommendations.
This current work has focused on the information
captured in past transactions. As pointed out in Introduc-
tion, other things such as competitors offers, recom-
mendation by friends or customers, consumer fashion,
psychological issues, conveniences, etc. can affect the
customers decision. Addressing these issues requires
additional knowledge, such as competitors offers, and
computers may not be the most suitable tool. One solu-
tion could be suggesting several best recommendations
to the domain expert, the store manager or sales person
in this case, who makes the final recommendation to the
customer after factoring the other considerations.
932
Profit MIning
CONCLUSION
Profit mining is a promising data mining approach
because it addresses the ultimate goal of data mining. In
this article, we study profit mining in the context of
retailing business, but the principles and techniques
illustrated should be applicable to other applications.
For example, items can be general actions and prices
can be a notion of utility resulted from actions. In
addition, items can be used to model customer demo-
graphic information such as Gender, in which case the
price component is unused.
REFERENCES
Agrawal, R., Imilienski, T., & Swami, A. (1993, May).
Mining association rules between sets of items in large
databases. ACM Special Interest Group on Manage-
ment of Data (SIGMOD) (pp. 207-216), Washington D.C.,
USA.
Agrawal, R. & Srikant, R. (1994, September). Fast algo-
rithms for mining association rules. International Con-
ference on Very Large Data Bases (VLDB) (pp. 487-499),
Santiago de Chile.
Brijs, T., Swinnen, G., Vanhoof, K., & Wets, G. (1999,
August). Using association rules for product assortment
decisions: A case study. International Conference on
Knowledge Discovery and Data Mining (KDD) (pp. 254-
260), San Diego, USA.
Chan, R., Yang, Q., & Shen, Y. (2003, November). Mining
high utility itemsets. IEEE International Conference on
Data Mining (ICDM) (pp. 19-26), Melbourne, USA.
Domingos, P. (1999, August). MetaCost: A general
method for making classifiers cost-sensitive. ACM SIG
International Conference on Knowledge Discovery and
Data Mining (SIGKDD) (pp. 155-164), San Diego, USA.
Han, J., & Fu, Y. (1995, September). Discovery of mul-
tiple-level association rules from large databases. Inter-
national Conference on Very Large Data Bases (VLDB)
(pp. 420-431), Zurich, Switzerland.
Kleinberg, J., Papadimitriou, C. & Raghavan, P. (1998,
December). A microeconomic view of data mining.
Data Mining and Knowledge Discovery Journal, 2(4),
311-324.
Lin, T. Y., Yao, Y.Y., & Louie, E. (2002, May). Value
added association rules. Advances in Knowledge Dis-
covery and Data Mining, 6th Pacific-Asia Conference
PAKDD (pp. 328-333), Taipei, Taiwan.
TEAM LinG
Profit Mining
Ling, C., & Li, C. (1998, August) Data mining for direct
marketing: problems and solutions. ACM SIG Interna-
tional Conference on Knowledge Discovery and Data
Mining (SIGKDD) (pp. 73-79), New York, USA.
Margineantu, D. D., & Dietterich, G. T. (2000, June-July).
Bootstrap methods for the cost-sensitive evaluation of
classifiers. International Conference on Machine Learn-
ing (ICML) (pp. 583-590), San Francisco, USA.
Masand, B., & Shapiro, G. P. (1996, August) A comparison
of approaches for maximizing business payoff of predic-
tion models. ACM SIG International Conference on
Knowledge Discovery and Data Mining (SIGKDD) (pp.
195-201), Portland, USA.
Pednault, E., Abe, N., & Zadrozny, B. (2002, July). Sequen-
tial cost-sensitive decision making with reinforcement
learning. ACM SIG International Conference on Knowl-
edge Discovery and Data Mining (SIGKDD) (p. 259-268),
Edmonton, Canada.
Quinlan, J.R. (1993). C4.5: Programs for Machine Learn-
ing. Morgan Kaufmann.
Resnick, P., & Varian, H.R. (1997). CACM special issue on
recommender systems. Communications of the ACM,
40(3), 56-58.
Schafer, J. B., Konstan, J. A., & Riedl, J. (1999, November).
Recommender systems in E-commerce. ACM Conference
on Electronic Commerce (pp. 158-166), Denver, USA.
Silberschatz, A., & Tuzhilin, A. (1996). What makes pat-
terns interesting in knowledge discovery systems. IEEE
Transactions on Knowledge and Data Engineering, 8(6),
970-974.
Srikant, R., & Agrawal, R. (1995, September). Mining
generalized association rules. International Conference
on Very Large Data Bases (VLDB) (pp. 407-419), Zurich,
Switzerland.
Wang, K., & Su, M. Y. (2002, July). Item selection by hub-
authority profit ranking. ACM SIG International Confer-
ence on Knowledge Discovery and Data Mining
(SIGKDD) (pp. 652-657), Edmonton, Canada.
Wang, K., Zhou, S., & Han, J. (2002, March). Profit mining:
From patterns to actions. International Conference on
Extending Database Technology (EDBT) (pp. 70-87),
Prague, Czech Republic.
Wang, K., Zhou, S., Yeung, J. M. S., & Yang, Q. (2003,
March). Mining customer value: From association rules
to direct marketing. International Conference on Data
Engineering (ICDE) (pp. 738-740), Bangalore, India.
Wong, R. C. W., Fu, A. W. C., & Wang, K. (2003, Novem-
ber). MPIS: Maximal-profit item selection with cross- P
selling considerations. IEEE International Conference
on Data Mining (ICDM) (pp. 371-378), Melbourne, USA.
Yao, H., Hamilton, H. J., & Butz, C. J. (2004, April). A
foundational approach for mining itemset utilities from
databases. SIAM International Conference on Data
Mining (SIAMDM) (pp. 482-486), Florida, USA.
Zadrozny, B., & Elkan, C. (2001, August). Learning and
making decisions when costs and probabilities are both
unknown. ACM SIG International Conference on
Knowledge Discovery and Data Mining (SIGKDD) (pp.
204-213), San Francisco, USA.
KEY TERMS
Association Rule: An association has the form I1 I2,
where I1 and I2 are two itemsets. The support of an
association rule is the support of the itemset I1 I2, and
the confidence of a rule is the ratio of support of I1 I2
and the support of I1.
Classification: Given a set of training examples in
which each example is labeled by a class, build a model,
called a classifier, to predict the class label of new
examples that follow the same class distribution as
training examples. A classifier is accurate if the pre-
dicted class label is the same as the actual class label.
Cost Sensitive Classification: The error of a
misclassification depends on the type of the
misclassification. For example, the error of
misclassifying Class 1 as Class 2 may not be the same as
the error of misclassifying Class 1 as Class 3.
Frequent Itemset: The support of an itemset refers
to as the percentage of transactions that contain all the
items in the itemset. A frequent itemset is an itemset
with support above a pre-specified threshold.
Over-fitting Rule: A rule has high performance (e.g.
high classification accuracy) on observed transaction(s)
but performs poorly on future transaction(s). Hence, such
rules should be excluded from the decision-making sys-
tems (e.g. recommender). In many cases over-fitting rules
are generated due to the noise in data set.
Profit Mining: In a general sense, profit mining refers
to data mining aimed at maximizing a given objective
function over decision making for a targeted population
(Wang, Zhou & Han, 2002). Finding a set of rules that pass
a given threshold on some interestingness measure (such
as association rule mining or its variation) is not profit
933
TEAM LinG

mining because of the lack of a specific objective function
to be maximized. Classification is a special case of profit
mining where the objective function is the accuracy and
the targeted population consists of future cases. This
paper examines a specific problem of profit mining, i.e.,
934
Profit MIning
building a model for recommending target products and
prices with the objective of maximizing net profit.
Transaction: A transaction is some set of items cho-
sen from a fixed alphabet.
TEAM LinG

Pseudo Independent Models
Yang Xiang
University of Guelph, Canada
INTRODUCTION
Graphical models such as Bayesian networks (BNs) (Pearl,
1988) and decomposable Markov networks (DMNs)
(Xiang, Wong & Cercone, 1997) have been applied widely
to probabilistic reasoning in intelligent systems. Figure1
illustrates a BN and a DMN on a trivial uncertain domain:
A virus can damage computer files, and so can a power
glitch. A power glitch also causes a VCR to reset. The BN
in (a) has four nodes, corresponding to four binary vari-
ables taking values from {true, false}. The graph structure
encodes a set of dependence and independence assump-
tions (e.g., that f is directly dependent on v, and p but is
independent of r, once the value of p is known). Each node
is associated with a conditional probability distribution
conditioned on its parent nodes (e.g., P(f | v, p)). The joint
probability distribution is the product P(v, p, f, r) = P(f |
v, p) P(r | p) P(v) P(p). The DMN in (b) has two groups
of nodes that are maximally pair-wise connected, called
cliques. Each clique is associated with a probability
distribution (e.g., clique {v, p, f} is assigned P(v, p, f)). The
joint probability distribution is P(v, p, f, r) = P(v, p, f) P(r,
p) / P(p), where P(p) can be derived from one of the clique
distributions. The networks, for instance, can be used to
reason about whether there are viruses in the computer
system, after observations on f and r are made.
Construction of such networks by elicitation from
domain experts can be very time-consuming. Automatic
discovery (Neapolitan, 2004) by exhaustively testing all
possible network structures is intractable. Hence, heuris-
tic search must be used. This article examines a class of
graphical models that cannot be discovered using the
common heuristics.
Figure 1. (a) a trivial example BN; (b) a corresponding
DMN
Copyright 2006, Idea Group Inc., distributing in print or electronic forms without written permission of I
935
P
BACKGROUND
Let V be a set of n discrete variables x1, , xn (in what
follows, we will focus on finite, discrete variables). Each
variable xi has a finite space Si = {xi,1, x i,2, , xi,D } of
cardinality Di. When there is no confusion, we write xi,j as
xij for simplicity. The space of a set V of variables is defined
by the Cartesian product of the spaces of all variables in
V, that is, SV = S1 x ... x Sn (or S ). Thus, SV contains the
i i
tuples made of all possible combinations of values of the
variables in V. Each tuple is called a configuration of V,
denoted by v = (x 1, , xn).
Let P(xi ) denote the probability function over x i and
P(xij ) denote the probability value P(xi = x ij ). A probabi-
listic domain model (PDM) over V defines the probabil-
ity values of every configuration for every subset A V .
Let P(V) or P(x1, , xn) denote the joint probability
distribution (JPD) function over x1, , xn and P(x1 j1, ,
xn jn) denote the probability value of a configuration (x1 j1,
, xn jn). We refer to the function P(A) over A V as the
marginal distribution over A and P(xi) as the marginal
distribution of x i. We refer to P(x1 j1, , xn jn) as a joint
parameter and P(xij) as a marginal parameter of the
corresponding PDM over V.
For any three disjoint subsets of variables W, U and
Z in V, subsets W and U are called conditionally indepen-
dent given Z, if
P(W | U, Z) = P(W | Z)
for all possible values in W, U and Z such that P(U, Z) >
0. Conditional independence signifies the dependence
mediated by Z. This allows the dependence among
W U U U Z to be modeled over subsets W U Z and U U Z
separately. Conditional independence is the key property
explored through graphical models.
Subsets W and U are said to be marginally indepen-
dent (sometimes referred to as unconditionally indepen-
dent) if
P(W | U) = P(W)
for all possible values W and U such that P(U) > 0. When
two subsets of variables are marginally independent,
there is no dependence between them. Hence, each subset