Decision Support Systems 37 (2004) 151 – 173

www.elsevier.com/locate/dsw

Metadata management: past, present and future

Arun Sen *
Department of Information and Operations Management, Mays Business School, Texas A&M University, College Station, TX 77843, USA
Received 1 October 2002; accepted 4 December 2002

Abstract

In the past, metadata has always been a second-class citizen in the world of databases and data warehouses. Its main purpose
has been to define the data. However, the current emphasis on metadata in the data warehouse and software repository
communities has elevated it to a new prominence. The organization now needs metadata for tool integration, data integration
and change management. The paper presents a chronological account of this evolution—both from conceptual and management
perspectives.
Repository concepts are currently being used to manage metadata for tool integration and data integration. As a final chapter
in this evolution process, we point out the need of a concept called ‘‘metadata warehouse.’’ A real-life data warehouse project
called TAMUS Information Portal (TIP) is used to describe the types of metadata needed in a data warehouse and the changes
that the metadata go through. We propose that the metadata warehouse needs to be designed to store the metadata and manage
its changes. We propose several architectures that can be used to develop a metadata warehouse.
D 2003 Elsevier B.V. All rights reserved.

Keywords: Metadata; Data warehouse; Decision support; Metadata warehouse; Repository

1. Introduction length (a physical property) of a stick (an object) is 5

Metadata is a term that has been used and misused
many times in the past. Webster defines ‘‘meta’’ as a
more comprehensive term needed to ‘‘describe a new
and related discipline designed to deal critically with
the original one.’’ Metadata consequently then
describes a discipline that fosters the study of data
about data.
The origin of metadata can be traced back to how
we use measurement units. The purpose of a unit is to
describe a property of an object. For example, the
* Tel.: +1-979-845-8370; fax: +1-979-845-5653.
E-mail address: Asen@cgsb.tamu.edu (A. Sen).
ft (a measurement unit). This example uses for one
object, a data item (the number 5) and two metadata
items (length and a measuring unit).
In the past, the metadata has often been treated as
a second-class citizen. With the advent of computers
and our incessant need for data, we have introduced
techniques to store data permanently on a secondary
storage. These data can then be retrieved and used by
application programs. File managers are used to store
and retrieve data from the secondary storage. To
accomplish their job, file managers use such meta-
data as field names and filenames. This use of
metadata, along with the actual data, now has exten-
sively been ingrained in the database management
technology.

0167-9236/$ - see front matter D 2003 Elsevier B.V. All rights reserved.
doi:10.1016/S0167-9236(02)00208-7
152 A. Sen / Decision Support Systems 37 (2004) 151–173
As a result, in the last 30 years, we have wit-
nessed a tremendous growth in the use of metadata
in developing information systems. The purpose of
this paper is to study this field and see how it helps
us in decision support. To do this, we first describe a
40-year chronological development of metadata con-
cept (see Section 2). Several management tools were
also designed to manage metadata in the last 40
years. In Section 3, we categorize these develop-
ments. We argue that the most neglected area in the
metadata management is the notion of managing
changes in metadata. Section 4 emphasizes the
changes in metadata and describes a real-life case
study where the changes are of utmost importance.
To manage these changes, we propose a new man-
agement tool called metadata warehouse. Unlike
other tools that focus on tool integration and data
integration, this tool manages the changes in meta-
data for organizational decision support. We con-
clude the paper in Section 5.
2. Evolution of the metadata concept
To describe the evolution of the concept of meta-
data, we look at each decade starting with the 1960s.
2.1. The 1960s
Early work with files presumed that files were on
tapes. Access was sequential and the cost of access
grew in direct proportion to the size of the file. Simple
indexes were used to speed up the access. However, as
the indexes grew, they too became difficult to manage.
Due to this reason, in the early 1960s, the idea of
applying tree structures emerged as a potential solu-
tion. In the late 1960s, using the work of B trees
and B+ trees, many commercial vendors created file
systems that were faster and were not sequential. For
all file systems, the format of the records is first
determined. The field names of the records and their
data types are essentially metadata used in the file.
During this period, the use of metadata in program
development has also been fairly discrete. For exam-
ple, in attempting to reuse code, programming lan-
guages allow applications to include codes from their
software libraries. We discuss these libraries in
Section 3.
2.2. The 1970s
The 1970s could be called the decade that started
the metadata phenomenon. With the advent of data-
base management system (DBMS), the use of meta-
data increased tremendously. For example, in
relational DBMS, the metadata is extensively used
to define data. These metadata include relation names,
attribute names, key and domain information. The
collection of metadata is used to define schema and
subschemas. In 1976, with the introduction of entity
relationship (ER) data model and later with the advent
of other semantic data models, higher level metadata
were being used. As the semantic data models (like
ER) did not have a supporting DBMS, a translational
mechanism was deployed to translate the higher level
(like ER) metadata to relational metadata.
2.3. The 1980s
With the success of database management systems
to store and retrieve business data (which are essen-
tially flat), efforts were made to look into non-business
data types. These data types came from diverse appli-
cation areas such as Computer-Aided Design/Com-
puter-Aided Manufacturing (CAD/CAM), Computer-
Aided Software Engineering (CASE), Geographic
Information System (GIS), document storage and
retrieval, science and medicine. The notion of data,
at this time, got replaced by a term called asset. An
asset is a piece of useful item that is a product or by-
product of application development process. An asset
can be more tangible, like data (as before), designs and
software code, or more intangible, such as knowledge
and methodologies. Tangible asset can take a form of a
large-grained component like framework or a com-
plete application; it can also be a fine-grained compo-
nent like subroutine, a class, or an encapsulated
component. It can include patterns [6] and algorithms
[6]. Examples of intangible asset can include program-
ming knowledge, programming plans, software system
architectures, project plans, design documents, user
documents and other relevant knowledge sources.
Just like data, assets need to be stored on disks to
be reused. Assets are, however, much more compli-
cated than simple business data and are difficult to
store and access. Several new database paradigms
were proposed at this time that deal with assets and
 A. Sen / Decision Support Systems 37 (2004) 151–173
their metadata semantics. This includes Complex
Object Model [3], Nested Relation Data Model
[4,20] and Object-Oriented Data Model [1].
Accessing these assets was mainly done through
object queries. An asset is stored as an object or a
collection of objects. The queries use metadata that
are usually class definitions and class hierarchies.
Class definitions are very much like table definitions
with some exceptions. They include attribute defini-
tions along with method definitions for the class.
Three predominant relationships used in building
class hierarchies are aggregation [3], part – whole [3]
and generalization [3]. Aggregation relationship dealt
with relationships among classes that are related. For
example, course class is related to student class and
instructor class. Part – whole, on the other hand, dealt
with composition. For example, vehicle class is com-
posed of chassis class, engine class and others. Gen-
eralization (also known as ‘‘is-a’’) relationship was
used to show classification among classes. For exam-
ple, motor-vehicle and airplane classes are the sub-
classes of vehicle class. Generation relationship
allows the subclasses to inherit the properties of the
superclass.
As database management systems access data
through indexing, the query to access an asset even-
tually finds all parts of the asset from the database and
combines them to create the asset. However, it is
sometimes difficult to store an asset as a collection of
objects in a database. This is because the asset may be a
package (like a code package or a document) that
cannot be decomposed into objects. To help in this
process, metadata was again utilized. A classification
scheme, based on metadata, was proposed by Prieto-
Diaz [18]. The idea of the classification scheme was to
show relationships by collection, that is, to keep related
classes more or less together according to the closeness
of the relationship. He introduced the notion of faceted
classification scheme. A facet is an arranged group of
descriptors. For example, in the Unix domain, a facet
could be {by action} [18]. A facet takes on terms or
values. The {by action} facet can take terms like get,
put, update, append, check and others.
2.4. The 1990s
In the 1990s, we see three separate research para-
digms emerge that were responsible to move the
153
metadata technology forward. We describe them one
by one as follows.
2.4.1. Metadata in code reusability
The notion of code reusability in software develop-
ment, started in the 1980s with Japanese software
factories [13], became very important in the 1990s.
To facilitate the code reusability, research on soft-
ware development environment flourished. A software
development environment (SDE) is a collection of
software and hardware tools, which explicitly tailored
to support the production of software systems in a
particular application domain [22]. A SDE uses
metadata in all of its operations. The objective of
the use of metadata is to support the selection of the
tools used in the SDE. Many kinds of SDEs are
currently in practice. These environments are classi-
fied into three major groups [22]: Programming
Environments (PE), CASE and Software Engineering
Environments (SEE). A PE is an environment that is
principally intended to support the process of pro-
gramming, testing and debugging. A CASE is an
environment that supports software specification and
design. It can also be used with a programming
environment. A SEE is intended to support the
production of large, long-lifetime software systems,
the maintenance costs of which typically exceed
development costs and are produced by a team rather
than individual programmers. REuse Based on
Object-Oriented Techniques (REBOOT) by Morel
and Faget [14] is an example of Programming Envi-
ronment tool that supports storage and retrieval of
code components. It uses the facet-based classifica-
tion scheme to create the metadata of the components.
Four such facets are used: abstraction, operations,
operates-on and dependencies. In systems like this,
the metadata is stored in a database. If code library
gets updated, corresponding entries in the database
are changed.
2.4.2. Metadata in asset repository
According to Bernstein [2], an asset repository
(also called a repository) is a shared database of
information about engineered artifacts, such as soft-
ware, documents, maps and other things. In other
words, a repository is a metadata manager. For exam-
ple, a repository that supports software development
and deployment tools could store metadata such as
154 A. Sen / Decision Support Systems 37 (2004) 151–173
database descriptions, form definitions, controls,
documents, interface definitions, source code, help
text and others. The objective of the use of metadata
in a repository context is to emphasize the selection
and integration of diverse tools that support the differ-
ent kinds of data.
2.4.3. Metadata in data warehouse
Metadata took a significant role in the 1990s due to
the advent of data warehouse concept. Kimball et al.
(Ref. [8], p. 22) define metadata in the context of data
warehouse as ‘‘all of the information in the data
warehouse environment that is not the actual data
itself.’’ The metadata in the data warehouse context
are basically of two kinds: back room metadata and
front room metadata. The back room metadata is
process related and guides the extraction, cleaning
and loading processes. Examples include specifica-
Fig. 1. The time line for metadata management.
tions typically related to source data, such as source
schemas, old formats for archived mainframe data,
ownership description of the source, automated
extract tool settings and others; data staging metadata
such as data cleaning specifications, slowly changing
dimension policies and others; data transform logs;
and DBMS system table contents. The front room
metadata is more descriptive, and it helps query tools
and report-writers function smoothly. Examples
include join specifications, network security user
privilege profiles, usage and access maps, network
security usage statistics and others.
The need of asset repositories became crucial in the
late 1990s as software projects increasingly were
focussed on integrating and reusing codes, classes,
components, patterns, frameworks and applications.
The need is also fueled by the massive use of data
warehouse techniques in the brick and mortar industry
 A. Sen / Decision Support Systems 37 (2004) 151–173
along with the web world to develop decision support
using huge operational data.
2.5. 2000 and beyond
Fig. 1 illustrates the evolution of metadata concept
in a chronological order. Since the 1960s, the concept
of metadata has grown quite a bit. Starting from the
simple filenames, field names and field types, meta-
data in the 1970s described data definitions as mod-
eled by various data models. In the 1980s, with the
advent of object-oriented programming, the metadata
started to include class definitions and class hierar-
chies (aggregation and generalization). A special kind
of classification scheme called faceted classification
was introduced in the 1980s for classifying assets. In
the early 2000, however, a major effort in dealing with
metadata is in the recognition of the need for the
creation of a metadata standard. This standard was
fueled by the fact that unlike the software develop-
ment, a data warehouse project needs heterogeneous
tool and data environments. For example, in the data
warehouse world, data quality tools, data modeling
tools, ETL tools and end-user tools developed by
different vendors with entirely different specifications.
The integration using a metadata standard allows them
to communicate with each other. The data in a data
warehouse can also be of various types and format. A
standardization effort will also help in data integra-
tion.
3. Evolution of metadata management
Metadata, although started as information to
describe data or an asset, now is competing to get
equal attention as an asset or data it defines. With
phenomenal use of metadata in software development,
in database management and in data warehouse design
and implementation, it is important to look into the
techniques used to manage the metadata.
To properly narrate the evolution of metadata
managers, we start with a set of tools that implicitly
manage data with metadata. From this stage, we
describe how metadata management has progressively
moved away from implicit management to co-man-
agement with asset and data, and then into explicit
management with repository managers.
155
3.1. Stage I. Implicit metadata management
Even before the advent of database management
system, software library manager was an early attempt
to discretely use metadata to reuse codes. It is
designed to co-manage data and metadata. It has been
so successful that it still exists today. Most program-
ming languages allow applications to include codes
from their software libraries. A software library is a
collection of programs that can be reused in building
larger program modules. In such a programming
environment, a language compiler converts a program
into an object module. The linker using underlying
metadata then combines all object modules that make
up a program including the object modules that are
obtained from the library. Some example libraries are
C++ Class Library [15], Standard C Library [16] and
Microsoft’s MFC Library.
Although the joint management of metadata and
data is not complicated, libraries like these are fairly
static. Once a library has been developed and released
to the external world, its structure and interface cannot
change without great difficulty for the user base [19].
Using a library is equivalent to using a programming
language. Changes in a class library are often more
difficult to deal with than changes in a programming
language.
3.2. Stage II. Co-management of metadata
With the advent of database management system,
efforts were made to manage metadata more explic-
itly. This idea had been exploited to make the library
managers more dynamic, where users need a mecha-
nism to insert, delete, modify and retrieve the content
of the library easily.
To create easy manipulation of the library, we use
metadata along with the library. In this technique,
metadata of codes need to be captured and used along
with the actual library. The metadata of a code contain
data about the code. The idea is to store the metadata
in a database. If software library gets updated, corre-
sponding entries in the database are changed.
The evidence of this approach can be seen in
software maintenance literature. Leiter et al. [9]
describe a software tool that entails a relational data-
base with an interactive interface that supports queries
about programs written in object-oriented languages.
156 A. Sen / Decision Support Systems 37 (2004) 151–173
Linos and Courtois [10] describe a tool set that
includes tools to detect a mixture of procedural and
object-oriented program information from the C++
code. Examples include files (both source and include
files), data types, functions, constants, variables,
parameters, classes, objects, etc. These are then used
to populate a database. Different browsing tools are
available to display metadata on the screen. IBM has
an information retrieval tool called ReDiscoveryk
based on OS/2 that can be used to create and search
databases of information about virtually any kind of
database or file system.
3.3. Stage III. Explicit metadata management
In this stage, metadata and their management come
of age. Instead of being a side issue, metadata now is
looked upon as glue that binds many enterprise
resources such as applications, ERP activities, Internet
technologies and data warehouses. The metadata
management tool (also known as repository) now
becomes crucial.
The first attempt in this area was the introduction
of data dictionary. The data dictionary typically was
very data focused. It provided a centralized repository
of information about data such as meaning, relation-
ships, origin, domain, usage and format [11,12]. The
purpose of data dictionary was to assist the DBAs in
planning, controlling and evaluating the collection,
storage and use of data.
To allow the library to hold other items that are not
software, a second mechanism, called repositories (see
Section 2), was introduced. According to Bernstein
[2], a repository is a shared database of information
about engineered artifacts, such as software, docu-
ments, maps and other things. For example, a reposi-
tory that supports software development and
deployment tools could store database descriptions,
form definitions, controls, documents, interface defi-
nitions, source code, help text and others.
In the late 1980s, IBM offered to build a universal
repository, a repository that would store the outputs of
a wide variety of CASE tools and make the stored
elements available to any tool that requested them.
Unfortunately, IBM’s repository was never delivered
due to software industry’s shift from COBOL orien-
tation to C/C++ orientation and also from main frame
to client – server [23]. In the late 1990s, however, this
idea got picked up again by other vendors who now
have developed several repositories. These vendors
include Microsoft (with Microsoft Repository),
Unisys (with Universal Repository), Computer Asso-
ciate (with Platinum Repository), ViaSoft (with
Rochade Repository), Softlab (with Enabler Reposi-
tory) and Oracle (with Oracle Repository).
To understand what constitutes a repository, let us
describe the important features of any repository. The
first feature of a repository is the types of metadata it
manages. The types include database metadata, data
model metadata, data movement metadata, business
rules metadata, application component metadata,
data access metadata and data warehouse related
metadata. Other features of a repository include an
information model that describes the core metadata
types of the repository; a specification language that is
the formal specification language for the tool; a
language that supports tool interoperability across
different products; a standard query language to query
the metadata; and others. We provide as an example
the features of Microsoft Repository Service (see
Table 1). From this table, it is apparent that the
repository architecture must include four layers. The
top-most layer is usually the user access layer. The
objectives of this layer are to provide support to
browsing through the repository data in a client–
server setup or in the web environment; to provide
an interface with CASE tools; and to help the com-
ponent-based development (CBD) tools develop and
manage components.
The second layer is called common infrastructure
support layer. The objective of this layer is to support
common issues among assets such as integration of
tools, cross-platform support, event management and
others. The next lower layer is called repository
engine layer. This layer is responsible to create and
manage repository objects, their versions and config-
urations. It is designed and developed using an object-
oriented paradigm. Finally, at the bottom level (called
data services layer), a repository has a data server and
supports insertion, deletion, modification and retrieval
of the repository data. The technology effectively
manages the information assets of the enterprise,
provides a metaview of the developmental process
across all software development tools, uses common
data store for development tools and allows version
control of any object.
 A. Sen / Decision Support Systems 37 (2004) 151–173
Table 1
Microsoft’s Repository 2.1 http://msdn.microsoft.com/library/default.asp?url=/library/en-us/reposit/htm/
reconthearchitectureofmicrosoftrepository.asp
Feature
Date repository was introduced
Major objectives of the repository
Types of metadata
Information model that describes the core
metadata types
Specification language
Repository architecture
Environment particulars
Development kit
Metadata browsing facility including web
browsing
Exchange metadata between multiple
heterogeneous repositories
Object management service support
Component-based framework support
Query language support
3.4. Stage IV. Metadata integration management
The focus of this stage is to manage the
integration of diverse metadata in an application.
There are two kinds of integration: tool integration
and data integration. Tools typically generate tool-
specific metadata. To establish communication
among multiple tools, these metadata need to be
integrated. Similarly, data can be of different for-
mats and can come from different worlds. These
data also need to be integrated to help decision
support. Repository technology (of Stage III) that
was originally created to manage metadata can be
used to integrate.
The repository technology described above sup-
ports the metadata integration as long as the metadata
follow a common model, like Microsoft’s Component
157
Availability of the feature in the repository
Repository 1.0 was introduced with Microsoft’s VB5.0 in 1997:
. To provide a metaview of the development process across all development tools
like VB, C++, etc.
. To study the impact of changes of any object.
. To provide a common data store for all development tools.
Component definitions, development and deployment models, reusable software
components, data warehouse descriptions, web pages, etc.
Supports Meta Data Coalition’s Open Information Model (OIM). OIM is a
set of metadata specifications to facilitate sharing between and reuse between
tools and systems. The OIM consists of over 200 types and 100 relationships,
described in UML and organized in easy-to-use and easy-to-extend subject areas.
Universal Modeling Language (UML)—an object-oriented language.
Repository Services 2.1 has four layers:
. Top Layer—Tools and Application Layer. This layer supports integrated tools,
metadata-driven applications and other utilities and browsers.
. Layer III—OIM that supports shared metadata objects and structures,
relationship among objects, etc.
. Layer II—Repository object manager responsible for life cycle management
and maps objects to tables.
. Bottom layer—persistence service with jet database engine or SQL server.
Windows-based system.
Systems development kit (SDK) is available.
Yes
Supports XML interchange format
(a) Version Control, (b) Composite Object Service, (c) Collection Service
(e.g. relationship service), etc.
Component Object Model (COM) based
SQL
Object Model (COM) architecture. In the world of
diverse metadata integration, such as currently being
used in data warehouse [17] domain, the tools do not
have a common model. This is because different tools
from different vendors with different specifications (or
metadata) are brought into the project. Tools can be
‘‘best-of-breed,’’ ‘‘most compatible,’’ or ‘‘most eco-
nomic.’’ This is also true for data. The repository-
based metadata management is not very effective in
this kind of metadata integration.
Noticing this kind of integration problem, two
standardization groups, Meta Data Coalition (MDC)
and Object Management Group (OMG), started to
work on the metadata standard in the 1990s. They
called the metadata standard a metamodel. The MDC
group along with Microsoft in the late 1990s devel-
oped and adopted a metadata standard called Open
158 A. Sen / Decision Support Systems 37 (2004) 151–173

Fig. 2. The data warehousing process for the TIPS project with changed statistics.
 A. Sen / Decision Support Systems 37 (2004) 151–173 159

Information Model (OIM). The other group, OMG,
created a model called Common Data Warehouse
Metamodel (CWM) [17] in early 2000. In June
2000, MDC members decided to join OMG and
support one standard in metadata for integration.
4. Proposing metadata warehouse—the final
frontier
In the earlier sections, we have seen how the
concept of metadata has evolved and how tools to

Fig. 3. A partial load script used in TIP data warehouse.

160 A. Sen / Decision Support Systems 37 (2004) 151–173

Fig. 4. A partial transform script used in TIP data warehouse.

 A. Sen / Decision Support Systems 37 (2004) 151–173 161

Fig. 4 (continued ).

manage them have matured from simple library man-
agers to techniques that help integrate tools metadata.
The integration is accomplished through a repository
manager, supporting a common model of metadata
under a single environment; or with a diverse set of
metadata that follow a metamodel, called CWM.
The study of CWM reveals that it does not only
support integration of tool metadata, it also supports
the integration of metadata created in a data ware-
house project. The metadata in a data warehouse
project are of two kinds—back-room types and
front-room types. The back room metadata is process
related and guides the extraction, cleaning and loading
processes. Examples include specifications related to
source data, such as source schemas, old formats for
archived mainframe data, ownership description of the
source, automated extract tool settings and others;
data staging metadata such as data cleaning specifi-
cations, slowly changing dimension policies and
others; data transform logs; and DBMS system table
contents. The front room metadata is more descrip-
tive, and it helps query tools and report-writer func-
tion smoothly. Examples include join specifications,
network security user privilege profiles, usage and
access maps, network security usage statistics and
others. It is not only important to capture all these
metadata, but we also need to worry about the changes
that happen to them.
Unfortunately, at the present time, no single robust
tool is available that completely supports the CWM.
Few vendors, like Microsoft’s Metadata Services 2000
and Teradata’s MDS, have offered extended reposi-
tory-based tools. These tools support repository fea-
tures but fail to cover the decision support questions
162 A. Sen / Decision Support Systems 37 (2004) 151–173
on metadata changes. We propose a new management
tool as the final stage of evolution of the metadata
management. We call this last stage metadata ware-
house. We adopt this term from the data warehouse
literature to emphasize the need for creating a ware-
house full of metadata only. We need this warehouse
to manage different kinds of changes that happen in a
regular data warehouse. These changes include organ-
izational changes, technological changes, physical
changes, process changes, end-user requirement
changes and website changes [21].
The best way to demonstrate the need of a meta-
data warehouse is to look at a real-life data warehouse
project. Once we describe the development of the data
warehouse, we need to see if we can answer the
following metadata questions. They include what
kinds of metadata we need to store? What are the
changes in this metadata? How frequent are these
changes? Where do they happen most?
4.1. Developing an academic data warehouse
To answer the above questions, we look at a
working data warehouse. With the popularity of the
data warehouse in the industry, several academic
institutions (like Arizona State University, University
of Wisconsin at Milwaukee and Texas A&M Univer-
sity) are currently developing data warehouses to meet
their decision support needs. In this paper, we use one
such data warehouse project to develop the concept of
metadata warehouse.
The goal of Texas A&M University System’s
(TAMUS) Data Warehouse Project, called TAMUS
Information Portal (TIP), was to develop an auto-
mated information system. This system uses decision
support analysis of financial, personnel and academic
data for the Board of Regents, and management of
the TAMUS and its member institutions and agencies
[5]. The current on-line transaction processing sys-
tems are unable to meet the growing need for
management information. Before the data warehouse,
the ability to perform analysis and cross-functional
analysis was a manual process. In this environment,
satisfying information requests were problematical
for several reasons:
 requests are often unpredictable, important and
urgent, which means that key information technol-
ogy resources have to be redirected, making other
project work discontinuous and extending project
completion dates;
 developing answers is laborious, because custom
queries or report programs have to be analyzed,
designed, developed and tested;
 answers can be inconsistent, because answers
developed by different analysts—or even by the
same analyst at different times—have the potential
of producing results that do not agree with other or
previous results.
The TAMUS, spurred by a state management
control audit, embarked on a process to define its
information needs and to develop a system to address
those needs. The resulting data warehouse project was
defined and managed by business users and supported
by an NCR’s Teradata system.
The primary objective of the project was to
develop an enterprise data warehouse environment
to:
 provide the ability to access historical information
currently stored in existing systems;
 provide interactive and ad hoc data exploration;
 provide information on a secure intranet website;
 provide access to information for ad hoc query,
analysis and report writing;
 serve as a credible source of information;
 support the decision-making process without
hindering the performance of the operational sys-
tems.
Fig. 2 describes the data warehousing process used
in the TIP project. The project has been successfully
completed in the beginning of 2001 and has realized
some initial benefits. They are as follows:
 integrating disparate administrative system data into
a common system for all management reporting;
 providing a ‘‘single version of the truth’’;
 serving as a source system for data mart projects
undertaken by various members;
 providing the end-user capabilities to users as
‘‘drill down’’ analysis of electronic reports; reports
that can include graphs and or tables; reports that
are easily modified; reports that can be requested
anytime by anyone (with appropriate security and
access privileges); reports that can be automatically
 A. Sen / Decision Support Systems 37 (2004) 151–173
run when the data warehouse is updated and
‘‘pushed’’ to the users.
4.2. Details of metadata in the TIP warehouse
Various types of metadata are at play in TIP data
warehouse. We start with data sets that are currently
sourced from the Texas Higher Education Coordinat-
ing Boards. There are nine such flat files. Each file
contains summarized data elements that are not related
to data elements in the other files. For example,
Student-Report data source (cbm001) has 34 attributes
such as record code, institution code, student identi-
fication number, gender, classification and others. In
the ETL process, we have Teradata Load Scripts and
Transform Scripts. Figs. 3 and 4 show a partial picture
of these scripts.
The logical and physical diagrams also form a
source of metadata. In TIPs, these diagrams were
created in Erwin Development tool and can be repro-
duced in XML. Fig. 5 shows a partial XML descrip-
tion of the physical ERD. The TIP data warehouse
runs on Teradata DBMS that provides another set of
metadata called target data table metadata (see Fig. 6
for a sample DDL). The view also is a source of
metadata (see Fig. 7 for a sample user view DDL).
Finally, the end-user application uses Business Object
end-user tool. A sample metadata, called Business
Object Universe, is also shown in Fig. 8.
4.3. Changes in the TIP metadata
As business needs or conditions change over time,
a data warehouse must be responsive, continuously
evaluating effectiveness of the changes. Change man-
agement spans all components of a data warehouse
and thus plays a vital role in the ongoing development
and overall success of a data warehouse. To manage
the data warehouse, it is necessary to balance two
conflicting environment goals: maximizing the use of
the data warehouse asset while consistently achieving
user expectations by continuously monitoring the
effect of business changes.
To study the need for a metadata warehouse, we
focus on the academic data of TIP. Changes happen in
two ways: push and pull. In the push-oriented changes,
changes happen at the data source level. Changes from
these sources can occur each reporting period, typically
163
a semester, and are the primary reason the academic
subject area of the TIP is the most volatile. Because this
data source is outside the control of TAMUS, changes
are pushed to the TIP with limited opportunity for
feedback and thus are implemented after the fact.
On the other hand, in pull-oriented changes, the
changes are actually at the end-user level. Any changes
in the end-user requirements will pull the changes in
the different metadata in TIP data warehouse.
A sample of these changes is shown in Fig. 9.
These changes can happen at any level of the schema
(a metadata). For example, the changes can happen at
the domain level, at the attribute level, or at the entity-
relationship level. Therefore, having a change man-
agement tool will greatly improve the identification of
the effect of the changes and aid in automation of
implementing the changes throughout the process.
4.4. Why do we need a metadata warehouse for the
TIP project?
Just as the data warehouse project was initiated in
response to business decision support, so also the
development of a metadata warehouse must also be
driven by business needs. Some of these business
needs related to the changes in TIP metadata are
described below:
 Tracking the life cycle for each data element. The
ability to provide life cycle information for each
data element—from data source to end-user
representation, such as source field name, ETL
processing, target table definition and representa-
tion to end-user (including transformations and
derived columns).
 Analyzing the effect of changes in TIP. This changed
information can be used to analyze the impact of a
change. Knowing the source field name aids in the
analysis of the effect of a change to the TIP when-
ever changes to the source system occur. In addition,
knowing the ETL processing can help determine the
impact of changing a source field. We provide two
examples to describe the effects of changes:
 Change of a two-character field to a four-
character field, as was the case with the ‘‘year’’
attribute in the cbm001 data source in September
1999, only affected the load elements of the TIP
(load scripts, staging tables and transform
164 A. Sen / Decision Support Systems 37 (2004) 151–173

Fig. 5. A partial XML description of physical ERD in TIP data warehouse.

scripts) as the target table column was already
defined as an integer data type.
 Deletion of ‘‘Nursing Program Acceptance’’
data element in the cbm001 data source caused
two major changes. It affected the load scripts
and the transform scripts. However, the target
table schema and view definition remain
unchanged because the historical values need
to be maintained. This change is then classified
as having an average impact.
 A. Sen / Decision Support Systems 37 (2004) 151–173 165

Fig. 6. A partial target table DDL in TIP data warehouse.

 Merging of current data with historical data. As
source data elements slowly change over time,
there is a need to merge the changing data
definitions with the existing historical data
definitions in a meaningful way for the user.
This merge should include strategies for handling
null values because new attributes will not have
values in the historical data. For example, the
‘‘Appointment 0.1%—Salary’’ data element in
the cbm008 data source was deleted and
replaced with other salary data elements at a
completely different level, that is, instead of
relating salary to the faculty appointment, salary
became related to the funding source (e.g. State
Appropriations, Auxiliary Enterprises, Restricted,
etc.). In this case, because there was no basis for
comparing the existing historical salary data with
the new salary definition, columns containing the
new data were added in addition to retain the
existing salary columns to facilitate historical
comparisons.
 Identifying and archiving obsolete data ele-
ments. Some data elements become obsolete
and do not provide any business value after a
given period of time. These elements need to
be identified and archived to historical tables.
The archived obsolete elements can then be
used for comparative historical trend analysis.
For example, the ‘‘Nursing Program Accept-
ance’’ data element in the cbm001 data source
is no longer being used in analysis and is
eligible for archival.
166 A. Sen / Decision Support Systems 37 (2004) 151–173

Fig. 7. A partial user view DDL in TIP data warehouse.

 Identifying required data elements to be used in
analysis of new data sources. As new data sources
are incorporated into the TIP, the data warehouse
needs to have some ability to analyze gaps between
the required data elements and the data elements
provided by the new data source. The gaps can
occur at the source (new data source) or target
(TIP). Identification of the gaps can be used to
drive changes to meet necessary business require-
ments. This is especially true for other areas of the
TIP, where there are foreign key relationships and
required attributes, such as the account data in the
financial subject area. Incorporating additional data
sources into the existing data model has uncovered
gaps in both the new data sources, that is, data
elements that are not stored in the source systems,
as well as gaps in the TIP, that is, data elements not
originally modeled.
4.5. Metadata warehouse architecture
Using Inmon’s data warehouse definition [7], we
define a metadata warehouse as a subject-oriented,
integrated, time-variant and somewhat volatile collec-
tion of metadata in support of change management
process. Like repository, a metadata warehouse stores
 A. Sen / Decision Support Systems 37 (2004) 151–173 167

Fig. 8. A partial business object universe in TIP data warehouse.

the warehouse metadata in a database, needs to have a
common information model and provides version
control. However, unlike repositories, the main focus
of a metadata warehouse is to manage changes that
continuously happen in the metadata. Typically, these
changes in the metadata ripple throughout the entire
metadata base.
From the earlier case study, we observe two
things. First, the size of the metadata used by the
data warehouse is not small. In fact, currently in the
TIP project, the academic part of the data ware-
house is 331 MB, while its metadata is about 1
MB. This is very typical in all data warehouses.
Second, the metadata in the TIP project can grow as
it goes through a lot of changes. This kind of
volatility is somewhat uncharacteristic of a regular
data warehouse. These changes are important and
need to be managed. This section discusses three
possible architectures of a metadata warehouse. The
first type uses existing data warehouse techniques,
the second type extends the repository technique
and the third type combines these two. We now
describe these architectures with their advantages
and disadvantages.
168 A. Sen / Decision Support Systems 37 (2004) 151–173

Fig. 9. A partial list of changes in TIP data warehouse metadata.

4.5.1. Type 1 approach (using data warehouse
architecture)
Like in a regular data warehouse, the metadata are
gathered from various metadata sources. The metadata
sources include descriptions of traditional data sour-
ces, load scripts, logical and physical data models,
staging table definitions and data warehouse table
definitions, view definitions and definitions used in
end-user applications. The metadata come in different
formats and are extracted from these sources by using
some specialized extract, transfer and load (ETL) tool.
They are then temporarily stored in an area called
Metadata Staging area. It is actually the construction
site of the metadata warehouse. Flat files or relations
are typically used in the metadata staging area. After
the extraction is done, metadata sets need to be
cleaned, standardized, duplicates removed, integrated
and transformed by using the ETL tool. Once we are
 A. Sen / Decision Support Systems 37 (2004) 151–173 169

satisfied with the quality of the metadata, we again
use ETL tool to load the metadata warehouse (see Fig.
10). Currently, not much is available in metadata ETL
tools domain.
The metadata warehouse needs to be located in a
metadata warehouse server. A metadata warehouse
server uses a relational database management system
(like NCR’s Teradata, IBM’s DB2, Microsoft’s SQL
Server or Oracle’s Oracle 9i). The logical and physical
models for the metadata warehouse must support the
metadata descriptions and their changes. The models
are translated to three NF relations before storing them
in a DBMS. The models can be designed from scratch
or can use some generic template. These relations are
used with specialized query-processing capabilities
that support decision support activities, indexing capa-
bilities and reside on very fast processors. The metadata
warehouse typically spans the entire enterprise. We feel
managing version control can become a serious prob-
lem in this kind of metadata warehouse.
Finally, a metadata warehouse provides end-user
facilities for various types of metadata access and
reporting. These tools need to allow decision support
queries with respect to change management, provide
mechanisms to manage metadata changes and
finally, to allow the metadata to be evaluated. The
end-user tools for the metadata warehouse are also
not well developed. Most vendors like NCR and
others provide simplistic browsing facilities through
the metadata.

Fig. 10. The metadata warehouse architecture (Type 1 approach).

170 A. Sen / Decision Support Systems 37 (2004) 151–173

4.5.2. Type 2 approach (using repository-based
architecture)
The advantage of using repository-based architec-
ture is to make use of some of its salient features such
as version control (to manage changes), inherent infor-
mation model to design the metadata warehouse
model, cross-platform support (i.e. the integration of
tool metadata), event management and others. The
architecture (see Fig. 11) is very similar to Fig. 9, but
uses repository engine as the main tool. The metadata
sources use the specialized ETL-like tools to bring the
metadata into the staging or holding area. The staging
area is very much like we see in a warehouse. It again
uses temporary storage and does not use any repository
infrastructure. The metadata warehouse is designed
using an information model and then implemented on
top of the repository tables. The end-user tools in this
case use standard repository tools to provide the
repository-styled decision support capabilities.
4.5.3. Type 3 approach (using integrated architecture)
Both techniques described above have their advan-
tages and disadvantages. We propose here an integra-
tion framework that takes the best of both worlds. We
capture the ETL tools, the concept of data staging and
some end-user tools ideas from Type 1 architecture. We
then connect these with the benefits of Type 2 archi-
tecture to get the use of repository engine, version
control mechanism and integration framework for tool
metadata that is inherent in the repository world. The

Fig. 11. The metadata warehouse architecture (Type 2 approach).

 A. Sen / Decision Support Systems 37 (2004) 151–173 171

Fig. 12. The metadata warehouse integrated architecture (Type 3 approach).

emphasis on CWM can be achieved through the
repository technology also. The result of this kind of
integration can be seen in Fig. 12.
5. Conclusion and future research
While TIP Data Warehouse is a specific data
warehouse, the need for metadata warehouse is uni-
versal. We see its need in the web data warehouses
also. In fact, the changes are much more prevalent in
the web world than in the regular data warehouse
world [21]. The current research provides a window to
show how metadata has come of age and now
demands its own warehouse. Several conclusions
and future research can be drawn from the Metadata
Warehouse Proposal for the TIP project.
First, the major challenge in the metadata ware-
house is to develop a design methodology that can
capture the metadata scattered in the data warehouse
projects and put them in the metadata warehouse. It is
unclear at this time how that can be done. Will the
design methodology follow any data warehouse
design technique or will it be a new one?
Second, the design and development of ETL tools
for the metadata warehouse need to be done. It is
172 A. Sen / Decision Support Systems 37 (2004) 151–173
somewhat unusual to make use of the current ETL
tools for the metadata. We feel that the metadata is
quite different from regular data and needs to be
treated differently.
Third, the end-user tools need to be developed for
the metadata warehouse. As one can see in Fig. 12,
end-users activities for a metadata warehouse span
from version control (a repository activity) to change
management decision support (a warehouse-type
activity). We may need to drill down or drill up the
data for end-users very much like we do in a regular
data warehouse. Current repository-based tools (see
Fig. 11) do not have these choices.
Fourth, it is obvious that the Type 3 architecture is
the best of both worlds. Unfortunately, none of the
current data warehouse and repository vendors imple-
ment this architecture.
Fifth, as indicated before, the metadata warehouse
becomes a big problem in a web world. The reason is
the incessant changes that happen in the website,
starting the site structure changes, page content
changes to the click-record log format changes and
others. To manage all these changes, metadata ware-
house seems to be the answer.
Finally, the question of integrating the metadata
warehouse with the existing data warehouse is very
important. As the metadata warehouse is a new
phenomenon, we need to figure out how to incorpo-
rate it with the existing data warehouse. Will the
metadata warehouse be kept apart from the data
warehouse, or will it be integrated with it?
Acknowledgements
The research was partially funded by a grant from
Teradata, a division of NCR. The author acknowl-
edges the help of Debbie Doran from the Texas A&M
University System, and David Riegel and Mary Gros
from Teradata.
References
[1] F. Banchilhon, C. Delobel, P. Kanellakis, Building an Object-
Oriented Database System, Morgan Kaufmann Publishers,
San Mateo, CA, 1992.
[2] P.A. Bernstein, Repositories and object-oriented databases,
SIGMOD Record 27 (1) (1998 March) 88 – 96.
[3] C.J. Date, An Introduction to Database Systems, Addison-
Wesley Publishing, New York, 1995.
[4] A. Deshpande, D. Van Gucht, An implementation for nested
relational databases, Tech Report Number 234, Department of
Computer Science, Indiana University, February 1988.
[5] D. Doran, TIP management change repository, An Internal
Report, Texas A&M University System, 2001.
[6] E. Gamma, R. Helm, R. Johnson, J. Vlissides, Design Pat-
terns: Elements of Object-Oriented Software, Addison-Wesley
Publishing, New York, 1995.
[7] W.H. Inmon, Building Data Warehouse, Wiley, New York,
1992 and 1996.
[8] R. Kimball, L. Reeves, M. Ross, W. Thronthwaite, The Data
Warehouse Lifecycle Toolkit, Wiley, New York, 1998.
[9] M. Leiter, S. Meyer, S.P. Reiss, Support for maintaining ob-
ject-oriented programs, Transactions on Software Engineering
18 (12) (1992) 1045 – 1052.
[10] P.K. Linos, V. Courtois, A toolset for maintaining hybrid C++
programs, Software Maintenance Research and Practice 8
(1996) 389 – 419.
[11] D. Marco, Building and Managing Meta Data Repository: A
Full Life Cycle Guide, Wiley, New York, 2000.
[12] D. Marco, Meta data repositories: where we’ve been and
where we’re going, DM Review, February 2002. http://
www.dmreview.com/master.cfm?NavID=198&EdID=4612.
[13] Y. Matsumoto, A software factory: an overall approach to
software production, in: P. Freedman (Ed.), Software Reus-
ability, Computer Society Press, Los Alamitos, CA, 1987, pp.
155 – 178. http://www.computer.org/cspress/.
[14] J.-M. Morel, J. Faget, The REBOOT environment, Proceed-
ings of Advances in Software Reuse, Lucca, Italy, March 24 –
26, IEEE Computer Society Press, Los Alamitos, CA, 1993,
pp. 80 – 88.
[15] M. Pace, Evaluation and Comparison of C+ Class Libraries,
http://www.desy.de/user/projects/C++/Projects.html.
[16] P.J. Plauger, The Standard C Library, Prentice-Hall, Engle-
wood Cliffs, NJ, 1992.
[17] J. Poole, D. Change, D. Tolbert, D. Mellor, Common Ware-
house Metamodel, Wiley, New York, 2002.
[18] R. Prieto-Diaz, A software classification scheme, PhD Thesis,
University of California, Irvine, 1985.
[19] D. Reed, Tools for software reuse, Object Magazine (1995
February) 63 – 67.
[20] H.-J. Schek, M.H. Scholl, The relational model with relation-
valued attributes, Information Systems 11 (4) (1986).
[21] A. Sen, Metadata warehousing: a methodological support to
data warehouse change management, Working Paper, Texas
A&M University, 2002.
[22] I. Sommerville, Software Engineering, Addison-Wesley Pub-
lishing, New York, 1992.
[23] K. Watterson, More than databases, repositories should hold
the Corporate IS Jewels. Why don’t they? Byte Magazine,
(1998 May) 1 – 9.
 A. Sen / Decision Support Systems 37 (2004) 151–173 173

Arun Sen is a full professor and Mays

Fellow in the Department of Information
and Operations Management in Texas
A&M University. Before joining the Texas
A&M University in 1986, he was an assis-
tant and a tenured associate professor in the
Department of Management Science, Uni-
versity of South Carolina. He holds an
MTech in Electronics (from Calcutta Uni-
versity, India in 1971), an MS in Computer
Science (from Penn State University in
1976) and a PhD in Information Systems (from Penn State University
in 1979).
He has published 42 research papers in many journals such as MIS
Quarterly, Information Systems Research, IEEE Transactions on
Systems, Man and Cybernetics, IEEE Transactions on Software
Engineering, IEEE Transactions on Engineering Management,
Decision Sciences, Communications of the ACM, Information
Systems, Computers and OR, Omega, European Journal of Oper-
ations Research, Decision Support Systems, Journal of MIS, Infor-
mation and Management and others. His research interests include
decision support systems, database management, repository man-
agement and software reuse, case-based reasoning, technical and
behavioral aspects of data warehouse and E-Commerce.
He was an associate editor of Journal of Database Management. He
was a special issue editor for Decision Support Systems, Commu-
nications of the ACM, Database and Expert Systems with Applica-
tion. He was the chair of INFORMS College on Information
Systems, a program chair for the 1996 Workshop on Information
Technology and Systems (WITS) Conference and a track chair
(Decision Support Systems and AI track) for the 1996 National DSI
Conference.