Beruflich Dokumente
Kultur Dokumente
Abstract: Similarity retrieval by spatial content (i.e., Focusing mainly on color, texture and shape, the
using multiple objects and their interelationships) in work referred to above does not show how to handle
Image DataBases (IDBs) is still an open problem and multiple objects or regions in example queries, nor
has received considerable attention in the literature. their inter-relationships. In this case, for two images
In this work, we focus our attention on “queries by to be similar, not only the shape, color and texture
example” image and we study methods for answering properties of individual image regions must be similar
such queries including the well accepted “editing dis- but, they must have the same arrangement (i.e., spatial
tance” on Attributed Relational Graphs (ARGs) and relationships) in the two images. 2D strings [10] and
the “Hungarian Method” for graph matching. We “Attributed Relational Graphs” (ARGs) [1] are exam-
evaluate the time responses and the accuracy of these ples of methods focusing on spatial relationships. 2D
methods using a database of synthetic by realistic med- string matching give binary (i.e., “yes/no”) answers
ical images. Our experiments indicate that the ARG and may yield “false alarms” (non-qualifying images)
editing distance, although the slowest, is the most ac- and “false dismissals” (qualifying but not retrieved im-
curate method. ages) [11].
In this work, we focus our attention on ARGs. We
experimented with medical tomographic images but
Key-Words: Image DataBase, Image retrieval, At- ARGs can handle any image type. Specifically, we
tributed Relational Graphs. deal with the following problem:
We have a collection of N medical images show-
1. Introduction
ing multiple objects. Each image has been seg-
mented (automatically or manually) to more than
The management of large volumes of digital images
one objects or regions. Figure 1 shows an exam-
in many application fields (e.g., medicine [1], crim-
ple of an original grey-level image (left) and its
inal investigation [2], trademark/copyright detection
corresponding segmented form (right).
[3] etc.) has generated additional interest in methods
and tools for real time archiving and retrieval of im- The queries are “by example”: The user specifies
ages by content. Consequently, content-based image a desirable image (e.g., “find the examinations
retrieval has become the object of intensive research which are similar to Smith’s examination”).
activities over the past few years [4, 5].
The system has to return all images showing sim-
Several approaches to the problem of content-based
ilar objects with the query in similar spatial re-
image management have been proposed and some have
lationships. There may exist extra objects in a
been implemented on research prototypes and commer-
retrieved image but not in the query.
cial systems [6, 7, 8, 9]. There are two general goals
common to all IDB systems: The contributions of this work are summarized in
the following:
Effectiveness: An IDB system must retrieve similar
images with as few errors as possible. We study methods capable of answering spatial
queries such as the well accepted “editing dis-
Efficiency: It must be fast to meet the speed require- tance” on ARGs and the so called “Hungarian
ments of an IDB environment. Method” for graph matching.
features form vectors. A set of features that has been
used successfully [11, 1] is the following:
5
UNKNOWN
Individual regions: size, computed as the size of the
LIVER
2 area it occupies, perimeter computed as the length
TUMOR
4 of its bounding contour, roundness, computed as
3 SPINE
the ratio of the smallest to the largest second mo-
BODY
OUTLINE
1
ment and orientation, defined to be the angle be-
tween the horizontal direction and the axis of elon-
gation. This is the axis of least second moment.
Figure 1. Example of an original grey-level im-
age (left) and its segmented form (right). Spatial relationships: The following features are
used to describe the spatial relationships between
regions: relative distance, computed as the min-
imum distance between their surrounding con-
We evaluate the effectiveness (i.e., accuracy) of
tours, relative orientation defined as the angle
all these methods in terms of precision and re-
with the horizontal direction of the line connect-
call. Our evaluation is based on human rel-
ing the centers of mass of their regions and relative
position taking values (;1 0 1) corresponding to
evance judgements following a well established
methodology from the information retrieval field.
regions which are the one inside the other (-1),
We evaluate the efficiency (i.e., time responses) outside each other (0) or, the second inside the
of all these methods. first one (1) respectively.
The rest of this paper is organized as follows: A To avoid discontinuities in the measurement of an-
short review of the underlaying theory is presented in gles (i.e., orientations of 1 degree should have mea-
Section 2. Our evaluation method is discussed in Sec- surements similar to those of orientations of 359 de-
tion 3. Experimental results are presented in Section 4 grees), angles are represented by both sin and cos. All
followed by conclusions in Section 5. attributes are normalized in the range ;1 1]: Angle
attributes (i.e., sin, cos take values in the range ;1 1];
2. Attributed Relational Graphs (ARGs) distances and areas are normalized in the range 0::1]
by dividing them respectively by the perimeter and
Figure 2 illustrates the ARG computed to the image the area of the largest region (i.e., body outline in our
of Figure 1. Nodes correspond to regions and arcs case). Notice that, an ARG representation can handle
correspond to relationships between regions. The arcs any set of features as labels.
are directed from the outer to the contained region but
their direction also depends on object labels: Arcs are 2.1. ARG Editing Distance
always directed (a) from body outline to the remaining
objects, (b) from liver to spine and (c) from the most Matching a query and a stored ARG is treated as
common objects (i.e., body outline, liver and spine in an “error-correcting subgraph isomorphism” problem
our application) to the remaining objects. [12]: Given two graphs G (query) and G0 (stored or
reference ARG) there is always a sequence of “edit”
operations that transform G to a subgraph of G 0 (i.e.,
object 4 object 2
object 3 object 1
involves not only finding a sequence of edit operations,
spine body outline
but also finding the one that yields the minimum total
cost. This can be formulated as a tree search problem
Figure 2. Example ARG. which is be solved by an A ? algorithm [13]: Each tree
node corresponds to a matching of a pair of subgraphs
Both nodes and arcs are labeled by attributes cor- from the two input ARGs. A transition from a state to
responding to properties (features) of objects and to another corresponds to the embedding (matching) of a
properties of relationships respectively. Node and edge pair of unmatched nodes (one from each ARG) and of
their associated edges, into the already matched sub-
graphs. The matching operations for each embedding
[0] [0] [0]
are recorded and their costs are accumulated. The tree
expands until a complete match is found. A complete (1,1) (1,2) (1,3)
The method for ARG matching referred to above (2,2) (2,3) (2,1) (2,3) (2,1) (2,2)
[2] [1] [2] [1] [2] [2]
finds the optimal solution but it has exponential time
(and space) complexity in the worst case (i.e., the [0 + 0] [0 + 1] [0 + 1] [0 + 1] [0 + 1] [0 + 1]
matching tree can grow exponentially for large graphs). (3,3) (3,2) (3,3) (3,1) (3,2) (3,1)
optimal solution.
Figure 3 shows an example of a query ARG (left) Figure 4. Matching tree.
and of a reference (model) ARG (right). All nodes and
all edges are labeled by attribute vectors. image I (the relationships are ignored). To compute
this association, we need the costs (weights) C (i j ) of
associating each object i in Q with each object j in I ,
i j n, where n in the number of objects in the two
(1,1) (2,2)
object 1 object 1
(1,1) (1,2)
images. C (i j ) is computed as the distance between
their corresponding feature vectors.
Let F () be a mapping from objects in Q to objects
(1,1) (1,2)
(2,1)
Xn C (i F (i)):
(1,1)
(2,0) (0,2) (0,2) (2,1)
object 2 object 3 object 2 object 3
(2,2)
DF (Q I ) = (1 )
i=1
4 5
3
3
2.3. Hungarian Plus 2
1
2
6 1
QUERY ANSWER
ANSWER ANSWER
D(Q I ) = 1;
Pij
(3) Figure 6. Example of a query image (top left)
+ ( )
] (
1
1n n n;1 =2
1 cos ij ;F (i)F (j )
2) and of three qualifying images.
Best 50 Answers
precision
dataset of 13 500 synthetic segmented images 1 of [1].
0.4
To evaluate our methods we created 20 characteristic
queries. All images and the queries contain between 4
0.2
and 8 objects.
The experiments are designed to demonstrate the:
0
0.2 0.3 0.4 0.5 0.6 0.7 0.8
recall
Hungarian method, is in turn, preferred over Hungar- [9] John R. Smith and Shih-Fu Chang. VisualSEEk:
ian Plus: It is almost as accurate as Hungarian Plus A Fully Automated Content Based Image Query
but, much faster. Future work includes the experimen- System. In Proceedings of ACM Multime-
tation with more methods (e.g., 2D strings) and the dia Conference, Boston Ma., November 1996.
investigation of indexing methods for speeding-up the (http://disney.ctr.columbia.edu/VisualSEEk).
retrieval response times of these methods.
[10] S.-K. Chang, E. Jungert, and G. Tortora, editors.
Intelligent Image DataBase Systems. World Sci-
Acknowledgements
entific, 1996.
We are grateful to Charalambos Genzis for his help [11] Euripides G.M. Petrakis and Stelios C. Or-
in the experiments. phanoudakis. Methodology for the Representa-
tion, Indexing and Retrieval of Images by Con-
References tent. Image and Vision Computing, 11(8):504–
521, October 1993.
[1] Euripides G.M. Petrakis and Christos Falout-
sos. Similarity Searching in Medical Image [12] B. T. Messmer. Efficient Graph Matching
Databases. IEEE Transactions on Knowledge Algorithms. PhD thesis, University of Bern,
and Data Engineering, 9(3):435–447, May/June Switzerland, 1995. (http://iamwww.unibe.ch/˜
1997. fkiwww/projects/GraphMatch.html).
[2] Nalini K. Ratha, Kalle Karu, Shaoyun Chen, and [13] N. J. Nilsson. Principles of Artificial Intelligence.
Anil K. Jain. A Real Time Matching System for Tioga, Palo Alto, 1980.
Large Fingerprint Databases. IEEE Transactions
[14] William J. Christmas, Josef Kittler, and Maria
on Pattern Analysis and Machine Intelligence, Petrou. Structural Matching in Computer Vision
18(8):799–813, August 1996.
Using Probabilistic Relaxation. IEEE Transac-
[3] Anil K. Jain and Aditya Vailaya. Shape-Based tions on Pattern Analysis and Machine Intelli-
Retrieval: A Case Study With Trademark Im- gence, 17(8):749–764, August 1995.
age Databases. Pattern Recognition, 31(9):1369–
[15] H. A. Almohamad and S. O. Duffuaa. A Lin-
13990, 1998. ear Programming Approach for the Weighted
[4] William I. Grosky. Management Multimedia In- Graph Matching Problem. IEEE Transactions
formation in Database Systems. Communications on Pattern Analysis and Machine Intelligence,
of the ACM, 40(12):73–80, 1997. 15(5):522–525, May 1993.
[5] Alberto Del Bimbo. Visual Information Re- [16] C. Papadimitriou and K. Steiglitz. Combinato-
trieval. Morgan Kaufmann Publishers, Inc., rial Optimization: Algorithms and Complexity,
1999. chapter 11, pages 247–255. Engelwood Cliffs:
Prentice Hall, 1982.
[6] Amarnath Gupta and Ramesh Jain. Vi-
sual Information Retrieval. Communica- [17] Venkat N. Gudivada and Vijay V. Raghavan. De-
tions of the ACM, 40(5):71–79, May 1997. sign and Evaluation of Algorithms for Image Re-
(http://www.virage.com). trieval by Spatial Similarity. ACM Transactions
on Information Systems, 13(2):115–144, 1995.
[7] M. Flickner et. al. Query By Im-
age and Video Content: The QBIC Sys- [18] D.K. Harman, editor. The Second Text Retrieval
tem. Computer, 28(9):23–32, September 1995. Conference. National Institute of Standards and
(http://wwwqbic.almaden.ibm.com). Technology Gaithersburg, MD, 1994.