Proceedings of the International Multiconference on
Computer Science and Information Technology pp. 273–279
An Evaluation of Concept Suggestion Strategies for
Professional Multimedia Archives
Marco A. Palomino and Michael P. Oakes
University of Sunderland
Sunderland, SR6 0DD
United Kingdom
E-mail: {marco.palomino,michael.oakes}@sunderland.ac.uk
Abstract—Choosing the optimal terms to represent a search
engine query is not trivial, and may involve an iterative process
such as relevance feedback, repeated unaided attempts by the
user or the automatic suggestion of additional terms, which
the user may select or reject. This is particularly true of a
multimedia search engine which searches on concepts as well
as user-input terms, since the user is unlikely to be familiar
with all the system-known concepts. We propose three concept
suggestion strategies: suggestion by normalised textual matching,
by semantic similarity, and by the use of a similarity matrix.
We have evaluated these three strategies by comparing machine
suggestions with the suggestions produced by professional anno-
tators, using the measures of micro- and macro- precision and
recall. The semantic similarity strategy outperformed the use of
a similarity matrix at a range of thresholds. Normalised textual
matching, which is the simplest strategy, performed almost as
well as the semantic similarity one on recall-based measures,
and even better on precision-based and F-based measures.
I. I NTRODUCTION
T HE STATE-of-the-art in content-based image retrieval
has progressed significantly in the past few years. How-
ever, it has not yet succeeded in bridging the semantic gap
between human concepts—as expressed by keyword-based
queries—and low-level visual features that are extracted from
images [1]. Hence, a recent trend in multimedia retrieval con-
sists of learning a controlled vocabulary of semantic concepts
and employing them as entry points to query a collection [2].
As part of the EU-funded VITALAS project [3], which
aims to offer a system dedicated to the intelligent access to
professional multimedia archives, we have created a vocab-
ulary of 525 concepts, whose entries refer to objects—like
an elephant—settings—like an interview—events—
like a celebration_after_scoring—and pure video
format—like a detected overlayed_text.
In the context of the VITALAS project, the concept vocabu-
lary gives users semantic access to picture and video, allowing
them to query on the presence or absence of content elements.
Nevertheless, selecting the right topic from a large vocabulary
may be a lengthy and resource-consuming task, and the
users are unlikely to be familiar with all the system-known
concepts. Therefore, we have developed a suggestion engine
that analyses textual captions contained in a picture archive
and automatically determines the most relevant concepts for
querying such an archive.
273
ISBN 978-83-60810-22-4
ISSN 1896-7094
Tom Wuytack
Belga News Agency
Rue Frederic Pelletier 8b
1030 Brussels, Belgium
E-mail: wut@belga.be
We propose three concept suggestion strategies: suggestion
by normalised textual matching, by semantic similarity, and
by the use of a similarity matrix. All three strategies have
been evaluated by comparing machine suggestions with the
observations made by professional annotators, using the mea-
sures of micro- and macro- precision and recall. The semantic
similarity strategy, where pictures are characterised by the TF-
IDF weights of the words in their captions, outperformed the
use of a similarity matrix at a range of thresholds. Normalised
textual matching performed almost as well as the semantic
similarity technique on recall-based measures, and even better
on precision- and F- based measures.
The remainder of this document is structured as follows:
Section II describes the multimedia collection that we em-
ployed in our study. Section III presents the VITALAS concept
vocabulary and elaborates on the acquisition of ground-truth
annotation. Section IV explains the three different strate-
gies that we propose to implement a suggestion engine.
Section V reports on the evaluation of our experiments.
Section VI introduces the related work, and, finally, Sec-
tion VII states our conclusions.
II. B ELGA’ S M ULTIMEDIA A RCHIVE
In order to carry out our research, we made use of a large
multimedia archive owned by the Belga News Agency [4].
Belga’s archive covers Belgian and international news and
current affairs—politics and economics, finance and social
affairs, sports, culture and personalities. Although Belga’s
content is published in four different formats—text, pictures,
audio and video—, this document concentrates on pictures and
their associated textual captions, exclusively.
A caption is a free-text field whose content is created
by photographers, and offers an explanation or designation
accompanying a picture posted on Belga’s website. Each group
of photographers has its own conventions and styles to present
information. As a consequence, certain captions include not
only text relative to pictures, but also names of photographers,
their initials, acronyms of press agencies, dates when the
pictures were taken or published and some other ancillary
information. Since none of these particulars are deleted before
posting, we decided to keep them in our analysis too.
274
To ensure that we had enough material, Belga granted us
access to a set of 1,727,159 pictures and captions that were
published on its website between 22 June 2007 and 2 October
2007. Belga also gave us access to its query logs for exactly
the same period. Figure 1 displays an example of a typical
picture and caption posted on Belga’s website.
Fig. 1. Example of a Belga Picture and Caption: Nicole Vaidisova of
the Czech Republic receives a service during her quarterfinal match against
Amelie Mauresmo of France at the Kremlin Cup tennis tournament in
Moscow, Friday 13 October 2006.
III. VITALAS A NNOTATED C ONCEPT VOCABULARY
The VITALAS concept vocabulary is largely derived from
the automatic extraction of keywords that characterise Belga’s
archive. However, it has been refined manually over time, in
order to improve its functionality.
Originally, the VITALAS vocabulary was produced from a
comparison between Belga’s captions and a model of general
English language. The words that deviated from the model
were very specific to the captions and thus made appropriate
keywords to characterise the archive. Professional annota-
tors evaluated the keywords and removed those that they
considered unsuitable. The remaining keywords became the
first entries in the VITALAS concept vocabulary. Later on,
these entries were extended manually to guarantee that the
vocabulary comprised as many categories as available in the
news domain. Finally, we mined Belga’s query logs program-
matically to extract keywords that reflected the most important
concepts from the users’ perspective. These concepts were also
added to the vocabulary.
Further details related to the VITALAS concept vocabu-
lary have been published by Palomino et al. [5], and the
entire concept vocabulary for the VITALAS project can be
browsed from the authors’ website (http://osiris.sunderland.ac.
uk/∼cs0mpl/VITALAS/).
A. Ground-Truth Annotation
The availability of multimedia annotations is essential for
large-scale retrieval systems to work in practice. Hence, VI-
TALAS selected 96,600 pictures published on Belga’s website,
and employed professional annotators to determine some of
the concepts that best described them. For annotation purposes,
the presence of a concept was assumed to be binary: it was
either visible in a picture or not. The location of a concept in
the picture was not taken into account.
To secure a sound basis for our experimental setup, a total
of 1,000 pictures were annotated for each of the 525 concepts.
PROCEEDINGS OF THE IMCSIT. VOLUME 4, 2009
Approximately 500 of the pictures annotated for a particular
concept were chosen from the results of queries submitted by
Belga users who had included that concept’s name. The rest
of the pictures were chosen randomly. The first half of the
pictures annotated for each concept contains mostly positive
samples, whereas the second one supplies mostly negative
samples. Achieving this balance was vital for the evaluation
of our experiments—and will also be useful when applying
supervised learning methods, which we envisage to do in later
stages of our work.
As part of the ground-truth annotation, we included the
provision of a glossary: a document containing a textual
definition—or description—for each concept in the vocabulary,
together with relevant keywords and references to positive
images. Table I shows an example of a VITALAS concept—
food—, accompanied by its description and reference to
positive images. Table I also shows a picture that has been
annotated positively as an image that does correspond with
the concept food.
TABLE I
E XAMPLE OF C ONCEPT D ESCRIPTION
Concept name: food
Concept description: An image showing any substance
reasonably expected to be ingested by a human or an
animal for nutrition or pleasure.
Relevant keywords: Cooking, meal.
Examples of positive images: A picture of a table
showing a served meal; a picture of dishes ready to be
consumed; a picture of meat, fish, fruit or vegetables for
sale in a market.
The VITALAS manual annotation process has yielded an
incomplete, but reliable ground-truth for our concept vocabu-
lary. Certainly, we would like to have all the pictures annotated
for all of the concepts; yet, despite resource limitations, we
have gathered a reasonably large subset of annotated pictures.
IV. C ONCEPT S UGGESTION S TRATEGIES
We identify three different strategies for suggesting to users
the most relevant concepts related to a particular picture
caption:
• Normalised textual matching: A method based on the dis-
covery of straight textual matches between the contents
of a caption and the concept names in the vocabulary.
MARCO A. PALOMINO ET. AL: AN EVALUATION OF CONCEPT SUGGESTION STRATEGIES
• Semantic similarity: A method based on the calculation
of the cosine measure of similarity between the caption
and the textual description of each concept, as it is found
in the glossary.
• Similarity matrix: A method based on the creation of a
similarity matrix that shows the degree of association
between every pair of concepts. Given a particular cap-
tion, whose relation to a certain set of concepts has been
detected by means of straight textual matches, we employ
the similarity matrix to establish all the concepts that are
relevant to it, even though their names do not appear in
the caption.
In the following subsections, we detail these strategies.
A. Normalised Textual Matching
This approach begins by normalising the caption and the
concept names: all text is converted to lower case, punctuation
is removed, and extremely common and semantically non-
selective words are deleted—the stop-word list which we use
was built by Salton and Buckley for the experimental SMART
information retrieval system [6]. The normalisation process
stems all text too, reducing inflectional and derivationally
related forms of a word to a common base—the particular
algorithm for stemming English words that we use is Porter’s
algorithm [7].
As a second step in this method, we look for exact matches
between the words in the normalised caption and those
available in the normalised concept names. For illustration
purposes, Table II displays an example of a caption, its
normalised version and the resulting matches with the concept
vocabulary.
TABLE II
E XAMPLE OF N ORMALISED T EXTUAL M ATCH
Original caption: Red Cross delegation chief for Peru, Michel
Minnig waves to photographers as he leaves
the Japanese Ambassador’s residence after
meeting with rebels of the Tupac Amaru
Revolutionary Movement (MRTA) in Lima,
Peru.
Normalised caption: red cross deleg chief peru michel minnig
wave photograph leav japanes ambassador
resid meet rebel tupac amaru revolutionari
movement mrta lima peru
Concept names: ambassador ... government ...
photographers ...
Normalised names: ambassador ... govern ...
photograph ...
Matches: ambassador, government,
photographers
In the case of concept names made of more than one word—
such as, davis_cup—different heuristics may be applied.
We may look for precise matches of all the words contained in
275
the concept name, which would limit the number of matches
considerably, but would ensure that only captions referring
explicitly to the concept name are matched.
A more relaxed approach would be to select one or more
words as headwords. For instance, we may say that davis
is the headword for the concept davis_cup, and we will
automatically associate all the matches of davis with this
concept. This is the approach that we have pursued.
Due to space limitations, we cannot list here all the head-
words for the concepts in the vocabulary. Readers are welcome
to visit the authors’ website for further details on this matter
[8]. Some concepts—such as ac_milan_soccer—have
two headwords—milan and soccer—though they do not
have to appear together to provide a match for the concept.
An appropriate selection of headwords is not a trivial task.
Indeed, such a task should be undertaken by a group of
specialists who are familiar with Belga’s collection and can
decide which words are relevant to each particular concept.
Nevertheless, the goal of this study is just to analyse the
feasibility of the approach. Refining the list of headwords and
improving the quality of our results is left for future work.
Future versions of our suggestion engine will evaluate the
impact of different headwords on the precision of the results
retrieved, and the final choice of headwords that are associated
with each concept will be left to specialists.
B. Semantic Similarity
As explained in Subsection III-A, each concept ω in the
VITALAS concept vocabulary is associated with a textual
description dω . Then, we can measure the semantic similarity
between a caption and the textual description of each different
concept. Both captions and descriptions are normalised before
examining their semantic similarity.
We represent each concept description as a vector, whose
entries correspond to unique normalised words. Since concept
descriptions are written in natural language, word distribution
corresponds, roughly, with Zipf’s law [9]. Therefore, the vector
space model proposed by Salton et al. [10] is appropriate
for our semantic analysis. Specifically, with a collection of
descriptions D, a concept description dω in D, a total of
N Con concepts, a caption q and a total of N Cap captions,
we use the following formula to compute the cosine similarity
between caption q and concept description dω :
P
wkq · wkdω
k∈(q∩dω )
sim(q, dω ) = r P r P ,
(wkq )2 (wkdω )2
k∈q k∈dω
where

N Cap

wkq = fkq · idfkq , idfkq = log2 Dkq
 
N Con
wkdω = fkdω · idfk , idfkdω = log2 Dkdω
Note that fkq is the frequency with which term k occurs in
caption q, and Dkq is the number of captions containing k.
276 PROCEEDINGS OF THE IMCSIT. VOLUME 4, 2009

Similarly, fkdω is the frequency with which term k occurs
in description dω , and Dkdω is the number of descriptions
containing k.
It can be demonstrated that the resulting similarity between
q and dω ranges from 0 meaning no match, to 1 meaning com-
plete match, with in-between values indicating intermediate
similarity [11]. Hence, we may choose a threshold and suggest
concept ω as a possible match for q only if the similarity
between q and dω is above the threshold.
Considering the relatively small size of both captions and
concept descriptions, it is computationally inexpensive to cal-
culate their semantic similarity. Using an Intel
c Xeon
c CPU being able to choose from a variety of possible additional
5150 processor with 2GB of RAM, running under Microsoft
Windows XP 2002 SP2, our Java-based suggestion engine can
calculate the semantic similarity between a single caption and
all of the 525 concept descriptions in only a couple of seconds.
Although the strategy described in this subsection is
based on term frequencies—TF—and inverse document
frequencies—IDF—we refer to it as semantic similarity, rather
than TF-IDF approach, because the words that are contained
in the textual description of a concept, and have high TF-IDF
weights, give us the best indication as to what the textual de-
scription is about and, therefore, reflect the semantic similarity
between captions and textual descriptions of concepts.
C. Similarity Matrix
Our final strategy aims to discover related concepts for
each picture, which are hopefully relevant to the users’ inter-
ests even though they do not appear in the picture caption.
The method requires the creation of a concept-to-concept
similarity matrix showing the degree of association between
every pair of concepts. To produce each entry in this ma-
trix, we first represent the relation between each concept
and all of the captions in a different vector. The entries
in this vector contain the TF-IDF weights of the concept
headwords in each caption of the collection. For concepts
with more than one headword, we sum the TF-IDF weights
of the different headwords. Then, the similarity of a pair
of concepts is given by the cosine similarity of their corre-
sponding vectors—each concept has a similarity of 1 with
on it. Afterwards, we derive from the similarity matrix the
“similarity” between the appearing headwords and all the
concepts in the vocabulary. As in the case of the semantic
similarity approach, we define a threshold and suggest to
the user only those concepts whose corresponding values are
above the threshold.
In the following section, we report on the use of different
thresholds for this strategy and the previous one. As we lower
the threshold a larger number of false positives is suggested,
but recall also increases, which is more important than preci-
sion for concept suggestion, because users will benefit from
concepts and can easily reject unsuitable ones.
V. E VALUATION
To evaluate the efficiency of our suggestion strategies, we
have employed two standard measures used in information
retrieval: precision and recall [13].
Precision and recall are defined in terms of a set of retrieved
documents and a set of relevant documents. Given the partic-
ular characteristics of Belga’s archive, and the conditions of
the ground-truth annotation that we have exploited, we have
taken a modified version of the traditional definitions. For the
remainder of this document, we refer to the recall for caption
q (Rq ), and the precision for caption q (Pq ), as
|ΩqA ∩ΩqM | |ΩqA ∩ΩqM |
Rq ≡ |ΩqA |
, Pq ≡ |ΩqM |
where ΩqA is the set of concepts that the annotators associated
with the picture whose caption is q, and ΩqM is the set of
concepts that our automatic suggestion strategy proposed for
the same picture.
Averaging over the total number of pictures in the collection
C, we made use of the following definitions for micro-recall
(µR ), micro-precision (µP ), macro-recall (MR ) and macro-
precision (MP ) [14],
|ΩqA ∩ΩqM | |ΩqA ∩ΩqM |
P P
q∈C q∈C
µR ≡ P q
|ΩA |
, µP ≡ P q
|ΩM |
,
q∈C q∈C

itself. For instance, the most similar concepts to actress P

Rq
P
Pq
were filming (0.40), film_festival_cannes (0.39), MR ≡
q∈C
, MP ≡
q∈C
.
|C| |C|
festival (0.27), academy_award (0.15), actor (0.15)
and award (0.15). Table III displays the values of each of the measures defined
Salton and Buckley [12] found that associative retrieval above for all the suggestion strategies described in Section IV.
methods work only if locally valid relations between terms are The highest values achieved for each measure are presented
used—i.e., relations which are valid for a specific document in bold font.
collection in a given domain. Semantic similarity with a low threshold (0.05) performed
By generating a similarity matrix automatically from better than all the other strategies on the recall-based measures.
Belga’s specific documents, we are able to produce concept Normalised textual matching performed very well too, in com-
suggestions adapted to that particular domain much better than parison with the other strategies at most thresholds. Indeed, if
if we had used the term relations available from a thesaurus we computed a micro F-measure, defined as µF ≡ µ2µ P µR
P +µR
, and
2MP MR
or library classification. a macro F-measure, defined as MF ≡ MP +MR , we would
Once the similarity matrix has been created, we can suggest realise that the F-based measures for the normalised textual
concepts relevant to a particular picture by scanning its caption matching strategy—namely, µF = 0.28 and MF = 0.33—are
and searching for appearances of the concept headwords almost as good as those of any other strategy at any threshold.
MARCO A. PALOMINO ET. AL: AN EVALUATION OF CONCEPT SUGGESTION STRATEGIES 277

TABLE III
E VALUATION R ESULTS textual matching, where the annotators pick up the headwords,
and possibly extend them manually to better reflect concept
Semantic Similarity Textual relations, may yield very good results.
Measure Threshold Similarity Matrix Match Figure 2 compares the efficiency of our suggestion strate-
µR 0.75 gies with a curve that represents the observations made by
µP 0.17 the annotators. The annotators curve displays the cumulative
MR 0.77
number of concepts chosen by them every time they annotate a
new picture. The curves representing our suggestion strategies
MP 0.21
indicate how many concepts we have proposed that match the
µR 0.20 0.34 0.32 observations of the annotators.
µP 0.20 0.26 0.11
MR 0.20 0.35 0.33 Fig. 2. Efficiency Curves
50,000
Annotators curve
MP 0.20 0.35 0.17 Semantic similarity (threshold = 0.05)
Normalised textual match
Similarity matrix (threshold = 0.05)
µR 0.15 0.49 0.34

Number of concepts chosen by the annotators

40,000
µP 0.15 0.17 0.09
MR 0.15 0.49 0.35
30,000
MP 0.15 0.27 0.15
µR 0.10 0.65 0.39
20,000
µP 0.10 0.08 0.06
MR 0.10 0.66 0.41
10,000
MP 0.10 0.16 0.11
µR 0.05 0.83 0.51
µP 0.05 0.02 0.02 10,000 20,000 30,000 40,000
Number of annotated pictures
MR 0.05 0.84 0.53
MP 0.05 0.06 0.05
The comparison between our suggestion strategies and the
observations made by the annotators reported on Figure 2 was
assessed for a total of 36,700 pictures. The average number
Table IV displays the F-based measure values for the of concepts extracted by each of our strategies is displayed
three different strategies at the analysed thresholds. Semantic in Table V. Although we have tested our strategies based on
similarity with the highest threshold (0.20), which produced semantic similarity and the use of a similarity matrix at a range
the best precision-based measures, also accomplished the best of thresholds, the curves shown in Figure 2 are those whose
F-based measures. efficiency is closer to the annotators curve.
TABLE IV TABLE V
F-M EASURE VALUES AVERAGE N UMBER OF C ONCEPTS S UGGESTED

Semantic Similarity Textual Semantic Similarity Textual

Measure Threshold Similarity Matrix Match Threshold Similarity Matrix Match
µF 0.28 8.29
MF 0.33 0.20 1.71 3.97
µF 0.20 0.29 0.16 0.15 3.83 4.89
MF 0.20 0.35 0.22 0.10 10.84 8.99
µF 0.15 0.25 0.14 0.05 43.78 26.97
MF 0.15 0.35 0.21
µF 0.10 0.14 0.10
MF 0.10 0.26 0.17
VI. R ELATED W ORK
µF 0.05 0.04 0.04
Helping users to explore multimedia collections, and appro-
MF 0.05 0.11 0.09
priately express their information needs, is a service supplied
by most search engines. Typically, this service is presented
in the form of query suggestions, employing techniques such
We believe that an improved version of the normalised as query auto-completion [15], query spelling-correction [16],
278
query expansion [17] and query rewriting [18]. While our
work may certainly benefit from some of these techniques,
our recommendations to the user must be limited to a specific
concept vocabulary, and our users’ queries may need to be re-
placed completely, rather than just expanded or spell-checked,
because concepts may be grammatically and orthographically
unrelated. Additionally, our particular setting does not require
a fully automated system: we are supposed to give users the
opportunity to select related terms, which allows for user-aided
disambiguation.
The study published by Hoogs et al. [19] is among the first
ones to add semantics to concept detection, by establishing
links with a general-purpose ontology, which connected a
limited set of visual attributes to WordNet [20]. However,
combining low-level visual attributes with concepts in an
ontology is a rather difficult task, due to the so-called semantic
gap between them [21].
To cope with the demand for ground-truth, Lin et al.
initiated a collaborative annotation effort for the TRECVID
2003 benchmark [22]. Using tools from Christel et al. [23]
and Volkmer et al. [24], a common annotation effort was again
made for the TRECVID 2005 benchmark, yielding a large set
of annotated examples for 39 concepts taken from a prede-
fined collection [25]. We have provided a larger compilation,
increasing the concept vocabulary to 525 concepts, and getting
1,000 annotated pictures per concept.
Snoek et al. [26] have published a study closely related
to ours. They also produced a concept suggestion strategy
based on semantic similarity; yet, they made use of the Lucene
search engine [27] as part of their implementation, and the
goal of their work was different, as they attempted to obtain
semantic descriptions and structure from WordNet [20]. Even
though the results presented by Snoek et al. are not conclusive,
we may consider following their recommendations on ontology
querying in future versions of our suggestion engine.
Another existing solution to the problem that we have
approached involves the use of latent semantic indexing (LSI)
[28], where relations between queries and documents are
determined according to a co-occurrence analysis. Unfortu-
nately, LSI is computationally expensive, especially consider-
ing the size of Belga’s collection and its continuously growing
nature—Belga claims to add between 5,000 and 10,000 new
pictures and captions to its archive on a daily basis [4].
A potentially better alternative than LSI has been proposed
by Wang et al. [29], who learnt concept relations from
short natural language texts, and stored them in a structure
called fuzzy associated concept mapping. New concepts, not
explicitly present in the original texts, were recommended to
the users based on this mapping.
Rather than using a mapping in which only selected rela-
tions between concepts are stored, our suggestion strategies
favour the utilisation of a similarity matrix, where every
concept has a relation with every other concept, namely a real
value in the range 0 to 1.
PROCEEDINGS OF THE IMCSIT. VOLUME 4, 2009
VII. C ONCLUSIONS
We have described three methods of concept suggestion,
with the aim of helping multimedia search engine users
enhance their initial keyword queries with additional terms
corresponding to system-known concepts, namely suggestion
by normalised textual matching, by semantic similarity and
by the use of a similarity matrix. Although normalised textual
matching is the simplest technique that we have assessed, and
the fastest one to execute, it performed very well on the recall-
based measures and F-based measures, and only slightly less
well on the precision-based measures. High recall is more
important than high precision for query term suggestion, since
the user will benefit from being able to choose from a range of
possible additional concepts, and can easily reject unsuitable
ones. However, in approaches such as the one proposed by
Palomino et al. [30], where discovered additional concepts
are automatically added to a query without prior user approval,
precision is more important, since the inclusion of non-relevant
concepts in the query can severely degrade performance.
Even though we have evaluated the quality of our concept
selection using recall- and precision- based measures, we
still need to measure the effect of our concept suggestion
facilities on the overall search engine performance. Such
evaluations are being undertaken by our research partners
at the Institut National de l’Audiovisuel [31], using recall,
precision, and subjective measures of user satisfaction with
the overall system.
ACKNOWLEDGMENT
This research was supported under the EU-funded VITA-
LAS project (project number FP6-045389). The authors are
very grateful to the Belga News Agency for providing the
data used to carry out their research.
R EFERENCES
[1] A. W. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain,
“Content-Based Image Retrieval at the End of the Early Years,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 22,
no. 12, pp. 1349–1380, December 2000.
[2] M. Worring, C. G. M. Snoek, B. Huurnink, J. C. van Gemert, D. C.
Koelma, and O. de Rooij, “The MediaMill Large-Lexicon Concept
Suggestion Engine,” in Proceedings of the 14th ACM International
Conference on Multimedia. Santa Barbara, CA: Association for
Computing Machinery, October 2006, pp. 785–786.
[3] VITALAS, Video and Image Indexing and Retrieval in the Large Scale,
http://www.vitalas.org/.
[4] Belga, Belga News Agency, http://www.belga.be/.
[5] M. A. Palomino, M. P. Oakes, and T. Wuytack, “Automatic Extraction
of Keywords for a Multimedia Search Engine Using the Chi-Square
Test,” in Proceedings of the 9th Dutch-Belgian Information Retrieval
Workshop, Enschede, The Netherlands, February 2009, pp. 3–10.
[6] C. Buckley, “Implementation of the SMART Information Retrieval
System,” Computer Science Department, Cornell University, Ithaca, New
York, Tech. Rep. TR85-686, May 1985.
[7] M. Porter, “An Algorithm for Suffix Stripping,” Progam, vol. 14, no. 3,
pp. 130–137, July 1980.
[8] M. A. Palomino, VITALAS Concept-Suggestion Engine, http://osiris.
sunderland.ac.uk/∼ cs0mpl/VITALAS/.
[9] W. J. Reed, “The Pareto, Zipf and Other Power Laws,” Economics
Letters, vol. 74, no. 1, pp. 15–19, December 2001.
[10] G. Salton, A. Wong, and C. Yang, “A Vector Space Model for Automatic
Indexing,” Communications of the ACM, vol. 18, no. 11, pp. 613–620,
November 1975.
MARCO A. PALOMINO ET. AL: AN EVALUATION OF CONCEPT SUGGESTION STRATEGIES
[11] D. Widdows, “Measuring Similarity and Distance,” in Geometry and
Meaning. CSLI Publications, November 2004.
[12] G. Salton and C. Buckley, “On the Use of Spreading Activation
Methods in Automatic Information Retrieval,” in Proceedings of the 11th
ACM SIGIR Conference on Research and Development in Information
Retrieval, Grenoble, France, June 1988, pp. 147–160.
[13] R. K. Belew, Finding Out About: A Cognitive Perspective on Search
Engine Technology and the WWW. Cambridge, UK: Cambridge
University Press, February 2001.
[14] T. Joachims, Learning to Classify Text Using Support Vector Machines:
Methods, Theory and Algorithms. Kluwer Academic Publishers, April
2002.
[15] K. Church and B. Thiesson, “The Wild Thing Goes Local,” in Proceed-
ings of the ACM SIGIR Conference on Research and Development in
Information Retrieval. Amsterdam, The Netherlands: Association for
Computing Machinery, Inc., July 2007, pp. 901–901.
[16] S.-P. Cucerzan and E. Brill, “Spelling Correction as an Iterative Process
that Exploits the Collective Knowledge of Web Users,” in Proceedings of
the Conference on Empirical Methods in Natural Language Processing,
Barcelona, Spain, July 2004, pp. 293–300.
[17] R. W. White and G. Marchionini, “Examining the Effectiveness of
Real-Time Query Expansion,” Information Processing and Management,
vol. 43, no. 3, pp. 685–704, May 2007.
[18] R. Jones, B. Rey, O. Madani, and W. Greiner, “Generating Query
Substitutions,” in Proceedings of the International World Wide Web Con-
ference. Edinburgh, Scotland: Association for Computing Machinery,
May 2006, pp. 387–396.
[19] A. Hoogs, J. Rittscher, G. Stein, and J. Schmiederer, “Video Content An-
notation Using Visual Analysis and a Large Semantic Knowledgebase,”
in Proceedings of the IEEE Computer Society Conference on Computer
Vision and Pattern Recognition, Madison, WI, June 2003, pp. 327–334.
[20] C. Fellbaum, WordNet: An Electronic Lexical Database. Cambridge,
MA: MIT Press, May 1998.
[21] C. Dorai, “Bridging the Semantic Gap in Content Management Systems:
279
Computational Media Aesthetics,” in Proceedings of the International
Conference on Computational Semiotics for Games and New Media.
Amsterdam, The Netherlands: Kluwer Academic Publishers, September
2001, pp. 94–99.
[22] C.-Y. Lin, B. L. Tseng, and J. R. Smith, “Video Collaborative Anno-
tation Forum: Establishing Ground-Truth Labels on Large Multimedia
Datasets,” in Proceedings of the TRECVID 2003 Workshop, Gaithers-
burg, MD, November 2003.
[23] M. Christel, T. Kanade, M. Mauldin, R. Reddy, M. Sirbu, S. Stevens,
and H. Wactlar, “Informedia Digital Video Library,” Communications of
the ACM, vol. 38, no. 4, pp. 57–58, April 1995.
[24] T. Volkmer, S. Tahaghoghi, and J. A. Thom, “Modelling Human Judge-
ment of Digital Imagery for Multimedia Retrieval,” IEEE Transactions
on Multimedia, vol. 9, no. 5, pp. 967–974, August 2007.
[25] M. Naphade, J. R. Smith, J. Tesic, S.-F. Chang, W. Hsu, L. Kennedy,
A. Hauptmann, and J. Curtis, “Large-Scale Concept Ontology for
Multimedia,” IEEE MultiMedia, vol. 13, no. 3, pp. 86–91, 2006.
[26] C. G. Snoek, B. Huurnink, L. Hollink, M. de Rijke, G. Schreiber, and
M. Worring, “Adding Semantics to Detectors for Video Retrieval,” IEEE
Transactions on Multimedia, vol. 9, no. 5, pp. 975–986, August 2007.
[27] Lucene, The Lucene Search Engine, http://lucene.apache.org/.
[28] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and
R. Harshman, “Indexing by Latent Semantic Analysis,” Journal of the
Society for Information Science, vol. 41, no. 6, pp. 391–407, 1990.
[29] W. M. Wang, C.F.Cheung, W. B. Lee, and S. K. Kwok, “Mining
Knowledge from Natural Language Texts Using Fuzzy Associated
Concept Mapping,” Information Processing and Management, vol. 44,
no. 5, pp. 1707–1719, September 2008.
[30] M. A. Palomino, M. P. Oakes, and Y. Xu, “An Adaptive Method to
Associate Pictures with Indexing Terms,” in Proceedings of the 2nd
International Workshop on Adaptive Information Retrieval, London, UK,
October 2008, pp. 38–43.
[31] INA, Institut National de l’Audiovisuel, http://www.ina.fr/.