Information Retrieval, 4, 295310, 2001

c 2001 Kluwer Academic Publishers. Manufactured in The Netherlands.

Changes in Search Tactics and Relevance

Judgements when Preparing a Research Proposal
A Summary of the Findings of a Longitudinal Study
PERTTI VAKKARI Pertti.Vakkari@uta.
Department of Information Studies, University of Tampere, Finland

Received December 28, 2000; Revised May 7, 2001

Abstract. The article summarizes the empirical results on relations between students problem stages in the
course of writing their research proposals for their masters theses and the information sought, choice of search
terms and tactics and relevance assessments of the information found for that task. The study is based on Kuhlthaus
model of the information search process. The results of the study show that there is a close connection between the
students problem stages (mental model) in the task performance and the information sought, search tactics used,
and the assessment of the relevance and utility of the information found. The corroborated hypotheses extend and
specify ideas in Kuhlthaus model in the domain of IR. A theory of task-based information searching based on the
empirical ndings of the study is presented.

Keywords: tasks, search tactics, relevance, usefullness on information, a theory of information retrieval

1. Introduction

The aim of IR systems is to support subjects in nding useful information for their tasks.
The tasks might relate to their work, interests or everyday life activities. This is reected in
the growing interest in studying information searching in IR systems in relation to the tasks
that generate it (Ingwersen 1996, Bystrom and Jarvelin 1995). These studies try to connect
information searching as a part of a more comprehensive process of task performance. Most
of them have focused on studying elements of information retrieval during a search session.
Thus, methodologically, these studies have been ableto observe the processes of information
searching, but not so much the process of task performance (Vakkari 1999). However, it is
important to take closer account of how changes in subjects understanding of their task
is reected in their information needs, and consequently in the way they formulate their
queries and assess the relevance and usefulness of the information items found. Variation
in this understanding and in consequent searching will result in a variation of search results
in terms of their usefulness for the task. By studying how task performance interacts with
searching in IR systems it may be possible to provide systems design with results and ideas
on how to tailor systems more exactly to support human tasks.
An important contribution towards a view of information searching as a process for
supporting task performance has been made by Kuhlthau (1993). She has shown in a
series of studies that (learning) tasks and problem solving consists of several stages. Each
stage contains typical thoughts, information search strategies and also information types.
296 VAKKARI

Her theory predicts that people search and use information differently in the various stages of
task performance. Dening a topic, and formulating a focus for the task seem to differentiate
information behavior accordingly. However, Kuhlthaus model does not directly address
searching in IR systems. Results from other studies suggest that her theory could be used
as a point of departure in analyzing IR tactics and relevance assessments of the information
found in task performance. Empirical studies by Bateman (1998), Spink et al. (1998),
Sutton (1994) and Wang (1997) have shown connections between relevance assessment
and task performance process. Bystrom and Jarvelin (1995), Ellis and Haugan (1997),
Wilson (1998) and Yang (1997) have shown links between information search tactics and
different phases of task performance. Theoretical ideas by Belkin (1993), Ingwersen (1996),
Marchionini (1995), Schamber (1994), Spink et al. (1998) and Vakkari (1999) are strongly
indications of the patterning of IR tactics and relevance assessments according to stages of
task performance.
This article provides a summary of the ndings of an empirical study on relations between
a particular task and information searching. Detailed results from the sub-studies have been
published elsewhere. In those articles we have analyzed how students problem stages in
the course of writing their research proposals for their masters theses are connected to
the types of information sought (Vakkari 2000b, Vakkari and Pennanen 2001), changes in
search tactics and term choices (Vakkari 2000a), and patterning of relevance assessments
of both references and full-texts (Vakkari and Hakala 2000). In this article we also present
a theory of task-based information retrieval based on the empirical ndings.
The framework of this study is Kuhlthaus (1993) model of the information search process
(ISP) enriched with some ideas from cognitive psychology. Kuhlthau (1993) has shown that
phases in task performance differentiate the types of information searched for and the major
ways ofsearching. The hypotheses of our study are based onher model. The basic hypothesis
is that the stages of task performance are connected to the types of information searched for,
to the changes of search tactics and terms and to relevance judgements. The derivation of
the consequences from Kuhlthaus model naturally included rening some of its concepts
to t the features of information retrieval. The explanatory factor of the modelproblem
stageremained the same and some variables to be explainedinformation types searched
for, search tactics, term choices and relevance judgementswere specied and connected
to the problem stages.

2. Framework

Kuhlthau (1993) has shown in a series of empirical studies that learning tasks and problem
solving by students and library users consist of several stages. Her theory holds that people
search for and use information differently depending on the stage of the process. The central
concepts of Kuhlthaus model are presented in Table 1.
Kuhlthau (1993) differentiates the search process into six stages, which are: initiation,
topic selection, topic exploration, focus formulation, collection and presentation. The nd-
ing of a focus is crucial in the search process. The focus is comparable to a hypothesis
for accomplishing the task. Prior to focus formation thoughts are general, fragmentary
and vague, and actions involve seeking background information. The searcher is unable to
CHANGES IN SEARCH TACTICS 297

Table 1. Main concepts in studies by Kuhlthau and Vakkari.

Kuhlthaus model Vakkaris hypotheses

Stages in ISP Stages in task performance

Initiation
Selection
Exploration
Formulation
Collection
Presentation
Types of information
General information (Background)
Specic information (Relevant)
Pertinent information (Focused)
Sources of information
PersonsInformation systems
Relevance judgements
Degree of usefulness
Search tactics
Browsing or querying
Search terms and operators
Mental models/Thoughts
General or vagueclearer or focused
a
Used as an auxiliary concept.
Pre-focus
--
--
Formulation
Post-focus
--
Types of information
General information (Background)
Faceted background information
Specic information
Sources of information
PersonsInformation systems
Relevance judgements
Degree of relevance
Relevance criteria used
Type of contributory information
Search tactics
A categorization containing 12 tactics
Search terms and operators
Amount
Types (Synonym, NT, BT, RT)
Operator types
Mental models/Thoughts
General or vagueclearer or focused
Differentiation of conceptual structurea
Integration of conceptual structurea

construe the task and unable to express precisely what kind of information is needed for
it. Browsing library collections and discussions with other people are the most frequently
used modes of information searching (Kuhlthau 1993). General background sources such as
encyclopedias and review articles are mostly used at this stage (Vakkari 1999). After a focus
has been construed, the search for information becomes more directed. Thoughts about the
task become clearer and more structured. A clearer understanding guides the person to seek
relevant, focused information using the whole range of information resources. At the end
of the process rechecking searches are made for possible additional information (Kuhlthau
1991).
It is obvious that the role of IR systems and the information they provide varies depending
on the task and situation of the subject (Belkin et al. 1983, Checkland and Holwell 1998,
Ingwersen 1996, Jarvelin 1987, Vakkari 1999). The nature of an IR system is a function
providing information to support people taking purposeful action. In studying information
298 VAKKARI

searching the task performance process that generates searching should be as the point of
departure. The methodological implication is to analyze successive searches connecting
them to the information requirements of the task as perceived by the subjects.
In order to analyze how the task performance process is connected to information search-
ing, and information retrieval in particular, a study was made of how the stages of writing
a research proposal for a masters thesis were related to the information types searched,
to the search tactics and term choices, and to the relevance judgements of references and
full-texts. Also the success of search tactics and term choices was analyzed in terms of the
portion of the relevant references retrieved (precision).
From the angle of systems design the major limitation of the study is that it did not
include variables describing IR techniques. The choice was conscious. Our knowledge about
the search activities of the subjects during task performance is limited (Ingwersen 1996,
Robertson and Hancock-Beaulieau 1992). We rst have to understand their key elements
properly before it is reasonable to explore which IR techniques might be efcient tools
to support searching in various conditions generated by the task process. Based on our
empirical ndings a new study including variables representing IR techniques is already
under way.

3. Major concepts

Table 1 lists the basic concepts in Kuhlthaus (1993) and Vakkaris studies (2000a, b). In
the following we will briey introduce and comment on the concepts from the angle of IR.
The reader can nd more information about the denitions in the original studies.
The explanatory variable in both projects was stages in task performance. The term
used by Kuhlthau is stages in information search process. In our study the six original
categories in Kuhlthaus modelwere collapsed into three. Pre-focus stages included the steps
ofinitiation, selection andexploration.Focusformulation wasidentical inbothstudies.Post-
focus stages contained collection and presentation stages. The choice to use three categories
was methodological. It was improbable that we would have been able to differentiate all
the six phases in the four month period the proposal writing lasted. Thus, we decided to
concentrate on the stages before and after the focus formulation, which is the most crucial
phase in the process (Kuhlthau 1993, Bystrom and Jarvelin 1995).
In both studies (Vakkari 2000b, Kuhlthau 1993) the information searched for was cat-
egorized into three classes. In both general background information is information which
cannot be described in detail and which is used to frame the task at hand. It is information
about the topic in general without categorizations into subelds. This type of information
is used to explore the general topic. By specic information Kuhlthau (1993, p. 39) under-
stands information that relates or applies to the matter at hand and has a connection or ts
with the topic under investigation. In Vakkari (2000b) faceted background information is
information about the topic categorized into its broad sub-elds, but which has not been
expressed in detail. Pertinent information in Kuhlthau (1993, p. 39) refers to information
that has a more decisive and signicant relationship to a topic than relevance and is related to
personal information need. Specic information refers in Vakkari (2000b) to information,
which has been expressed in detail revealing the central variables of the task at hand. It
CHANGES IN SEARCH TACTICS 299

seems that Kuhlthau differentiates between topical relevance and situational relevance of
information in her denitions of the two latter categories (Saracevic 1975, Schamber 1994),
whereas Vakkari uses as his criterion the degree of the differentiation of information in
terms of number and specicity of dimensions it consists of. In addition to that distinction,
Vakkari utilizes another categorization of information in terms of its contribution to the task
(Vakkari 2000b).
Both Kuhlthau (1993) and Vakkari (2000b) differentiate source types between human
information sources and information systems. Both distinguish between several source
types within the major groups, the difference being that Vakkari (2000b) takes more types
of information systems into account.
Kuhlthaus (1993) study did not report direct relevance assessments by the subjects. She
included them in her denition of information types. Vakkaris (Vakkari 2000b, Vakkari
and Hakala 2000) studies contained relevance assessments of references and full-texts,
criteria for them and the contributing information in documents. Vakkari (2000a) also
studied how choice of search terms and tactics resulted in different rates of precision in
searches.
The categories of relevance criteria were created by combining the schema developed
by Barry (1994) with the categories derived from the data. The schema consisted of 26
categories. They were grouped into six major categories describing (1) the information
content of the document; (2) the sources of documents; (3) the document as a physical entity;
(4) the users situation; (5) the users experience and background; and (6) information types
(Vakkari and Hakala 2000).
For Vakkari mental model is used as an auxiliary concept. A synonym for mental model
is cognitive structure or conceptual structure. A mental model can be described as con-
sisting of concepts and their relationships. Differentation refers to the number of concepts
in the conceptual structure of a subject. Integration refers to the amount of interrelations
between the concepts (Vakkari 1999, Driver and Streufert 1969, Gavin 1998). Although our
data contain material about the changes in the understanding of the topic by the students,
it was not possible to infer meaningful mental models from it that could have been used
to explain aspects of information searching. Without empirical indicators we were forced
to use mental model as an auxiliary concept for explaining variation in search activities.
In this role the mental model of the subjects was connected to the stages of Kuhlthaus
model. It was supposed that in the beginning of the process the mental model is undiffer-
entiated (diffuse) and fragmentary. At the end of task performance the knowledge structure
is differentiated and integrated. Thus, we deduced the features of the mental model of the
students from the stages of the task performance process.
Search tactics in information systems in Kuhlthau (1993) can be roughly termed as
browsing and querying. Vakkari (2000a, b) developed a classication scheme including 12
search tactics. The major search tactics are explained in Table 2. Vakkari (2000a, b) also
included variables describing search terms and operators in his research design, whereas
these were left outside the scope of Kuhlthaus (1993) study.
The role of mental models during the task performance differs in the two studies.
Kuhlthau (1993, pp. 4351, 1991, p. 367) describes the thoughts concerning the task on
the trichotomies general-narrowed-focused and vague-clearer-clear. It seems that the
300 VAKKARI

Table 2.Denitions and operationalizations of major search tactics.

Denitionsa Operationalizationsb

Strategies to begin a session

Select: To break down complex search queries into At most two thirds of all search terms entered in
subproblems and work one problem at a time the beginning of the search
Exhaust: To include most or all elements of the More than two thirds of all search terms entered
query in the initial search formulation in the beginning of the search
Search formulation tactics
Intersect: Add conjunction representing Terms added to the query using an AND operator
another query facet
Vary: Replace one existing query term with another At least one new term was substituted for one of
the terms in the preceding move so that the
number of terms remained the same
Parallel: To make the query broader by introducing At least one synonym or conceptually parallel
synonyms or disjunctive terms to a facet term was added
Reduce: To subtract one or more of the query Set of terms was repeated, minus at least one term
elements from an already-prepared search
formulation
Negate: To eliminate unwanted elements by At least one AND NOT operation was used
using AND NOT operator
Other tactics
Limit: To limit the query by operational moves Use of eld restrictions provided by the system
a
Denitions are based on Bates (1990), Fidel (1991), Sormunen (2000) and Wildemuth et al. (1991).
b
Operationalizations are based on Sormunen (2000) and Wildemuth et al. (1991).

distinctions are used in her model more as predicates of the stages in the information search
process than to explain the types of information sought for or search strategies. However,
Kuhlthau uses these constructs in both ways, emphasizing the former.

4. Data and methods

The subjects were 11 students from the Department of Information Studies at the University
of Tampere who attended a seminar on preparing a research proposal for a masters thesis.
It took place over a period of four months during the spring term of 1999. At the beginning
of the seminar the students selected a topic and by the end they had come up with a research
proposal.
Data for describing their understanding of the task, problem stage, search tactics and
relevance assessments were collected in several ways during the process. Subjects were
asked to make a search in the LISA data-base (Dialog), which was running under a Boolean
retrieval system, three times during the seminar: when the seminar had just started, in the
middle, and when the students were nishing or had completed the proposal. The aim was to
acquire data at the pre-focus, focus formation and post-focus phases. A pre- and post-search
CHANGES IN SEARCH TACTICS 301

interview was conducted in each case. The pre-interview consisted of Kuhlthaus (Vakkari
1999)processsurveyquestionnaireandasemi-structuredinterview.Bothmeasuredfeelings,
thoughts and actions in the respective problem stages. The latter concentrated on measuring
participants state of knowledge and experience of the topic (including drawing a mind map)
and their goals and intended actions. Moreover, they were asked what kind of information
they were looking for and what they expected to do with the results in the present search.
The students were asked to think aloud during the search session. The transaction logs
were kept and the thinking aloud was recorded. The accepted sets of bibliographic records
were printed out and evaluated by the students in terms of relevance criteria and their
usefulness.
The students also kept a research diary recording their ideas, thoughts and feelings as well
as a structured search diary recording ideas sought, documents found, their sources, and their
relevance and contribution in terms of useful information. They entered in the diaries three
times, at the ends of each of the rst three months. The nished research proposals were also
collected. The reader will nd more detailed discussion about the operationalizations of the
concepts in each particular article (Vakkari 2000a, b, Vakkari and Hakala 2000, Vakkari
and Pennanen 2001).
The study subjects were graduates in information studies, who had participated in courses
in information retrieval including Boolean systems. These also included the use of LISA. It
is evident that these students mastered the use of a Boolean system. We can conclude that
the search expertise of the students was fairly advanced, and relatively constant, and thus
controlled in the research design. This implies that the variation in search tactics depended
on the growth in domain knowledge and not in the search expertise.

5. The main results of the empirical studies

Next we will briey present the main results of the study. We will begin with the ndings
concerning search terms and tactics, continue with the relevance experienced and useful-
ness of the documents and nish with the results about the sources used for obtaining the
documents.
Ingeneral, allthestudents proceededintheirtaskatvaryingpacesaccording toKuhlthaus
(1993) model. In the rst search session, the students were moving from topic selection to
exploration of the topic. In the middle of their task they were typically exploring the topic
and trying to formulate a research problem. By the end of the project they were logically in
the presentation stage, but while half of the students had been able to construct a focus, the
other half were still involved in constructing it. A detailed analysis of the students problem
stages can be found in Vakkari and Hakala (2000).

5.1. Search terms and tactics

Our study (Vakkari 2000a) demonstrated that the students problem stages during task
performance were systematically related to their choice of search terms and of some tactics.
The growing and focusing understanding of the task by the students led them to use more
302 VAKKARI

and more specied search terms, more and more varied operators as well as more versatile
tactics in the course of their project.

Terms. When selecting their topic, the students represented it with a few search terms. As
their mental models differentiated, the vocabulary in the searches grew in each round with
synonyms, narrower and related terms. Broader terms were dropped, especially in the last
round. Students were able to express the topic with a larger and more specic vocabulary in
the successive searches (Vakkari 2000a). This is consistent with the ndings by Wang (1997)
on researchers vocabulary changes generated by information needs at request, document
selection and post project stages.

Tactics. Only some tactics and patterns of tactics depended on the stage of the task per-
formance and subjects state of knowledge. Others were related to the size of the retrieved
set, which was independent of the phase of the process (Vakkari 2000a).
The students began the search either by introducing all the search terms or entering only
a fraction of the terms in the initial query. The former tactic was called Exhaust and the
latter Select. As the task performance proceeded, the students use of Exhaust decreased
and Select increased. Those who used Exhaust were not as far advanced in their process,
and their conceptual construct was less developed than those who started with Select. The
former group was able to represent only some, in the rst round only two, search terms.
The Exhaust tactics was typically followed by operational moves, limitation of language or
printing year, which did not require conceptual changes in the query (Fidel 1991, Vakkari
2000a).
In all stages of the project Select was followed by Intersect (combining terms by AND
operator)which was the most used tacticor Vary tactic depending on the size of the
retrieved set. If the set was large, Intersect was a typical move. If the set was small, Vary
was the preferred option. In Vary, the number of terms was similar to those of the preceding
move, but one term was substituted with a new one. It was typically used when the students
tried to expand a small set. An alternative for Vary was Reduce in enlarging the size of a set.
It consists of reducing the number of terms combined by AND operator in a query, and was
used more seldom. It was selected when the search reformulation was more complex than
on average. It was introduced after three tactical moves. Vary was used in earlier phases of
the query reformulation (Vakkari 2000a).
The students began to use more synonyms and parallel terms when their knowledge of
the topic increased. This was reected in the increasing use of Parallel tactics search by
search. It was the second most common tactic at the end of the process (Vakkari 2000a).
Those features in the searches yielding high precision (the share of relevant and partially
relevant references in the result set) were the use of more precise and parallel terms and their
combination by OR operators in facets producing Parallel tactics. A facet is an aspect of a
query. The mere use of AND operators (Intersect) with few facets containing only one term
each led to search results with a lower precision. The use of precise terms in a query results
naturally in a precise search result. The ndings also demonstrate that an active utilization
of synonyms and parallel terms for increasing recall seemed to lead to a higher precision
compared to other tactics like Intersect. The explanation for this is that queries consisting of
CHANGES IN SEARCH TACTICS 303

facets with several and typically specic terms have more discriminatory power than queries
consisting of an intersection of facets expressed with a single and perhaps vague term each,
given that the number of facets is the same. As a result, the former queries retrieve sets
containing proportionately more relevant references. The results also suggest that students
with a more structured domain knowledge tend to formulate queries with more facets (with
greater complexity) and with more specic terms (with greater specicity and coverage)
resulting in a higher share of relevant items (Vakkari 2000a).
This study has shown a systematic and many-sided connection between the students
growing domain knowledge and their IR interaction in successive search sessions. The
growth in domain knowledge seems to predict users IR interaction in successive search
sessions if the subjects search expertise is equal.

Supporting searching. We found that the degree of students knowledge of the topic
predicts their ability to express search terms and formulate tactics. The less they know, the
fewer, broader and more vague terms they use and the shorter queries and simpler tactics
they formulate. Brajnik et al. (1996) have shown that in more difcult problems, where
users may fail to achieve a good conceptualization, automatic query reformulation does not
help them to achieve a better search result. Fowkes and Beaulieu (2000) have shown that
the effectiveness of relevance feedback depends on the characteristics of the topics. They
found that interactive query expansion was more productive for more difcult and complex
topics as perceived by the searchers, whereas automatic query expansion was more efcient
in simple topics. All these results seem to suggest that automatic query expansion does
not work if the person has slight domain knowledge of a complex topic. He is not able to
conceptualize and articulate the information need exactly. It is evident that the person is
unable to express all the central facets of the topic due to his insufcient mental model.
He fails to bring into the query related terms expressing crucial aspects of the topic. Often
automatic query expansion generates terms which are parallel terms or synonyms within the
expressed facets. It does not bring new facets, i.e. related terms, to the query. The relative
inability of automatic query expansion to generate related terms seems to be the reason
for its inefciency in cases when the searchers domain knowledge is slight in a complex
topic. The obvious conclusion is that people with scarce domain knowledge need support in
expanding and differentiating their conceptual model of the topic (Ingwersen 1996, Bates
1990). A search thesaurus is a partial means for this. Such tools would help people to develop
ideas on how to structure the topic and how to express their vague information needs in
greater detail. Equipped with related terms, synonyms and narrower terms provided by the
system they could reformulate their query using terms with greater differentiation power.
This would enable more relevant information items to be found.
Our ndings support the latter claim. We found that queries containing facets with terms
combined by OR operators produced a higher share of relevant references than queries
containing facets represented by one term each and combined by AND operators. This result
suggests that, in addition to support in nding appropriate vocabulary, users also need help
in formulating the query, especially in using OR operators. Interfaces recommending users
to utilize Parallel tactics would support them in retrieving a higher proportion of relevant
information items.
304 VAKKARI

5.2. Relevance and contributing information in documents

The ndings in Vakkari and Hakala (2000) show that the students problem stage during
task performance is related to the use of relevance criteria in assessing retrieved references
and consequently documents.
As the students moved from the beginning of the process to its middle the share of
relevant references decreased, the share of partially relevant references remained constant
and the share of irrelevant references increased. There was a differentiation between clearly
relevant and other references. The differentiation remained on the same level as the students
progressed to the nal stage of their projects. The results suggest that when students have
chosen their research topic and are exploring it, they become knowledgeable and their
mental constructs become differentiated, which enables them to distinguish more clearly
between highly relevant and other references.
The changes in relevance criteria of references remained relatively modest throughout
the task performance. The biggest change in a criterion between the stages of the task was
10% units, the next ones being at the level of 5% units. These ndings resemble those by
Bateman (1998). The relevance criteria for assessing the full texts changed more than those
for judging the references in the course of the process. The importance of information types
(theoretical, methodological and empirical) increased compared to topicality, especially in
the middle of the process. It is evident that the relevance of the documents is easier to assess
on the basis of full text than its surrogates (Harter 1992).
The major groups of the relevance criteria expressed by the students for assessing ref-
erences changed slightly during the process. Information content was clearly the major
predictor of the relevance of the references in each stage of the project (cf. Schamber 1994).
The most crucial factor in this group was topicality. The information type of the document
indicated by the reference mattered in the beginning and middle stages of the task, but its
signicance diminished towards the end. References to general background and theoretical
information especially were chosen most from the initiation to the focus formulation stages,
but to a lesser extent at the collection and presentation stages. At those stages, references
to methodological sources were accepted.
The results suggest that the subjects were not searching for documents on the topic in
general, but on particular aspects of the topic reecting the stage of the task performance.
The variation of the importance of information types in the course of the process is an
indication of this. General background and theoretical information were judged as relevant
in the beginning of the process. At that stage the subjects were searching for information
giving an overall picture of the topic and providing with various conceptualizations of it in
pursuit of a structured and focused problem. The relevance of these types of information
diminished when the focus was constructed. It seems plausible that individuals actually
mean different things when they speak about topical information. They are seeking for
information that would contribute to their understanding in performing the task.

5.3. Aspects of topicality

The ndings reported in Vakkari (2000a, b) conrm that the topicality of documents refers
to different types of information depending on the stage of the task (cf. Saracevic 1975).
CHANGES IN SEARCH TACTICS 305

The students were asked to describe the information sought for and the contribution of the
documents they had obtained in their search diaries. The categorization for the contributing
information types was created by combining categories emerging from the data with ideas
by Kuhlthau (1993) and a classication of citation types by Suhonen and Rautio (1981).
The categories were background information, theories and models, methods, cases, facts,
empirical results and focused information.
The results show that the students progressively more structured mental model differ-
entiated their search for information in terms of the specicity of information, and the
types of contribution they construed from the information in documents. At the beginning
their representation of the task was undifferentiated. It led them to look mostly for general
background information. The contribution of the documents found consisted typically of
background information about the topic as well as models and conceptualizations of it.
Most of the texts obtained were assessed as being partially relevant, because the students
were unable to identify clearly what was not relevant at this stage (Vakkari 2000b).
In the focus formation stage the students frequently searched for faceted background
information. The documents obtained typically supported the process by providing still
more background information and theories, but to a lesser extent. Texts with method-
ological advice or information about cases had gained more footing. Towards the end
of the process the students were searching mainly for specic information. They used
the documents obtained to gain focused information and empirical research results
(Vakkari 2000b).
The results suggest that it is possible to construct tentative categories for contributing
information types at least for the domain studied. It seems that these categories can be
understood to a great extent as being facets of topicality if topicality is what the document
is about as the user sees it with respect to the task at hand (Schamber 1994). The results
also suggest that it is not only the topicality as such, but these aspects of topicality that are
decisive when the subjects judge the relevance and usefulness of the documents (Vakkari and
Hakala 2000). The results conrm that what is felt to be contributive information depends
on the mental model of the subject, and consequently, on the phase of the task performance
process (Vakkari 1999).

5.4. Channels of documents

Vakkari and Pennanen (2001) showed that there is a systematic connection between the stu-
dents problem stages while writing a research proposal and pointers (means of identifying
documents) to useful literature as well as sources for obtaining the documents.
We found that in the pre-focus stages the students were mostly searching for background
information and theories and models for framing their research proposal. At this stage their
means for identifying useful documents were concentrated on three ways of searching:
They sought literature most frequently by using TAMCATthe OPAC of their university
libraryby consulting supervisors and by relying on their own prior knowledge (Vakkari
and Pennanen 2001).
At the focus construction phase the students still mostly sought for background informa-
tion and models, but methods and focused information were gaining in salience. The range
306 VAKKARI

of pointers used at this phase was more comprehensive than in the beginning. They found
the references most commonly by consulting TAMCAT, supervisors or other OPACs.
In the nal phase the students clearly sought most specic information, but methods and
empirical research results were also useful. These types of information were identied by
means of TAMCAT, supervisors, and chaining. The role of TAMCAT and other OPACs
was especially crucial in nding specic information. The role of supervisors for hinting
at specic information was surprisingly minor. However, at the post-focus phase the range
of pointers and information resources used was at its most comprehensive compared to
previous stages. This nding is consistent with Kuhlthaus (1993) model.
Although the documents found in LISA searches made a very modest contribution to the
writing of the research proposals, other information systems, mainly the OPACs used, had
a crucial role in identifying contributory information sources throughout the whole process
and especially in the post-focus stages. Thus, information systems have a signicant role
in the whole information seeking process of students preparing their research proposals.
All in all, we have been able to demonstrate that the stages of Kuhlthaus (1993) model
and thus, the increasing differentiation of the subjects mental model has a systematic impact
on the information types sought, on the choice of search terms and tactics as well as on the
assessment of relevance and contribution of the references found and full-texts acquired in
the task performance process.

6. A Theory of the task-based IR process

The core empirical ndings divided according to the central factors to be explained from the
study are presented in Table 3. Based on these supported hypotheses we can outline a theory
of the information search process in task performance (gure 1). This is an idealization of
the ndings summed up in Table 3. The theory systematically supports Kuhlthaus (1993)
model of the information search process and expands it in the eld of information retrieval.
The point of departure in the search for information is the task at hand. The subject has
a mental model of the task, which is reected in the stage of the task performance process.
The subjects ability to express what kind of information is needed for the task depends on
the differentiation and integration of the model and thus, on the stage of the task. This also
determines his ability to articulate the anticipated contribution of the information sought.
The choice of search tactics (including the combination of search terms and operators) is
affected by the subjects ability to articulate the information need and transform it into
search queries. The queries retrieve sets of accepted references, which will be selected
based on certain relevance criteria. The references will be judged in terms of their relevance
and usefulness for the task. The success of the chosen search tactics can be assessed by
the precision and recall of the searches. The chosen references will guide the subject to
select some of the documents for use in the task. This use is reected in the utilization of
the information in the documents in task performance. The theory predicts that information
types sought for, search tactics and term choices for identifying the documents as well as
patterning of relevance assessments and consequently, the success of searches in terms of
the relevance of the documents found depend on the stage of the task performance and the
mental model of the searcher.
CHANGES IN SEARCH TACTICS 307

Table 3. The central hypotheses supported empirically.

As the task performance process proceeds,

Types of information sought

1. The share of general (background) information declines and the share of specic
information grows
2. The share of faceted background information increases in the middle and decreases in the
end of the process (curvilinear relation)
Contributing information types
3. The use of background information and theoretical information declines especially in the
post-focus stage, whereas the utilisation of documents containing methods, empirical
results and focused information grows
Degree of relevance
4. The share of relevant references retrieved decreases, the share of partially relevant
references remains constant and the share of non-relevant references increases from the
beginning to the middle of the process. The difference between relevant and other
references (partially relevant and non-relevant) remains at the same level from the middle
to the nal stage of the project.
Relevance criteria
5. The proportion of information content, especially topicality, as a relevance criterion
remains constant
6. The signicance of the users experience and preferences grows and the importance of
information types declines as predictors of relevance
Search terms
7. The number of search terms used increases
8. The number of synonyms, narrower terms and especially related terms increases and the
number of broader terms decreases.
The subjects are able to express the topic with a larger and more specic vocabulary in the
successive searches
Operators
9. The number of AND and OR operators used increases
10. The proportion of AND operators declines and OR operators grows
The number and variety of operators used increases
Search tactics
11. The number of search tactics used increases
12. Among search formulation tactics the number of Intersect, Vary and Parallel tactics increases
13. The portion of Intersect tactics diminishes and that of Parallel tactics increases resulting in
a higher precision of search results
14. The frequency of monitoring moves and browsing of search results grows
15. The share of conceptual search tactics increases and that of operational tactics diminishes

The theory does not yet include factors indicating the subjects IR knowledge or tools
for supporting searching such as terminological support or query formulation aid. This is
a limitation. We can expect that peoples knowledge of and experience in IR systems have
an impact on their search behaviour. Thus, it is necessary to include this dimension in the
308 VAKKARI

Figure 1. A theory of the task-based IR process.

theory. On the other hand, if studies on information searching in natural settings want to
inform systems design they have to include in their research design variables describing IR
techniques like search support tools. (These new relations of the theory are represented by
dashed arrows in gure 1.) By analyzing dependencies between various search behaviours
and the use of support tools, the research is able to create results that might help in designing
IR systems which better match the search behaviour of subjects performing their tasks.
The theory is naturally very tentative because of the restricted number of cases in the
studies that were used for generating it. However, the results from our studies on different
CHANGES IN SEARCH TACTICS 309

aspects of information searching are consistent with each other. They are also in line with
the ndings of cognitive psychology on domain expertise, reasoning and concept learning
(Heit 1997, Patel and Ramoni 1997). Moreover, Kuhlthaus (1993) model systematically
supports our ndings. It is not only empirical evidence from our studies that justies the
theory, but its congruence with more general theories of information seeking and domain
expertise (Suppe 1989, pp. 103169). Moreover, the theory could be transferable from
preparing a masters thesis to the whole domain of research. The theory naturally calls for
both more empirical testing and conceptual renement. Our next step in elaborating the
theory has been to collect fresh empirical data with a great number of research subjects to
test it, and to include variables describing IR experience and IR techniques in the research
design.
Our results demonstrate clearly that the information sought, search tactics, term choices
and relevance judgements as well as the contributory types of information in documents
depend systematically on the stage of the task performance process and the mental model of
the searcher. The obvious conclusion is that it is productive to study information searching
and information retrieval in particular, as a process in connection with the task that generates
it. By understanding the task of the searchers we will be able to obtain research results which
will also provide useful information for designing information systems.

References

Barry C (1994) User-dened relevance criteria: An exploratory study. Journal of the American Society for Infor-
mation Science, 45(3):149159.
Bateman J (1998) Changes in relevance criteria: A longitudinal study. In: Proceedings of the 61st American Society
for Information Science Annual Meeting 35, pp. 2332.
Bates M (1990) Where should the person stop and information interface start? Information Processing & Man-
agement, 26(5):575591.
Belkin N (1993) Interaction with texts: Information retrieval as information seeking behavior. In: Information
Retrieval93, Universitetsverlag Konstanz, Konstanz.
Belkin N, Seeger R and Wersig G (1983) Distributed expert problem treatment as a model for information system
analysis and design. Journal of Information Science, 5(5):153167.
Brajnik G and Mizzarro S and Tasso C (1996) Evaluating user interfaces to information retrieval systems. A case
study. In: Proceedings of the SIGIR96, Zurich, 1996.
Bystrom K and Jarvelin K (1995) Task complexity affects information seeking and use. Information Processing
& Management, 31(2):191213.
Checkland P and Holwell S (1998) Information, Systems and Information Systems. John Wiley, Chichester/New
York.
Driver M and Streufert S (1969) Integrative complexity: An approach to individuals and groups as information
processing systems. Administrative Science Quarterly, 14:272285.
Ellis D and Haugan M (1997) Modeling the information seeking patterns of engineers and research scientists in
an industrial environment. Journal of Documentation, 53(4):384403.
Fidel R (1991) Searchers selection of search keys: III searching styles. JASIS, 42(7):515527.
Fowkes H and Beaulieu M (2000) Interactive searching behavior: Okapi experiment for TREC-8. In: Proceedings
of the BCS-IRSG: 22nd Annual Colloquium on Information Retrieval Research, Cambridge.
Gavin H (1998) The Essence of Cognitive Psychology. Prentice Hall Europe, London.
Harter S (1992) Psychological relevance and information science. Journal of the American Society for Information
Science, 43(9):602615.
Heit E (1997) Knowledge and concept learning. In: Lamberts K and Shanks D, Eds., Knowledge, Concepts and
Categories. Psychology Press, Hove, pp. 741.
310 VAKKARI

Ingwersen P (1996) Cognitive perspectives of IR interaction: Elements of a cognitive IR theory. Journal of

Documentation, 51(1):350.
Jarvelin K (1987) On information, information technology and the development of society: An information science
perspective. In: Ingwersen P, Kajberg L and Mark Pejtersen A, Eds., Information Technology and Information
Use. Graham Taylor, London, pp. 3555.
Kuhlthau C (1991) Inside the search process: Information seeking from the users perspective. JASIS, 42(5):361
371.
Kuhlthau C (1993) Seeking Meaning. Ablex, Norwood, NJ.
Marchionini G (1995) Information Seeking in Electronic Environments. Cambridge University Press, Cambridge,
UK.
Patel V and Ramoni M (1997) Cognitive models of directional inference in expert medical reasoning. In: Feltovich
P, Ford K and Hoffman R, Eds., Expertise in Context: Human and Machine. AAAI Press, Menlo Park, CA,
pp. 6799.
Robertson S and Hancock-Beaulieau M (1992) On the evaluation of IR systems. Information Processing &
Management, 28(4):457466.
Saracevic T (1975) Relevance: A review of the literature and a framework for thinking on the notion in information
science. JASIS, 26:321343.
Schamber L (1994) Relevance and information behavior. In: Williams M, Eds., Annual Review of Information
Science and Technology 29. Information Today, Medford, NJ, pp. 348.
Sormunen E (2000) A method of measuring wide range performance of Boolean queries in full-text databases.
Acta Universitatis Tamperensis 748. Doctoral Diss., Tampere University Press, Tampere.
Spink A, Greisdorf R and Bateman J (1998) From highly relevant to non-relevant: Examining different regions of
relevance. Information Processing & Management, 34(5):599622.
Spink A, Wilson T, Ellis D and Ford N (1998) Modeling users successive search in digital environments. D-Lib
Magazine, March 1998.
Suhonen P and Rautio P (1981) Yhteiskuntatieteiden tietovirrat ja suomalainen tutkija (Information Flows in
Social Sciences and the Finnish Researcher). Academy of Finland. Publications 2/1981.
Suppe F (1989) The semantic conception of theories and scientic realism. University of Illinois Press,
Urbana/Chicago.
Sutton SA (1994) The role of attorney mental models of law in case relevance determinations: An exploratory
analysis. JASIS, 45(3):186200.
Vakkari P (1999) Task complexity, problem structure and information actions. Integrating studies on information
seeking and retrieval. Information Processing & Management, 35(6):819837.
Vakkari P (2000a) Cognition and changes of search terms and tactics during task performance: A longitudinal
study. In: Proceedings of the RIAO 2000 Conference, C.I.D, Paris, 2000. pp. 894907.
Vakkari P (2000b) Relevance and contributory information types of searched documents in task performance. In:
Proceedings of SIGIR2000 Conference, Athens 2000. ACM, New York, pp. 29.
Vakkari P and Hakala N (2000) Changes in relevance criteria and problem stages in task performance. Journal of
Documentation, 56(5):540562.
Vakkari P and Pennanen M (2001) Sources, relevance and contributory information of documents in writing a
research proposal. A longitudinal case study. In: Proceedings of the ISIC2000 Conference, to be published.
Wang P (1997) Users information needs at different stages of a research project: A cognitive view. In: Vakkari
P, Savolainen R and Dervin B, Eds., Information Seeking in Context. Graham Taylor, London/Los Angeles,
pp. 307318.
Wildemuth B, Jacob E, Fullington A, de Blieck R and Friedman C (1991) A detailed analysis of end-user search
behaviors. In: Proc. of the 54 ASIS Annual Meeting, pp. 302312.
Wilson T (1998) Exploring models of information behaviour: The uncertainty project. In: Wilson T and Allen
D, Eds., Exploring the Contexts of Information Behaviour. Graham Taylor, London/Los Angeles, pp. 5566.
Yang S (1997) Information seeking as problem-solving using a qualitative approach to unfover the novice learners
information-seeking process in a perseus hypertext system. Library & Information Science Research, 19(1):71
92.