Automating the Assignment of
Submitted Manuscripts to Reviewers
Susan T. Dumais and Jakob Nielsen
Bellcore
445 South Street
Morristown, NJ 07960
USA
Email: std@bellcore.com and nielsen@bellcore.com
Abstract
The 117 manuscripts submitted for the Hypertext’91 confer-
ence were assigned to members of the review committee,
using a variety of automated methods based on information
retrieval principles and Latent Semantic Indexing. Fifteen
reviewers provided exhaustive ratings for the submitted
abstracts, indicating how well each abstract matched their
interests. The automated methods do a fairly good job of
assigning relevant papers for review, but they are still some-
what poorer than assignments made manually by human
experts and substantially poorer than an assignment per-
fectly matching the reviewers’ own ranking of the papers. A
new automated assignment method called “n of 2 n ”
achieves better performance than human experts by sending
reviewers more papers than they actually have to review
and then allowing them to choose part of their review load
themselves.
Keywords: Conferences, Program Committees, Review-
ers, Referees, Manuscripts, Papers, Assignment, Matching,
Interests, Latent Semantic Indexing, LSI, Hypertext, Infor-
mation Retrieval.
Introduction
The assignment of submitted manuscripts to reviewers is a
common task in the scientific community and is an impor-
tant part of the duties of journal editors, conference pro-
gram chairs, and research councils. Finding reviewers for
journal submissions and some types of grant proposals can
normally be done for a small number of submissions at a
Reprinted from:
Proceedings of the ACM SIGIR’92 15th International Confer-
ence on Research and Development in Information Retrieval
(Copenhagen, Denmark, 21–24 June 1992), 233–244.
 1992 by the ACM, reprinted by permission.
0-89791-524-0/92/0006/0233...$1.50
Automating the Assignment of Submitted Manuscripts to Reviewers
time and at a more or less leisurely pace. For conference
submissions and other forms of grant proposals (e.g., NSF,
Esprit), however, the reviews and review assignments
must be completed under severe time pressure, with a very
large number of submissions arriving near the announced
deadline, making it difficult to plan the review assignments
much in advance.
These dual problems of large volume and limited time
make the assignment of submitted manuscripts to review-
ers a complicated job that has traditionally been handled
by a single person (or at most a few people) under quite
stressful conditions. Also, manual review assignment is
only possible if the person doing the assignments (typi-
cally the program chair for the conference) knows all the
members of the review committee and their respective
areas of expertise. As some conferences grow in scope
with respect to number of submissions and reviewers as
well as the number of sub-domains of their fields, it would
be desirable to develop automated means of assigning the
submitted manuscripts to appropriate members of the
review committee.*
We tried using one such automated information retrieval
method to help the program chair for the ACM Hyper-
text’91 conference (San Antonio, TX, 15–18 December
1991) in assigning the 117 submitted manuscripts to 25
members of the program committee. Before the submis-
* A related issue (just seen from the reverse perspective) is the
problem of matching funding agencies to research projects
[Cohen and Kjeldsen 1987]. In general, the principles
described in this paper should be relevant whenever one has
two large sets of objects that need to be matched such that each
object in one set gets assigned a small number of objects from
the other. Other examples include the assignment of press
releases to newspaper reporters and summer student job appli-
cations to researchers with job openings.
233
sion deadline, the reviewers were asked to send us
abstracts of their papers to be used as descriptions of their
sub-areas of interest and expertise within the hypertext
field. After the deadline, the abstracts of the submitted
papers were scanned into a computer and subjected to opti-
cal character recognition to generate machine-readable
files.*
The actual application of assigning manuscripts to review-
ers involves two further considerations in addition to the
matching of manuscripts and reviewers. First, one needs to
guard against conflicts of interest by not assigning any
reviewers their own papers or those of close colleagues.
Second, there is a need to balance the review assignments
to ensure that no single reviewer is overworked just
because that person happens to be an appropriate choice
for many papers, and that each paper gets assigned a cer-
tain minimum number of reviewers. All these constraints
can be expressed as linear inequalities that can be handled
by a linear programming package, so the entire review
assignment can in principle be done automatically. For the
Hypertext’91 conference, however, these two final steps
were done manually.
Automated matching requires four elements, each of
which is discussed further below:
1. A representation of the domain is required by some
methods, including ours. We used a variety of written
materials about hypertext that were available in
machine-readable form.
2. A representation of the submitted papers. Here, we
used their titles and abstracts.
3. A representation of the reviewers’ expertise. We used
abstracts supplied by the reviewers. Mostly, these were
abstracts of papers written by the reviewers for previ-
ous conferences or for journals, but some reviewers
also sent other abstracts describing their interests.
4. A concrete algorithm to compute the similarity between
submissions and reviewers on the basis of the above
three representations. We used Latent Semantic Index-
ing (LSI) [Deerwester et al. 1990], but other informa-
tion retrieval methods such as vector or probabilistic
retrieval could also have been used.
The LSI Method
Latent Semantic Indexing (LSI) is an extension of the vec-
tor retrieval method (e.g., [Salton and McGill 1983]) in
which the associations between terms are explicitly taken
into account. This is done by modeling the association of
terms and documents. LSI assumes that there is some
underlying or “latent” structure in the pattern of word
* The OCR files contained errors even after they had been
cleaned up by humans in several independent passes. For
future use of automated review assignment, we recommend
having the authors provide machine-readable files directly by
email or diskette if at all possible.
234
usage across documents, and uses statistical techniques to
estimate this latent structure. A description of terms, docu-
ments, and queries based on the underlying latent semantic
structure (rather than surface level word choice) is used for
representing and retrieving information. See [Deerwester
et al. 1990; Furnas et al. 1988] for mathematical details
and examples.
The Latent Semantic Indexing (LSI) analysis described by
Deerwester et al. [1990] uses singular-value decomposi-
tion (SVD), a technique closely related to eigenvector
decomposition and factor analysis [Cullum and Wil-
loughby 1985]. The term-by-document matrix is decom-
posed into a set of k , typically 100 to 300, orthogonal
factors from which the original matrix can be approxi-
mated by linear combination. Instead of representing docu-
ments and queries directly as sets of independent words,
LSI represents them as continuous values on each of the k
orthogonal indexing dimensions. Since the number of fac-
tors or dimensions is much smaller than the number of
unique terms, words will not be independent. For example,
if two terms are used in similar contexts (documents), they
will have similar vectors in the LSI representation. Simi-
larly, a word can be near a document that does not contain
it if that is consistent with the overall pattern of word
usage in the collection. The LSI technique captures deeper
structure than simple term-term correlations or clustering
and is completely automatic.
One can interpret the analysis performed by SVD geomet-
rically. The result of the SVD is a k-dimensional vector
space in which each term and document is located. These
k-dimensional term and document vectors are simply the
left and right singular vectors of the SVD analysis. In this
space the cosine or dot product between vectors corre-
sponds to their estimated similarity. Since both term and
object vectors are represented in the LSI space, similarities
between any combination of terms and objects can be eas-
ily obtained. Retrieval proceeds by using the terms in a
query to identify a point in the space, and all documents
are then ranked by their similarity to this query vector.
The LSI method has been applied to several standard infor-
mation retrieval collections with favorable results. The LSI
method has equaled or outperformed standard vector meth-
ods and other variants in every case, and was as much as
30% better in some cases. As with standard vector meth-
ods, differential term weighting and relevance feedback
can improve LSI performance substantially [Dumais
1991]. For the experiments reported in this paper, we used
an entropy term weighting which is known to produce
good retrieval performance [Salton and McGill 1983].
Assessing the Review Assignments
In order to assess the goodness of fit between the auto-
mated assignments and the reviewers’ true interests, we
asked the members of the Hypertext’91 papers committee
to estimate the degree to which they would be good
reviewers for each of the 117 submitted papers. All 25
Dumais and Nielsen
members of the papers committee were given a question-
naire reprinting the abstracts for each of the 117 papers,
and 15 members returned the questionnaire (60% response
rate).
Each reviewer rated the relevance of each paper based on
its abstract, using a 0–9 scale with 9 indicating the best
match between the paper and the reviewer’s competence
and interests. The anchor points for the scale were:
8–9 = right up my alley
6–7 = good match
4–5 = somewhat relevant
2–3 = I’m following it, sort of
0–1 = how did I get this one?
We should note that relevance judgments are context
dependent. Studies of fisheye interfaces [Furnas 1986]
found that academicians make much bigger distinctions
between disciplines close to their own: e.g., an experimen-
tal psychologist will say that Psychiatry and Psychology
are much more different than Management and Marketing,
whereas someone in the business school would say the
reverse [Furnas 1991]. It is thus likely that the hypertext
specialists on the Hypertext’91 review committee made
much finer distinctions between various topics for papers
on hypertext than an “objective” outsider might have
made. For some other conferences, any of the papers sub-
mitted for Hypertext’91 might just have been classified
under the general topic “hypertext,” and any of the Hyper-
text’91 reviewers would have been considered among the
best qualified to review them.
The reported results are based only on the 15 reviewers for
whom a complete list of relevance judgements was avail-
able. The mean relevance rating across all papers and
reviewers is 4.7.
These exhaustive relevance judgments serve as a bench-
mark against which performance is evaluated. The 117
submitted abstracts form the database of “documents.” The
self-descriptive abstracts given to us by the 15 reviewers
are the “queries” used by the automatic information
retrieval methods. We use Latent Semantic Indexing (LSI)
to compare each reviewer’s abstract with all the submis-
sions, and rank the submissions from most to least similar
to each reviewer. These results are then compared with the
reviewers’ relevance judgments. Although we use LSI to
compare reviewers and submitted abstracts, this is a
straightforward matching problem and any other informa-
tion retrieval method could have been used.
It is interesting to note that the “queries” used by the auto-
matic retrieval method are somewhat different than those
used by the reviewers in making their relevance judg-
ments. While it is always the case that a judge’s internal
representation of the query is not what is literally available
to the automatic methods, this is especially true in the
present application, making it a difficult task for the auto-
Automating the Assignment of Submitted Manuscripts to Reviewers
mated methods. Before the submission deadline, the
reviewers sent us abstracts which described their interests
and areas of expertise. After the program committee meet-
ing, the reviewers were given a questionnaire asking them
to rate the relevance of each of the submissions to their
interests and expertise. In many cases, reviewers rated
quite highly some abstracts about topics not covered in
their self-descriptions. Often these represented cases
where the reviewer simply failed to give us an abstract
about an area of interest. We could have asked our review-
ers to match their self-descriptive abstracts against the sub-
missions (as the automatic methods do), but this is a
difficult and unrealistic task. It also points to the important
problem of getting reviewers (and, more generally, users of
information retrieval systems) to specify appropriate que-
ries.
In addition to the practical problem of solving the reviewer
assignment problem, this case study will be used to
explore a number of issues of general interest to the infor-
mation retrieval community. We compare several different
performance measures, methods for representing the
domain, and methods for representing the reviewers' inter-
ests.
For use in calculating the precision of the review assign-
ments and the number of relevant articles in the top- n
assigned, papers rated six or higher (“good match” and
“right up my alley”) were considered “relevant” and
papers rated five or lower were considered “not relevant”
for a particular reviewer. The mean number of relevant
papers across all reviewers is 47 or 40% of the total. This
is a fairly high proportion of relevant material compared to
many other information retrieval applications and may
reflect the specialized nature of the Hypertext’91 confer-
ence.*
Four main measures of performance were used as
described below and shown in Table 2. Some measures
such as precision are commonly used in information
retrieval research; other measures like mean rated rele-
vance are appropriate for this particular application.
Mostly we report the results of assigning ten papers per
reviewer, as that seems a common review load. The theo-
retical maximum for each measure is noted in parentheses
and is achieved by using the reviewers’ own ratings to
rank the submitted articles.
* We have also examined different cutoff values for “rele-
vance.” When only papers receiving ratings of 8 or 9 are con-
sidered as relevant, there are obviously many fewer relevant
papers (an average of 15 per reviewer or 12%), but the same
general pattern of results are observed as those reported below.
The relative advantages of the automatic methods compared
with random assignment are much larger when this more
restrictive definition of “relevance” is used.
235
• Number of Relevant Articles in Top 10. This is the num-
ber of articles rated six or higher by the reviewers in the
top ten articles assigned to them by the automated
method. (For the ratings given by our reviewers, it
would in theory be possible to have an average of 10.0
relevant papers in the top-ten.)
• Precision. This is the mean precision (proportion of rel-
evant papers) calculated at the 25%, 50%, and 75% lev-
els of recall. (Ideally, this number should be 1.00,
indicating that no irrelevant papers have been ranked
higher than any relevant paper.)
• Mean Relevance of Top 10. This measure indicates the
mean rated relevance for the first ten papers that were
assigned to each reviewer by our automated method.
(For the relevance ratings actually given by the 15
reviewers, the average top-ten rating is 8.20. That is, if
submitted abstracts were ranked by the reviewers’ own
ratings, an average rating of 8.20 would have been
obtained for the first ten papers assigned to each
reviewer.)
• Mean Relevance of Top 5 & 5 More Picked from Top
20. This measure is intended to simulate an alternative
review assignment method where the reviewers are sent
a larger number of papers than they actually must
review. To ensure that each paper is reviewed by a min-
imum number of reviewers, the automated assignment
method would still assign some specific papers for
review, but the reviewers would be free to fill the rest of
their review responsibilities by choosing an appropriate
number of additional papers from the papers they have
been sent. Here, we have simulated sending each
reviewer twenty papers, with five review assignments
being pre-specified and each reviewer selecting five
additional papers for review from the remaining fifteen.
(The theoretical maximum value of this measure is the
same as for the mean top-ten rating, since the two
methods would assign the same papers if a perfect
review assignment method were used.)
It is important to note that we would observe the theoreti-
cal maximum for each measure of performance only if
reviewers were perfectly reliable and consistent in their
relevance ratings. We know that this is generally not the
case, so the maximum values represent an upper bound on
performance.
Representing the Domain
We were fortunate to have several textual databases about
the domain of the conference (hypertext) available to us in
machine readable form. This enabled us to examine the
extent to which a richer representation of relevant domain
knowledge could be used to improve the assignment of
submitted papers to reviewers.
Different information retrieval methods depend on domain
knowledge to different degrees. At one extreme, standard
vector matching methods, for example, generally ignore
available domain knowledge and simply match submitted
abstracts against reviewers. However, vector methods
236
could also use existing domain knowledge to set more
appropriate term weights—e.g., a large corpus of relevant
material (as compared with a relatively small number of
submitted papers) could be used to more reliably establish
appropriate term weights. At the other extreme, many AI-
based methods depend very heavily on more conceptually-
based representations of domain knowledge, and more
comprehensive coverage of the domain would almost cer-
tainly improve retrieval in such systems.
The LSI method falls somewhere between these two
extremes. The SVD analysis is based on the term by docu-
ment matrix. At a minimum, different domain representa-
tions will result in different term weightings. In addition,
the full term by document matrix will influence the orthog-
onal factors obtained from the SVD analysis. The impor-
tant associative relations that the method extracts will
depend on the pattern of term usage across documents. The
nature and extent of this influence on the derived factors
and retrieval performance is an empirical issue. In the
experiments described in this paper, we use several differ-
ent collections of materials about hypertext to automati-
cally generate different term by document matrices. These
matrices are automatically decomposed using SVD and the
resulting k-dimensional representations used for matching.
Available Datasets
Table 1 summarizes some of the major characteristics of
the databases used in our experiments. As a baseline, we
used the 117 abstracts submitted to the conference. We
also analyzed these submitted abstracts combined with the
83 abstracts that our reviewers used in their self-descrip-
tions.
Of more interest are analyses of the “pre-existing” data-
bases about hypertext. These collections represent a cur-
rent and comprehensive coverage of the hypertext field and
might provide a better modeling of term dependence than
the conference abstracts alone. The LSI analyses of these
databases do not take very long (about 1–2 hours on a
workstation), but could be performed well before they are
needed for the assignment task.
We obtained the full text of three books on hypertext—
Nielsen’s Hypertext and Hypermedia, Rada’s HYPER-
TEXT: From Text to Expertext, and Shneiderman & Kears-
ley’s Hypertext Hands-On! In addition, we analyzed the
union of these three books as a separate dataset. Each para-
graph in the Nielsen book was treated as a separate “docu-
ment” or unit of analysis. For the Rada and Shneiderman
& Kearsley books, we used the hypertext nodes (as defined
by the authors for the online versions of their books) as the
unit of analysis. We did not make any explicit use of the
hypertext links.
We also obtained the ACM compendium on hypertext
[Akscyn 1992]. This consists of a collection of articles
about hypertext. In one analysis we used the full text of the
128 available articles, treating each article as a unit of
Dumais and Nielsen
 Number of Number of
Size Units Terms in
Dataset Name Dataset Contents
(Kbytes) (“Docu- More Than
ments”) One Unit

Abstracts of all the 117 papers submitted for the

Submitted Abstracts 137 117 1,180
Hypertext’91 conference

Abstracts of all the 117 submitted papers combined

Submitted and Review-
with 83 abstracts of papers written by the 25 208 200 1,661
ers’ Abstracts
reviewers

Full text of the book Hypertext and Hypermedia

Nielsen 655 1,249 3,337
[Nielsen 1990], split by paragraphs

Full text of the book HYPERTEXT: From Text to

Rada 517 775 3,274
Expertext [Rada 1991], split by nodes

Full text of the hypertext version of the book Hyper-

Shneiderman &
text Hands-On! [Shneiderman and Kearsley 1989], 299 252 2,426
Kearsley
split by nodes

The Nielsen, Rada, and Shneiderman & Kearsley

Three books 1,472 2,276 6,008
datasets combined

Full text of The ACM Hypertext Compendium

ACM Articles [Akscyn 1992], split by entire articles as written by 4,687 128 9,714
the original authors

Full text of The ACM Hypertext Compendium, split

ACM Frames 4,841 4,511 12,489
by hypertext nodes as defined by the editors

Bibliographic database of the human–computer

Perlman Bibliography interaction and hypertext literature, mostly including 1,147 1,725 5,267
abstracts

The combination of the Nielsen, Rada, Shneider-

All Pre-Existing
man & Kearsley, ACM Frames, and Perlman 7,460 8,629 15,625
Datasets (union)
datasets

Table 1 Summary of the datasets used to build the domain representation of the hypertext field.

analysis. These articles contain 4,511 hypertext nodes in
the form of frames. Another analysis treated the 4,511
hypertext frames as “documents.” Note that the size
(Kbytes) of the ACM dataset is slightly larger when it is
split up by hypertext nodes. This is because we repeated
the title of the parent article in each of its nodes. This repe-
tition ensures that nodes from the same article will have
some term overlap and thus a somewhat higher similarity.
In principle, we could have taken advantage of the hyper-
text links in a similar manner to increase the surface-level
similarity for connected nodes in different articles.
Perlman’s HCI bibliography [Perlman 1991] is a more
general bibliographic database of articles about Human–
Computer Interaction, many of which are directly about or
relevant to the field of hypertext. This database contains
the titles, authors and usually the full text of the abstract
for papers in relevant journals, conferences, and edited
books. For many entries, additional fields are used to indi-
cate journal or conference name, publisher, year, etc. On
Nov. 26, 1991 this set contained 1,725 entries.
A final analysis used the union of all pre-existing datasets
(the three books, ACM Frames, and Perlman's bibliogra-
phy).
The size and coverage of these databases varies consider-
ably, from the .14 Mbyte collection of 117 submitted
abstracts to the 7.46 Mbyte collection of 8,629 documents
which represents the union of all pre-existing datasets. For
our experiments, each database was processed in a com-
pletely automatic fashion. A term by document matrix was
generated for each collection. Cell entries represent the
frequency with which a given term occurs in a given docu-
ment multiplied by the entropy weight for that term. An
SVD analysis was performed on this matrix and the k-larg-
est singular values and their corresponding term and docu-
ment vectors were used for retrieval. The number of

Automating the Assignment of Submitted Manuscripts to Reviewers 237

 Number of Mean Rated Mean Relevance of Top
Dimensions
Dataset Name Relevant Articles Precision Relevance of 5 & 5 More Picked from
Used
in Top 10 Top 10 Top 20

50 5.7 .49 5.58 6.69

Submitted Abstracts
100 5.7 .51 5.57 6.67

Submitted and Reviewers’ 50 5.8 .49 5.66 6.93

Abstracts 100 5.5 .50 5.49 6.53

100 5.7 .51 5.76 6.73

Nielsen
500 6.3 .53 5.94 6.82

50 6.0 .51 5.80 6.67

Rada
100 6.1 .52 5.77 6.73

Shneiderman & 50 4.9 .46 5.30 6.13

Kearsley 100 4.9 .47 5.27 6.27

100 5.9 .52 5.79 6.74

Three books
250 6.3 .52 6.00 6.77

50 6.4 .53 5.95 6.89

ACM Articles
100 6.5 .50 6.06 6.84

100 5.9 .51 5.76 6.95

ACM Frames
200 6.0 .52 5.81 6.83

100 6.5 .53 5.95 7.00

Perlman Bibliography
200 6.4 .52 5.86 6.85

100 5.9 .51 5.70 6.74

All Datasets
230 6.3 .52 5.89 6.82

Mean of Above 5.9 .51 5.75 6.73

Random Assignment 4.0 .41 4.70 5.84

Result of the Perfect Match 10.0 1.00 8.20 8.20

Table 2 Performance of the automated review assignment with various domain representations and dimensions. All results
are based on matching the reviewers’ abstracts with the submitted abstracts and then normalizing the results for
those reviewers who supplied multiple abstracts.

dimensions, k, varied from 50 to 500 in the experiments
reported here.
Results
Table 2 summarizes the results of some of our experi-
ments. For each dataset, we present results from the four
different measures of performance described earlier. For
each dataset, we present results from two different LSI
spaces, one with 100 dimensions (which previous research
indicates is a reasonable choice), and smaller or larger val-
ues than this depending on the dataset. As in previous
experiments [Dumais 1991], we found a large initial
increase in performance as the number of dimensions
increased, a peak at some intermediate value, and then a
gradual decline to a level that corresponds to a standard
vector retrieval method.
In general, performance is well above chance; the results
are consistent for the four different measures of perfor-
mance; and there is somewhat less difference in perfor-
mance across the domain representations than we
expected. Averaged over all datasets and dimensions
(some of which we know to be sub-optimal), the mean
number of relevant articles in the top 10 articles returned
by our automated method is 5.9. So, more than half of the
articles are rated as at least a “good match” given our rat-
ing scale. This represents an advantage of 48% over what
would have been obtained with random assignment of arti-
cles to reviewers (an average of 4.0 relevant in the top 10).
The average three-point precision value is .51. This repre-
sents a 24% advantage over random assignment (for which
precision was estimated using 100 different random order-
ings of articles). The average rated relevance of the top 10
returned articles is 5.75, where a rating of 6 or above indi-
cates a “good match.” This represents a 22% advantage

238 Dumais and Nielsen

over random assignment. The average relevance is even
higher (6.73), when the 10 items assigned to each reviewer
are determined by a mixture of 5 preassigned articles and 5
reviewer-selected articles from the next 15.
All performance measures are well above what would have
been obtained by random assignment, and, in some cases,
are better than the assignments of human experts (see the
following section on “Comparing the Automated Method
to Human Review Assignment” for details). However, the
automatic assignment is also far from optimal.
We should note that relevance judgments are not perfect
indicators of the “true” relevance of the papers. First, they
were based on abstracts only, and the abstracts were some-
what misleading in a few cases. Second, reviewers are
unable to judge their own interests with perfect consis-
tency. One of the reviewers completed a second relevance
judgment form about seven months after his initial rele-
vance judgments. The correlation between the ratings
given by this same individual on these two occasions was
.76, which is certainly high enough to indicate better than
random consistency (p<.001), but still low enough to indi-
cate some uncertainty in estimating exact relevance values.
Generally speaking, our reliability results are consistent
with those reported by Lesk and Salton [1969] although
quite different reliability and performance measures are
used. We observed relatively large differences in relevance
assessments for the same observer on two different occa-
sions (.76 correlation), but these differences did not result
in large differences in mean rated relevance (only 5%)
when we used judgments obtained on one occasion to pre-
dict performance using those obtained on the other occa-
sion.
There is some variability in performance across the differ-
ent datasets that we used to represent domain knowledge.
For example, the average number of relevant articles in the
top 10 ranges from 4.9 (Shneiderman & Kearsley, 50 and
100 dimensions) to 6.5 (ACM Articles, 100 dimensions;
Perlman Bibliography, 100 dimensions). Overall, the best
performance is obtained with the Nielsen, ACM Articles,
and Perlman datasets, and the worst performance occurs
for the Shneiderman & Kearsley book. The variability
across datasets is smaller than one might have expected,
and does not appear to be related to the absolute size of the
dataset. Performance may be related to some other charac-
teristic of the datasets such as how representative they are
of the hypertext field. Quite good performance is obtained
simply by analyzing the small collection of submitted
abstracts. This may not be as surprising as it appears since
the abstracts span exactly the relevant range of topics.
However, it does suggest that additional effort to incorpo-
rate domain knowledge (in so far as it is taken into account
by LSI) may not always be necessary.
These results represent unconstrained review assignments,
where no attempt has been made to ensure that each paper
is reviewed by a minimum number of reviewers or that
Automating the Assignment of Submitted Manuscripts to Reviewers
each reviewer is assigned a reasonable number of papers.
As mentioned in the introduction, a balanced assignment
can be derived from our data by adding a number of linear
constraints. By using an efficient implementation of
Edmonds’ b-matching algorithm [Applegate and Cook
1992; Edmonds 1965], we were able to handle the con-
straints of a balanced review assignment in 3–5 seconds on
a standard workstation. We tested the assignment of 2, 3,
and 4 reviewers per paper based on several different
datasets and found that the mean relevance rating was
reduced by 2.3%, 0.9%, and 0.7%, respectively, when tak-
ing the balancing constraints into consideration. Basically,
it seems that the assignment of multiple reviewers per
paper involves so many possible combinations that the
addition of constraints does not reduce the quality of the
solution much.
Representing the Reviewers’ Interests
Reviewers are automatically represented as points in the k-
dimensional LSI space. Recall that each reviewer
described his/her interests by giving us one or more
abstracts. The simplest use of the multiple abstracts is to
concatenate them to a single file, but one can sometimes
improve the performance of matching methods by consid-
ering smaller focused components of large, heterogenous
texts [Salton and Buckley 1991]. We therefore examined
three methods of representing reviewers’ interests—com-
bined, split-nonorm, split-norm. For the “combined”
method, each reviewer was represented as a single point.
The location of that point was determined by taking the
weighted average of all the terms in all the abstracts given
to us by that reviewer. For the “split” methods, each
reviewer was represented as many points, one correspond-
ing to each of the abstracts they gave us (an average of 3.3
per reviewer). For both split methods, we compute the sim-
ilarity between every abstract representing a reviewer and
every document. The two split methods differ only in how
the information from the multiple abstracts representing a
given reviewer is combined to obtain a single ranking of
the similarity between submitted abstracts and the
reviewer. In the “split-nonorm” method, we simply pick
the largest cosine for each submitted abstract from the
cosines for the separate abstracts representing that
reviewer. These cosines are then sorted to obtain the
ranked list of reviewer to abstract similarities. In the
“split-norm” method, we first normalize the cosines for
each of the reviewers’ abstracts (i.e., transform the cosines
so that they have a mean of 0 and a standard deviation of
1), then proceed as described above.
Thus, for each reviewer we have three rankings of submit-
ted abstracts. In all cases, the similarity between each
reviewer and each submitted abstract is measured by the
cosine between the point(s) representing the reviewer and
the submitted abstract in LSI space. The resulting rankings
are used in our evaluations.
For the present case study, there is not much difference in
the outcome of the review assignment whether reviewers
239
 Representation of the
Number of
Reviewers Dimensions
Relevant Articles
(Matching based on Used
in Top 10
“All Datasets”)
100 5.9
Reviewer’s Abstracts
230 6.3
100 5.7
Reviewer’s Name only
230 5.5
Table 3 Performance of the automated review assignment method when “All Datasets” are used for the domain representa-
tions. The first row is based on matching the reviewers’ abstracts with the submitted abstracts and then normalizing
the results for those reviewers who supplied multiple abstracts. The second row is based on matching the reviewers’
family names with the submitted abstracts.
are represented by a single point (as the combination of
their abstracts) or as several separate points (as the individ-
ual abstracts). Sometimes one method is slightly better,
and sometimes the other method is slightly better. Aver-
aged over the measures of performance shown in Table 2,
the split-norm method is 0.6% better than the combined
method.
Not normalizing the ratings before combining the values
for each reviewers’ abstracts generates worse results in
practically all the cases we have looked at. Averaged over
the measures of performance shown in Table 2, the split-
norm method is 3.5% better than the split-nonorm method.
The data in Table 2 represent the use of the split-norm
method which we chose due to its overall advantage over
the combined and split-nonorm methods.
We believe that the advantage of normalizing arises
because it, in effect, gives equal weight to all reviewers’
abstracts. In some cases, the lengths of the reviewers’
abstracts varied quite a bit. If the cosines are not normal-
ized before combining, shorter abstracts will tend to have
higher similarities with target documents—i.e., if two
abstracts of different lengths have the same overlap (dot
product) with a document, the shorter abstract will have a
higher cosine with the document. Thus, in cases where the
reviewers’ abstracts vary in length, shorter abstracts can
dominate the combined ranking for partly artifactual rea-
sons. By normalizing the cosines for each of the reviewers’
abstracts before combining, these effects are minimized
and the distance in the LSI space between the query and
submitted abstracts determines the combined ranking.
Although the performance differences between the split
and combined representations are not large in the present
experiment, we have reason to believe that they would be
larger and important in other contexts. Salton and Buckley
[1990] and Kane-Esrig et al. [1989] have suggested that
retrieval performance can be improved when heteroge-
neous texts are split into smaller units. The advantage of
such an approach will be greatest when the original text is
quite heterogeneous (i.e., when the inter-item similarity of
the smaller units of the split representation is low). In our
240
Mean Rated Mean Relevance of Top
Precision Relevance of 5 & 5 More Picked from
Top 10 Top 20
.51 5.70 6.74
.52 5.89 6.82
.48 5.73 6.74
.49 5.62 6.63
application, the reviewers’ multiple abstracts were often
quite similar to each other. When the split representations
are not very different from each other or from the com-
bined representation, small performance differences are to
be expected. In other applications such as information fil-
tering where a single person may have several quite dispar-
ate interests, we believe that the advantages of the split
methods would be larger.
For practical applications, it may sometimes be difficult to
gather machine-readable abstracts from all the members of
a review committee. Therefore, we have also considered a
minimalist method of representing reviewers by their fam-
ily names and nothing else. This simple method is feasible
when the reviewers’ names occur as terms in the dataset
used for the domain representation. This will often be the
case for research conferences where reviewers are mostly
people who have published extensively and are cited fre-
quently by other authors.
Representing reviewers by their name can be done for 24
of the 25 reviewers (14 of the 15 for which full relevance
ratings were collected). Unfortunately, the name of one of
the reviewers did not occur as a term in the pre-existing
datasets. There are many ways of handling this problem.
We chose a simple and general method—we added some
text containing this reviewer’s name to the database. For
our particular application, we added the reviewers’ self-
descriptive abstract to the database. We then use the loca-
tion of the abstract to determine where the new term (the
reviewer’s name in this case) is located. Since terms in the
LSI space are at the centroid of the documents which con-
tain them, this is roughly the same location that would
have been obtained if the database contained this abstract.
Table 3 shows the results of representing reviewers with
their names only, when All Datasets are used to form the
domain representation. The results are somewhat worse
than most of the analyses using the reviewers’ full
abstracts. The use of full abstracts is 4.3% better on aver-
age than using names only (averaged over the performance
measures shown in Table 3). Review assignments using
the reviewers’ names as the only representation of their
interests is thus a feasible fall-back method if abstracts are
Dumais and Nielsen
 Perfect Match of Reviewers’ Interests
9 n of 2n (All Datasets)
LSI (All Datasets)
LSI (Submitted Abstracts)
LSI (Shneiderman & Kearsley)
Random Review Assignments
8 Human Experts
Mean Relevance Rating

4
0 10 20 30 40
Number of Papers per Reviewer

Figure 1 Curves showing how the mean relevance ratings of the assigned papers drops as more papers are assigned to
each reviewer when using different datasets for the domain representation. All curves (except the top and bottom
curves) have been calculated with the number of LSI dimensions being equal to 100. The top curve shows the the-
oretically optimum relevance rating that would be achieved by perfectly matching the reviewers’ own ratings. The
n of 2n curve (stippled gray) shows the result of sending the reviewers twice as many papers as they are asked to
review and having them pick half of their review assignment themselves from among these papers. The black dots
indicate the performance of three humans experts who simulated the program chair’s job.

unavailable for some reviewers, but it is not recommended
as a general approach.
One obvious problem with representing the reviewers by
their family name only is that several different individuals
may have the same name even though they have widely
differing research interests and review expertise. The
Hypertext’91 reviewers mostly had fairly rare family
names that to our knowledge are not shared by other active
researchers in the hypertext field. It is likely, however, that
some name overlap would occur if the same method were
to be used for broader fields.
One can also represent reviewers by the keywords given
for their papers rather than the full text of their abstracts.
Doing so gives slightly better performance than using the
reviewers’ names only, but using the full abstracts is still
4.1% better than using the keywords.
Unless otherwise noted, all results reported in this paper
represent the reviewers by several independent abstracts
for which the cosine similarity measures have been nor-
malized before combining (the split-norm method).
Varying the Number of Papers
Assigned to Each Reviewer
Figure 1 compares the mean relevance rating for various
assignment methods as more and more papers are assigned
to each reviewer. The figure has curves for three of the
automatic LSI methods listed in Table 2 (All Datasets, the
117 submitted abstracts, and the Shneiderman & Kearsley
book). Figure 1 also contains curves for the n of 2n method
discussed further below, and for perfect matches and ran-
dom assignment.

Automating the Assignment of Submitted Manuscripts to Reviewers 241

The perfect match curve indicates the results of following
the reviewers’ own ranking and is by definition monotoni-
cally non-increasing. In contrast, the curves representing
the various automatic review assignments sometimes con-
tain bumps, indicating improved performance with the
assignment of an additional paper. These bumps reflect the
fact that the automatic methods are not perfect and some-
times will rank an extremely relevant paper fairly low.
When the review load for a reviewer is expanded to
include such a highly rated paper, the mean relevance of
the review assignment goes up slightly.
The overall mean relevance rating for all the submitted
papers for all the reviewers is 4.7 as indicated by the lower
flat line in Figure 1. So all the automatic review assign-
ments do select considerably better than random paper
assignment regardless of how many papers are assigned to
each reviewer (from 1 to 40).
Figure 1 also shows the simulated results of an alternative
review assignment method where the reviewers help
choose their own review assignments. The method is
called “n of 2 n” and is based on sending each reviewer
twice as many papers as that reviewer is actually asked to
review. This is a generalization of the “Top 5 and 5 More”
method reported in Table 2. The 2n papers are simply cho-
sen as the top 2n papers for each reviewer, calculated by
the automatic LSI method (here, we used All Datasets at
100 dimensions). The top n/2 papers are assigned to the
reviewer. This ensures that each paper receives a minimum
number of reviews. The other n/2 papers to be reviewed
are chosen by the reviewer from among the remaining 3/2
n papers sent to that reviewer. We believe that there is also
a motivating effect in having reviewers choose part of their
own review load, but we do not have any empirical evi-
dence to support this claim. The motivation of the review-
ers is important for conference program committees which
normally consist of unpaid volunteers.
As can be seen from Table 2, the mean rated relevance for
a review load of ten papers goes up by about one rating
unit (from 5.75 to 6.73, or 17%) when the reviewers are
sent twenty papers instead of ten and are allowed to choose
five of their papers themselves. Figure 1 shows that the
advantage of the n of 2n method is fairly constant except
when very few papers are assigned to each reviewer.
The n of 2n curve must fall between the LSI All Datasets
curve and the perfect match curve because it represents a
combination of these two assignment methods. As the
number of articles assigned to each reviewer increases, the
n of 2n curve approaches the perfect match curve.
Comparing the Automated Method to
Human Review Assignment
In addition to comparing the various automated review
assignment methods to each other, we also compared them
to the performance of human program chairs assigning
reviews manually. This section presents two such compari-
242
sons: A simulation of manual review assignments for the
same submissions and reviewers used for the automated
methods, as well as the actual manual assignments from a
different conference.*
In order to simulate manual assignment of the Hyper-
text’91 submissions, three human hypertext experts† inde-
pendently read through all the abstracts and judged which
of the members of the review committee would be appro-
priate as reviewers for each paper. No attempt was made to
balance the number of papers assigned to each reviewer or
the number of reviewers assigned to each paper. Therefore,
the number of papers assigned to the individual reviewers
varied from 11 to 42 (mean 27.7) for the first expert, 17 to
40 (mean 27.7) for the second expert, and 20 to 41 (mean
29.7) for the third expert.
For the Hypertext’91 conference, the members of the pro-
gram committee actually reviewed an average of 25.8
papers each, so the 28–30 papers assigned by our human
experts come fairly close to reflecting the actual review
load for this conference. Most conferences would probably
place a smaller review load on their reviewers, however.
The manual assignment method assigns more papers to
reviewers that are seen by the human expert as having
broader coverage. Since the number of articles assigned to
reviewers varies, one would expect somewhat different
performance than that observed when just picking the top
28 or 30 papers for each reviewer. In practice, however, the
differences are very small (less than 1%). Roughly speak-
ing, then, we can compare the automatic methods with the
human experts by considering the performance of the auto-
matic methods at 28 and 30 papers per reviewer, respec-
tively, as shown in Figure 1. The mean performance of the
three human experts is represented as points in Figure 1.
Table 4 compares the mean rated relevance of the papers
assigned by the human experts with the relevance of the
same number of papers assigned by the automated meth-
ods (the LSI method and the n of 2n method, both using All
Datasets at 100 dimensions). These results indicate that
the simple LSI method is not as good as the best human
experts but that it performs in the same general range and
* We could not use the actual assignments from Hypertext’91
because the program chair used a mixture of automated and
manual methods to make the original assignments.
† The first human expert is the editor of a respected informa-
tion systems journal which has published many hypertext
papers. This person is not a specialized hypertext expert to the
same degree as the two other experts, however. The second and
third human experts were people who have served as program
chairs for a major hypertext and a major user interface confer-
ence, respectively, and are recognized as knowledgable in the
hypertext field. It is reasonable to expect all three experts to be
able to perform the task well, and the second and third experts
are probably very close to achieving the optimum performance
one could get from a human.
Dumais and Nielsen

Mean Number of Papers Assigned to each Reviewer
Mean Rating of Papers Assigned by This Expert
Mean Rating of Papers Assigned by the LSI Method (All Datasets, n=100)
Mean Rating of Papers Assigned by the n of 2n Method
Table 4 Comparison of the performance of human experts with two of the automated methods.
achieves the same performance as our Expert 1. As indi-
cated by Table 4, the n of 2n method does achieve better
performance than all the human experts.
As noted above, our human experts created unbalanced
review assignments since they were asked to find the best
reviewers for each paper without having to ensure that the
reviewers received the same review load or that each paper
was reviewed by the same number of reviewers. It is thus
fair to compare the performance of the human experts with
the performance of our unbalanced automated assignment.
As noted above, it is possible to add constraints to the
automated methods fairly easily and to use an efficient
optimization algorithm that satisfies the constraints. The
assignment generated in this way resulted in only a small
decrease in mean relevance. In contrast, a human would
probably apply the balancing constraints in a sub-optimal
manner since it is virtually impossible for a human to con-
sider all the relevant combinations. We would thus expect
the relative performance of the automated methods to
improve compared to human experts when balancing con-
straints are also considered.
Another estimate of the performance of human experts
assigning reviews can be derived from the ACM CHI’92
conference on computer–human interaction. Admittedly,
this conference is different from the Hypertext’91 confer-
ence in many ways, so a direct comparison would not be
reasonable. The CHI conference is somewhat more estab-
lished and covers a larger field, in terms of number of sub-
missions (312), number of reviewers (137), and scope of
topics addressed by the conference.
For the ACM CHI’92 conference, submissions were manu-
ally assigned to reviewers by the papers chair with the
assistance of a small number of area specialists for certain
sub-fields. After their meeting, 97 members of the papers
committee for whom electronic mail addresses were avail-
able were asked to provide estimates of the appropriate-
ness of having them review the papers they had been
assigned to review, using the same 0–9 scale used for the
Hypertext’91 reviewers. Replies were received from 45
reviewers, for a response rate of 46%. On average, the
reviewers had been assigned 6.8 papers to review, and their
mean relevance rating was 6.95.
Automating the Assignment of Submitted Manuscripts to Reviewers
Expert 1 Expert 2 Expert 3
27.7 27.7 29.7
5.39 5.92 5.94
5.33 5.33 5.31
6.35 6.35 6.34
In our Hypertext’91 case study, the reviewers’ mean rele-
vance rating was 6.15 for the top seven papers assigned to
them by our automated review assignment based on All
Datasets and using 100 dimensions in the scaling. This
number is certainly substantially lower than the 6.95 rating
from the manual CHI’92 paper assignment but it is at least
in the same general range of the ten-point rating scale. Our
n of 2n method based on All Datasets and 100 dimensions
achieves a 6.72 mean relevance for n=6 and 6.74 for n=8.
This is still lower than the 6.95 CHI’92 rating, but not
much so.
Conclusions
It is possible to automate the assignment of submitted
manuscripts to relevant reviewers using information
retrieval methods. When reviewers are sent only the manu-
scripts they are asked to review, even the best automated
methods perform somewhat worse than manual assign-
ments by human experts, but they do generate fairly good
results. A new method called n of 2n results in substan-
tially higher mean ratings and is better than human experts.
This method involves sending reviewers twice as many
manuscripts as they are actually asked to review and
allowing them to choose half of their review assignment
themselves from the papers they are sent (the other half is
pre-assigned to guarantee coverage of all papers). This
method may also have some motivational advantage over
traditional assignment methods where the reviewers have
no choice.
Although we applied the n of 2n method to the assignment
of submitted conference papers to reviewers, we believe
that it is also applicable to other cases where one has a
large set of information objects that need to be matched
with a large set of humans such that each human gets
assigned a small number of objects. A further condition for
the applicability of the method is the ability of the humans
to quickly estimate the relevance of the information
objects they must select from.
For the Hypertext’91 case study, the performance of the
automated review assignments was fairly independent of
the dataset picked to generate the initial domain represen-
tation. Of course, we only tested datasets that were rele-
vant for the hypertext domain, and this field is still fairly
243
small and focused. One of the datasets did exhibit substan-
tially poorer performance than the others, however, so it is
worth some effort to try to avoid such datasets or to com-
pensate by combining them with others.
We know in retrospect that the Shneiderman & Kearsley
book turned out to be comparatively poor as a basis for
forming a domain representation of hypertext in LSI.
Combining several books does not improve performance
over using a single book that happens to be appropriate for
the purpose, but one cannot know whether any given book
will in fact be appropriate without first collecting exhaus-
tive relevance feedback from the reviewers. Since it would
seem hard—if not impossible—to determine in advance
which books are appropriate as domain representations,
the best advice is probably to rely on as large collections of
books and other text as possible.
For the Hypertext’91 case study, we found that represent-
ing reviewers as several points corresponding to each of
their self-described interests resulted in slightly better
matching performance. We believe that the advantage of
split representations will be larger the more different a per-
son’s areas of interest are from each other. Applications
such as information filtering (where a single person may
have several disparate interests), or databases where que-
ries and/or target documents are large and heterogeneous
are most likely to benefit from such methods. Normaliza-
tion is important when combining information from multi-
ple representations.
Acknowledgments
The authors would like to thank Ben Shneiderman, Greg
Kearsley, and Paul Hoffman for providing a machine-read-
able version of the full text of the book Hypertext Hands-
On! and Roy Rada for providing a machine-readable ver-
sion of the full text of the book HYPERTEXT: From Text to
Expertext. We also thank Robert Akscyn and Elise Yoder for
providing early access to the files of The ACM Hypertext
Compendium and Gary Perlman for making his extensive
HCI Bibliography Project collection of abstracts available.
Our analyses on the basis of the word usage in these manu-
scripts are solely our responsibility and should not be taken
as necessarily corresponding to the positions of the authors
and editors of these publications.
The authors thank the anonymous SIGIR referees, Tom
Landauer, Michael Littman, John Patterson, and Scott Stor-
netta for helpful comments on earlier versions of this paper.
We also thank Bill Cook for helping us use his implementa-
tion of an efficient matching algorithm for optimization
under constraints.
Finally, the authors thank several members of the Hyper-
text’91 program committee for letting themselves be volun-
teered into providing us with exhaustive relevance ratings
of the submitted papers. We also thank the three human
experts for helping us with the time consuming job of esti-
mating the match between each individual reviewer and
each submitted paper.
244
References
Akscyn, R. (Ed.) (1992). The ACM Hypertext Compendium.
ACM Press, New York, NY.
Applegate, D., and Cook, W. (1992). Solving large-scale
matching problems. Paper to appear.
Cohen, P.R., and Kjeldsen, R. (1987). Information retrieval
by constrained spreading activation in semantic net-
works. Information Processing & Management 23, 4,
255–268.
Cullum, J.K., and Willoughby, R.A. (1985). Lanczos algo-
rithms for large symmetric eigenvalue computations—
Vol. 1 Theory, (Chapter 5: Real rectangular matrices).
Brikhauser, Boston.
Deerwester, S., Dumais, S.T., Landauer, T.K., Furnas, G.W.,
and Harshman, R.A. (1990). Indexing by latent seman-
tic analysis. Journal of the Society for Information
Science 41, 6, 391–407.
Dumais, S.T. (1991). Improving the retrieval of information
from external sources. Behavior Research Methods,
Instruments and Computers, 23, 2, 229–236.
Edmonds, J. (1965). Maximum matching and a polyhedron
with 0,1 - vertices. Journal of Research of the National
Bureau of Standards 69B, 125–130.
Furnas, G.W. (1986). Generalized fisheye views. Proc.
ACM CHI’86 (Boston, MA, 13–17 April), 16–23.
Furnas, G.W. (1991). Personal communication.
Furnas, G.W., Deerwester, S., Dumais, S.T., Landauer, T.K.,
Harshman, R.A., Streeter, L.A., and Lochbaum, K.E.
(1988). Information retrieval using a singular value
decomposition model of latent semantic structure. In
Proceedings of SIGIR’88, 465–480.
Kane-Esrig, Y., Casella, G., Dumais, S.T., and Streeter, L.A.
(1989). Ranking documents for retrieval by modeling of
a relevance density. In Proceedings of the 12th IRIS
Conference (Information System Research Seminar In
Scandinavia), Skagen, Denmark, August 1989.
Lesk, M.E., and Salton, G. (1969). Relevance assessments
and retrieval system evaluation. Information Storage
and Retrieval 4, 343–359.
Nielsen, J. (1990). Hypertext and Hypermedia. Academic
Press, San Diego, CA.
Perlman, G. (1991). The HCI bibliography project, ACM
SIGCHI Bulletin 23, 3 (July), 15–20.
Rada, R. (1991). HYPERTEXT: From Text to Expertext .
McGraw-Hill, London, U.K.
Salton, G., and Buckley, C. (1991). Automatic text structur-
ing and retrieval—Experiments in automatic encyclo-
pedia searching. Proc. ACM SIGIR’91 Conf. (Chicago,
IL, 13–16 October), 21–30.
Salton, G., and McGill, M.J (1983). Introduction to Modern
Information Retrieval. McGraw-Hill.
Shneiderman, B., and Kearsley, G. (1989). Hypertext
Hands-On! An Introduction to a New Way of Organiz-
ing and Accessing Information . Addison-Wesley,
Reading, MA.
Dumais and Nielsen