Noname manuscript No.

(will be inserted by the editor)

Comparison of academic search engines: Google

Scholar and Semantic Scholar
Michael J Mathew

Received: 12/01/2016

Abstract Research in any field heavily depends on past works performed and
hence easier systematic ways to access relevant publications are a great boon
to researchers. Apart from the individual publishing houses giving search engines to their content, there are several other academic search engines used
by researchers for finding articles, journals and conference publications. Two
such free academic search engines of relevance are Google Scholar(GS) and Semantic Scholar(S2). This work compares various aspects like user friendliness,
quality and quantity of search results of both these search engines for finding
various published works in the domain of Computer Science and draws conclusion from the results obtained. The S2 search engine demonstrates significant
improvement in search results with some typical bottlenecks faced by an early
stage product.
Keywords Search Engines Google Scholar Semantic Scholar

1 Introduction
The advent of the internet along with newer and efficient algorithms helped
in the creation of search engines which increased the capability to access large
databases of research articles publicaly available throughout the world. Knowledge about the existing works done, not only helps in avoiding repetition but
also speeds up research to an extent. The scientific world should thank the
legendary scientist Eugene Garfield for his ideas of citation based searching,
resource discovery and quantitative evaluation of publications which serve as
Michael J Mathew
The School of Computer Science Tel.: +44 (0)121 414 43710
E-mail: mjm522@cs.bham.ac.uk
University of Birmingham,
Edgbaston, B15 2TT, UK

Michael J Mathew

the basis for many of the powerful and innovative search engines these days [9].
Prior to this type of indexing the techniques used to store and sort publications
where either alphabetically or numerically which resulted in a rather cumbersome way to find various articles [9]. Though most of the publishing houses
provided search services for their content, most of the good ones (e.g. Scopus)
were either paid services and the free ones were insufficient. This created a
huge economic based division particularly for students from developing world.
The idea to provide a free service to index research articles in various domains
under a single umbrella would increase efficiency of researchers substantially.
With the advent of GS[7] other similar free academic research indexing services came in later years such as Microsoft Academic Research, PubMed[5],
JSTOR etc. and finally S2 [6]. The availability of high computational resources
of google coupled with their other services like Google Books has made GS the
leading academic search engine [12].The emergence of machine learning and
other statistical techniques has made the search engines to produce tailor fit
results for the researcher. Though there exists several options for searching,
the search results variance among the search engines are indeed a problem.
The quantity of search results is proportional to the total indexed database
size and quality is determined by various ranking algorithms that are used
to sort the search results. Out of the several free search engines available in
academic research one of the oldest (GS) and the latest (S2) are compared on
some measures regarding the search outputs produced by them.
The article is organized as follows. Section 2 and 3 gives an overview of the
search engines under consideration. Section 4 gives the methodology and topics
based on which the comparison was performed. Section 5 gives the results
obtained on comparison and finally the concluding remarks are presented in
section 6.

2 Overview of Google Scholar

Google Scholar was launched in November 2004 as a subsidiary service of
then the Google (now Alphabet) [8] with an aim to help researchers around
the world to find several research abstracts, articles (corresponding revisions),
theses, books, technical reports and preprints online across all disciplines. A
recent article on research impact shows there are at least 261 subject areas and
10 broad subject areas listed in GS [16]. Though Google does not go behind
the paywalls of various publishing houses, it does provide alternate links to the
same articles in cases when they are available. Google Scholars indexes major publishing presses like IEEE, ACM, HighWire Pressm MetaPress, Ingenta
and government agencies like American Physical Society, NOAA and preprint
servers likes arXiv.org, RePEc, CiteBase etc [10]. Though there are not officially confirmed database sizes of Google Scholar, a recent statistical article
estimates around 160 million research articles [15] which would cover around
80-85 % of the total research articles online. It was the efforts of Google engineers and their continued requests to various publishing houses that gained

Comparison of academic search engines: Google Scholar and Semantic Scholar

them access to index articles available in their paid services. Though Google
does not surpass the paywall created by the publishing firms, the knowledge
of existence of a work and their abstract itself could help totally igonorant
researcher [13].

3 Overview of Semantic Scholar

S2 is a new academic search engine launched in November 2015 by the nonprofit Allen Institute for Artificial Intelligence(AI2) based in Seattle, Washington, USA [11] specifically with focus on Computer Science publications. According to [17], S2 is composed of three main parts: (i) a crawler that searches
the web and downloads publicly available academic papers (ii) a pipeline that
builds semantic index of downloaded papers (iii) innovative user interface.
The main focus of S2 is to provide the experience provided by existing search
engines with the core metadata available in a research paper like the title,
authors, venues, abstract, year of publication etc along with some additional
features. The additional features include citing the sentence boundaries, classification of paper into various classes like survey, theoretical, experimental
etc, finding the key cited papers and also provides links to implementation
code if available in various repositories like GitHub, BitBucket etc. S2 is also
currently experimenting on extracting entities, relations and visual elements
present in the paper to understand the paper in a better way so as to improve the user experience. S2 is aimed not only as a search engine but also as
a Artificial Intelligence (AI) test-bed for various algorithms.

4 Comparison studies
Since S2 just concentrates on Computer Science publications, it would be erroneous to compare keyword searching from other domains. So our experiments
will be restricted to keywords and author names will be tested from Computer
Science domain. The comparison between S2 and GS are done using four experiments. The four planned experiments and the corresponding strategy is
mentioned.
1
2
3
4

Coverage testing: Finding relevant publications by 5 different authors.

Quantity testing: Assessing relevance of a generic keyphrase.
Quality testing: Assessing relevance a specific key-phrase.
Resolving capability testing: Comparing search results of two synonymous
key-phrases.

Accepting the fact that S2 is a recent search engine and its number of indexed
papers would be less compared to the GS, all the above-mentioned test will
be conducted on research papers published between years 2001 and 2016.

Michael J Mathew

4.1 Test 1: Coverage testing

This experiment aims to test the indexing capability of GS and S2. This test
can be conducted by comparing search results produced for a particular author against their personally maintained publication lists. In this test tries
to validate the hypothesis that GS will produce more number of search results compared to S2 and the number will match the true count given by the
authors.
4.1.1 Methodology
Four authors - Dr. Jeremy L Wyatt, Dr. Xin Yao, Dr. Ales Leonardis and Dr.
Peter Tino from the department of Computer Science, University of Birmingham were chosen and searched for their number of publications from years 2001
to 2016 and were manually checked. Then these search results are compared
to their personal homepage publication list [2],[4],[1],[3]. A difference in the
number of search results - if exists, would provide a useful metric to compare
the indexing capability of both GS and S2. This test is on the assumption that
the personal webpages of these authors are updated regularly.
4.1.2 Results and discussion

Fig. 1: Coverage testing results

The results above shows variations in the number of publications by each

of the authors. While both search engines dont match the true count given

Comparison of academic search engines: Google Scholar and Semantic Scholar

in each of the authors homepage, GS results seem to be more precise than

S2. The existence of GS for a longer time could explain the discrepancy in
the data. An additional observation is that S2 gives more importance to the
surname while GS gives importance to both surnames as well as forenames.
This results in seeping in of some incorrect results in later pages of S2. We can
observe that a part of our hypothesis was tested true.

4.2 Test 2: Quantity testing

To test the quantity of results produced by each of the searches engines, three
generic key phrases were selected. Here also we hypothesize that S2 will produce more relevant results than GS due to the usage of semantics support in
S2. A generic key phrase is usually difficult to resolve by a search engine due
to the presence of inherent ambiguity in them.
4.2.1 Methodology
A set of three generic phrases selected: (i) Target tracking(ii) Reinforcement
learning and (iii) Dexterous manipulation. These key phrases are searched in
both search engines and first 30 results are analyzed manually.
4.2.2 Results and discussion

(a) S2 Results

(b) GS Results

Fig. 2: Phrase 1: Target tracking

Curious enough, we can see that searching phrase - target tracking produced almost equal irrelevant results on both search engines. This is due to
the presence of same phrase in multiple research domains like missile control literature, mobile robots literature, manipulator robotics literature. This
is supported by the results of phrase 3, where all first 30 results were relevant. Dexterous manipulation is a phrase that belongs to mainly manipulator
robotics literature.

Michael J Mathew

(a) S2 Results

(b) GS Results

Fig. 3: Phrase 2: Reinforcement learning

(a) S2 Results

(b) GS Results

Fig. 4: Phrase 3: Dexterous manipulation

4.3 Test 3: Quality testing

The main intention behind the usage of page ranking in search engines is to
produce quality results in accordance with the corresponding search phrase.
Giving few specific keywords in a sub-domain in one of the above three topics
will help us to understand the effectiveness of the page ranking algorithm used.
The hypothesis will be in support of GS due to its popularity and long ranging
indexes to many previous years. Though it was mentioned previously that S2
mainly indexes computer science papers, the quality of search results can only
be tested by searching a slightly off topic term.

4.3.1 Methodology
The specific keyword selected will be in the missile control literature since
this phrase produced the maximum number of irrelevant results. The specific
keyword to be searched is target tracking in ballistic missiles. The first 30
results of the search term will be compared to find the number of relevant and
irrelevant results.

Comparison of academic search engines: Google Scholar and Semantic Scholar

4.3.2 Results and discussion

Looking at the first few results of both search engines itself shows that GS
has a better quality for searching the specific keyword. While searching for a
specific term, it is always good to get some survey papers in the area since it
gives more idea about the field. We can see that GS satisfies this compared to
S2. It can be observed that the results produced by S2 are almost in relation to
the domain of CS. The comparison of first 30 results are shown below. Though

(a) S2 Results

(b) GS Results

Fig. 5: Screenshot of Quality testing

there is a significant improvement in the search results of both engines, GS
still outsmarts S2.

(a) S2 Results

(b) GS Results

Fig. 6: Phrase 3: Dexterous manipulation

4.4 Test 4: Resolving capability testing

Research domains are rich with jargon of various size and forms, which in
many cases have a similar meaning. Hence researchers familiar with different
key phrases which have same meaning in literature needs to be resolved by
the search engine and provide relevant results to the query. This is a difficult

Michael J Mathew

job as it involves a classification according to a feature which is hidden. The

hypothesis to be tested is that S2 would produce significantly better results
than GS with the help of its semantic parsing.
4.4.1 Methodology
The synonymous keywords selected for testing are motion planning and path
planning. Both keywords are used interchangeably in the literature. Our method
will be to search each keyphrase individually in each of the search engines and
compare the first 10 results.
4.4.2 Results and discussion

(a) S2 Results

(b) GS Results

Fig. 7: Resolving ability test results (pp : path planning, mp: motion planning)

The resolving ability test for both search engines produced similar results.
This proves the null hypothesis. Even if Google does not employ semantic
methods, it is using some alternative method to resolve various keyphrases.

4.5 User interface

The user interface is an important aspect of any product. Availability of huge
amount of research publications ranging from original research to mediocre
quality makes it almost impossible for a researcher to find relevant works.
Techniques to cut through clutter to get relevant works drastically reduce
the effort of the enquirer. It is true that citations cannot alone determine
the quality of a work; for example, a researcher working in a gray area, not
currently beeing traversed by other researchers may not get cited. Of all the
citations received almost 10 % of citations of a paper is of most importance
[17]. So a user interface that produces results taking into consideration such
factors greatly improve the quality of papers found.

Comparison of academic search engines: Google Scholar and Semantic Scholar

(a) S2 default options (b) GS default options

Fig. 8: Default options given by search engines

5 Summary of studies
The studies conducted so far can be arranged into a set of tables for quick
reference. These are listed in tables 1,2,3 and 4.
Table 1: Search ability: S2 : Pros and Cons [14]
Search Engine Name
Pros

Cons

Semantic Scholar
lower recall and higher precision
easily filter survey papers
sorting based on venue
links to GS results
lacks advanced search
no links to Find@BHAM
no coverage to patents and case law

10

Michael J Mathew

Table 2: Search ability: GS: Pros and Cons [14]

Search Engine Name
Pros

Cons

Google Scholar
higher recall and lower precision
links to papers in Find@BHAM (internally hosted page)
covers patents and case law
presence of advanced search option
redundant search results
there is no way to know the relevance of citation
no option to sort according to venues
no option to search among relevant sub-keywords

Table 1 represent few of the features one notice while searching similar
keywords on both search engines. Similarly table 2 denote variation in design
and usability of both search engines.
Table 3: Design, Usability: S2: Pros and Cons [14]
Search Engine Name
Pros

Cons

Semantic Scholar
streamlined with essential features
intuitive search by finding relevant sub keywords
easy breakdown of citation articles
features like access to figures, citation velocity
fewer external links
lacks internally hosted
no email alerts could be set

Table 4: Design, Usability: GS: Pros and Cons [14]

Search Engine Name
Pros

Cons

Google Scholar
layout is similar to Google search results
alerts can be scheduled to email users with query
can track and save user citations
features like metrics and language
clutter produced by search results
simple design with lack of some essential features
no intention to understand searches

6 Conclusion
With four different experiments conducted on both search engines, we can
conclude the following aspects. Though the results of GS and S2 shows some
visible differences in terms of coverage (with GS leading); it would be highly
inaccurate to conclude that the search result of S2 is worse than GS. The
features like easy sorting of survey papers are highly useful during the start of

Comparison of academic search engines: Google Scholar and Semantic Scholar

11

any research which is produced by S2. At the same time, lack of giving results
of internally hosted pages of a university puts an additional burden on the
user to check the article again in his own University resources. But overall the
search features specified by S2 seems to be more relevant to a researcher at the
early stage of a work. The presence of some short comings mentioned above
could be the result of a comparatively smaller database size of S2 which could
be improved in time. Even if most of the tests conducted resulted in favor of
GS, one can observe a strong positive correlation between the results. This
supports the fact that S2 is on the right path. Apart from this, the semantic
support provided by S2 is not used in GS leading to finding some more relevant
results in some cases.
One must not forget the fact that GS since its inception in 2004 and has a
well-indexed database (more than 10 years of crawling) along with remarkable
aid from other Google products like Web search and Google books repository.
While S2 is still at its BETA stage with specificity to Computer Science publications and has not fully achieved their true goals(see Sect. ??)- which are
different in many aspects to GS. It might be an early improper to compare
their strengths in their current states. With a more open source intention of
S2, given time and a steady progress at current pace, there is no doubt that
S2 would be a better research friendly search engine than GS.
References
1. Dr. Ales Leonardiss home page.
https://www.cs.bham.ac.uk/~leonarda/
Publications.html. [Accessed Aug 2016].
2. Dr. Jeremy L Wyatts home page. http://www.cs.bham.ac.uk/~jlw/publications.
php. [Accessed Aug 2016].
3. Dr. Peter Tinos home page. https://www.cs.bham.ac.uk/~pxt/my.publ.html. [Accessed Aug 2016].
4. Dr. Xin Yaos home page. http://www.cs.bham.ac.uk/~xin/publications.html. [Accessed Aug 2016].
5. Pubmed. http://www.ncbi.nlm.nih.gov/pubmed. [Accessed Aug 2016].
6. Allen AI Institute. Semantic Scholar. https://semanticscholar.org. [Accessed Aug
2016].
7. Google. Google Scholar. https://scholar.google.co.uk. [Accessed Aug 2016].
8. Tracey Hughes. An interview with anurag acharya, google scholar lead engineer. http:
//www.indolink.com/SciTech/fr010305-075445.php, December 2006. [Accessed Aug
2016].
9. Peter Jacso. As we may search-comparison of major features of the web of science,
scopus, and google scholar citation-based and citation-enhanced databases. Current
Science -Banglore-, 89(9):1537, 2005.
10. P
eter Jacs
o. Google Scholar: the pros and the cons. Online information review,
29(2):208214, 2005.
11. Nicola
Jones.
Artificial-intelligence
institute
launches
free
science
search
engine.
http://www.nature.com/news/
artificial-intelligence-institute-launches-free-science-search-engine-1.
18703, November 2015. [Accessed Aug 2016].
12. Madian Khabsa and C Lee Giles. The number of scholarly documents on the public
web. 2014.
13. Steven Levy. Making the worlds problem solvers 10https://medium.com/backchannel/
the-gentleman-who-made-scholar-d71289d9a82d#.16jjodjhu, October 2014. [Accessed Aug 2016].

12

Michael J Mathew

14. Free Infographic Maker. Academic search engines: Semantic Scholar vs Google Scholar.
https://infograph.venngage.com/p/66001/semantic-scholar-vs-google-scholar.
[Accessed Aug 2016].
15. Enrique Ordu
na-Malea, Juan Manuel Ayll
on, Alberto Martn Martn, and Emilio Delgado L
opez-C
ozar. About the size of Google Scholar: playing the numbers. arXiv
preprint arXiv:1407.6239, 2014.
16. Alex Verstak, Anurag Acharya, Helder Suzuki, Sean Henderson, Mikhail Lakhiaev, Cliff
Chiung Yu Lin, and Namit Shetty. On the shoulders of giants: The growing impact of
older articles. arXiv preprint arXiv:1411.0275, 2014.
17. S2 Team Member Vu Ha. What is Semantic Scholar and how will it work? Accessed
Aug 2016.