Sie sind auf Seite 1von 13

The Journal of Systems and Software 113 (2016) 101113

Contents lists available at ScienceDirect

The Journal of Systems and Software


journal homepage: www.elsevier.com/locate/jss

What recommendation systems for software engineering recommend: A


systematic literature review
Marko Gasparic, Andrea Janes
Free University of Bolzano, Italy

a r t i c l e i n f o a b s t r a c t

Article history: A recommendation system for software engineering (RSSE) is a software application that provides informa-
Received 19 November 2014 tion items estimated to be valuable for a software engineering task in a given context. Present the results
Revised 14 November 2015
of a systematic literature review to reveal the typical functionality offered by existing RSSEs, research gaps,
Accepted 17 November 2015
and possible research directions. We evaluated 46 papers studying the benets, the data requirements, the
Available online 26 November 2015
information and recommendation types, and the effort requirements of RSSE systems. We include papers de-
Keywords: scribing tools that support source code related development published between 2003 and 2013. The results
Recommendation system for software show that RSSEs typically visualize source code artifacts. They aim to improve system quality, make the de-
engineering velopment process more ecient and less expensive, lower developers cognitive load, and help developers
Systematic literature review to make better decisions. They mainly support reuse actions and debugging, implementation, and mainte-
nance phases. The majority of the systems are reactive. Unexploited opportunities lie in the development of
recommender systems outside the source code domain. Furthermore, current RSSE systems use very limited
context information and rely on simple models. Context-adapted and proactive behavior could improve the
acceptance of RSSE systems in practice.
2015 Elsevier Inc. All rights reserved.

1. Introduction software engineering: recommendation systems for software engi-


neering (RSSE). The goal of this work is to analyze which systems
Many software engineering techniques support the develop- have been developed in the scientic community and how they can
ment of high-quality software, but the effort they require and the be used. Based on the results, we identied research gaps and possi-
costs of learning them and applying them productively can be ble directions for the future research.
high (Elberzhager et al., 2012). Every (useful) information system processes some form of input
To overcome such diculties, the software engineering commu- and generates an output. The type of required input and the type of
nity develops tools that support the software engineer in her task. produced output have an inuence on the acceptance of a given tool
One type of such tools aims to suggest the best items according to or method. Venkatesh et al. identify four aspects that play a signi-
the users needs. The approach is similar to the one used in search en- cant role as direct determinants of user acceptance and usage behav-
gines and e-commerce recommendation systems1 ; however, recom- ior (Venkatesh et al., 2003):
mendation systems are being used for software engineering purposes
only since recently (Rodrguez-Gonzlez et al., 2010). 1. Performance expectancy: the degree to which an individual be-
lieves that using the system will help him or her to attain gains in
job performance.
1.1. Goal of this work 2. Effort expectancy: the degree of ease associated with the use of
the system.
In this paper, we present the results of a systematic litera- 3. Social inuence: the degree to which an individual perceives that
ture review we performed to explore an emerging research eld of important others believe he or she should use the new system.
4. Facilitating conditions: the degree to which an individual believes

that an organizational and technical infrastructure exists to sup-
Corresponding author. Tel.: +39 3884658700.
E-mail addresses: marko.gasparic@stud-inf.unibz.it, m.gasparic@gmail.com (M.
port use of the system.
Gasparic), ajanes@unibz.it (A. Janes).
1
One of the most known examples of software that suggests to users what to do is
Based on these aspects, we were interested to study the litera-
the recommendation feature of Amazon.com, which recommends users what to buy, ture to extract data about the rst two determinants of acceptance
based on their past acquisitions. and use: performance expectancy, i.e., the produced output and the

http://dx.doi.org/10.1016/j.jss.2015.11.036
0164-1212/ 2015 Elsevier Inc. All rights reserved.
102 M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113

therefore expected benet, as well as effort expectancy, i.e., the ex- valuable for a software engineering task in a given con-
pected effort to provide the required input. The rst two aspects (per- text (Robillard et al., 2010).
formance and effort expectancy) are highly inuenced by the way
how the recommendation tool is conceived, while the last two as- 2. Related work
pects (social inuence and facilitating conditions) are less impacted
by the technical decisions behind the tool but by the context in which We are aware of two papers (Happel and Maalej, 2008; Robillard
it is used: the technical infrastructure, social, and organizational as- et al., 2010) and a book (Robillard et al., 2014) in which the eld is de-
pects. scribed and state-of-the-art tools are presented. Happel and Maalej
Furthermore, we were interested in how such tools are used dur- (2008) made a short survey of recommendation systems in software
ing software engineering, more precisely, during the creation or mod- development, which includes six tools. They found that existing tools
ication of source code, which represents the core activity of software are focusing on you might like what similar developers like scenar-
engineers to satisfy customer requirements (Beck and Andres, 2004). ios, and they encouraged researchers to focus on two aspects: what
Other software engineering activities, e.g., user requirements man- to recommend and when to recommend.
agement, can be also supported by recommendation systems, but we In 2010, Robillard et al. (2010) published a short overview of the
excluded such systems from our considerations since we wanted to RSSE eld, where they also presented the denition which emerged
focus on the core activity of handling source code. as a standard denition of RSSE systems. They focused on discussing
Using the Goal-Question-Metrics template proposed by Basili what RSSEs do for developers, what their design dimensions are, and
et al. (1994, 2014), we formulate the research goal as follows: what the limitations and potentials of RSSE systems are. In the ex-
amples, Robillard et al. mentioned eight tools. They emphasized the
Object of study: Study recommendation systems for software cold-start problem3 , the need for recommendations explanations,
engineering and the lack of variance in the output, since RSSEs mostly recommend
Purpose: To characterize them source code. They recognized proactive tools and tools that can adapt
Focus: With respect to the produced recommendation,
to users feedback as promising directions for RSSE development.
the benets of this recommendation for the
user, the required input, and the required The book Recommendation Systems in Software Engineer-
effort to provide the input by the user ing (Robillard et al., 2014) presents the current research on RSSE
Viewpoint: From the view point of the software engineer systems, focusing on system design, implementation, and evaluation
Context: In the context of tasks that involve the use of
aspects of RSSEs. It is a comprehensive collection of techniques and
source code
approaches used in the eld, without a particular focus on existing
tools.
In this study we do not distinguish between different software en- We are not aware of any comprehensive and systematic review
gineering roles (e.g., software design, coding, testing) as we are inter- of RSSE tools. With this SLR we ll the gap and summarize imple-
ested to obtain an overview of what recommendation systems can mented RSSE systems that were presented in scientic journals and
achieve and what kind of effort they expect in the context of the cre- conferences.
ation and maintenance of source code.
We formulated the following research questions characterizing 3. Methodology
relevant attributes of the object of study with respect to the fo-
cus (Basili et al., 1994): This section summarizes the methodological approach we used to
identify the rst set of papers and to lter out the relevant papers for
1. What output do RSSE systems produce?
our research questions.
2. Which benets does a software engineer get from using RSSE sys-
We adopted the approach proposed by Kitchenham (2004) for
tems?
conducting software engineering SLRs; it is used to identify, evaluate
3. What input data do RSSE systems require?
and interpret all available research relevant to a particular research
4. Which effort does a RSSE system require by the software engi-
question, or to a topic area, or to a phenomenon of interest. It is rec-
neer?
ognized and commonly used by the software engineering community
The outcome of this systematic literature review can help practi- (Aleti et al., 2013; Ali et al., 2010; Dieste and Juristo, 2011; Hall et al.,
tioners and researchers to obtain an overview of how RSSE systems2 2012; Niazi et al., 2013; Salleh et al., 2011; Unterkalmsteiner et al.,
are used and which research challenges still exist. 2012).
Following Kitchenhams guidelines, we dened a protocol before
1.2. Denition of the term recommendation system for software the execution of the SLR and followed it during the execution. The
engineering protocol contains research questions, the search strategy, the inclu-
sion and exclusion criteria, the data extraction strategy, the synthe-
According to Happel and Maalej (2008), RSSEs provide devel- sis strategy, the study quality assessment, and the project timetable.
opment information (source code, artifacts, quality measures, and Fig. 1 depicts the process, which we describe below.
tools) and collaboration information (people, awareness, and status On 12th April 2013 we queried four digital libraries: IEEE Xplore,4
and priorities). After obtaining and evaluating relevant decisions they ScienceDirect,5 the ACM Digital Library,6 and Scopus.7 We selected
help software engineers to nd relevant information (Robillard et al., these libraries, because they include all high quality software engi-
2010). We are using the following denition of a recommendation neering journals and proceedings of conferences.
system for software engineering (RSSE): To formulate the actual queries to extract the data from the se-
lected digital libraries, we framed the research questions using the
A recommendation system for software engineering is a soft-
ware application that provides information items estimated to be
3
Cold-start problem denotes a situation in which a recommender system does not
have enough data (yet) to make quality recommendations.
4
2
The term RSSE system is a tautology since the word system is already contained IEEE Xplore, http://ieeexplore.ieee.org.
5
in the abbreviation RSSE. Nevertheless, we use the term since it was introduced by ScienceDirect, http://www.sciencedirect.com.
6
the founders of this line of research in Robillard et al. (2010) and is used as well by its ACM Digital Library, http://dl.acm.org.
7
research community. Scopus, http://www.scopus.com.
M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113 103

Fig. 1. Research process.

PICOC (Population, Intervention, Comparison, Outcome, Context) cri- To maximize recall, we did not include the comparison, outcomes,
teria as suggested by Petticrew and Roberts (2005) and Kitchenham and context criteria in the search term. We consider the context crite-
et al. (2007): rion in exclusion criteria 1 and 2, while the outcomes and the compar-
ison criteria dened our data extraction strategy (the comparison cri-
Population: software development activities.
terion denes what we consider the benet of using a recommender
Intervention: the use of recommendation systems to support the
system in software engineering).
creation or modication of source code.
IEEE Xplore returned 139, ScienceDirect 2, Scopus 189, and ACM
Comparison: the situation in which the software engineer creates
23 search results. We were interested only in the publications de-
or modies source code without the support of recommendation
scribing recommendation systems related to software development,
systems.
particularly the ones that support source code development phases.
Outcomes: characterization of the produced output, the therefore
We dened the following exclusion criteria:
expected benet, and the expected effort to provide the required
input. 1. Papers from elds different from software engineering, e.g., e-
Context: any context in which software engineers create or mod- commerce or health-care recommendation systems.
ify source code. 2. Papers describing tools that do not support source code based de-
velopment, e.g., tools supporting a requirements elicitation or a
We used the following query to obtain all results for the popula-
division of work.
tion and the intervention:
3. Non-primary studies.
(recommendation systems OR recommendation system OR 4. Papers describing unimplemented solutions, e.g., a new approach
recommender systems OR recommender system) AND ( soft- or a new algorithm.
ware development OR software developer OR software engi- 5. Papers presenting implementations that are not tools, e.g.,
neering) reusable source code libraries.
104 M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113

Fig. 2. Filtering approach to lter 43 tools out of 260 papers.

The exclusion criterion 3 is a typical feature of systematic litera- were (Antunes et al., 2012a, 2012b; Niu et al., 2012, 2013; Cubranic
ture reviews, i.e., systematic literature reviews assess primary studies et al., 2004, 2005). Fig. 2 shows the ltering approach we used.
(Kitchenham and Charters, 2007). The exclusion criteria 4 and 5 are We prepared digital forms to accurately record any information
based on the quality consideration that we wanted to consider only needed to answer the research questions. We extracted these data
papers where the authors present already implemented recommen- categories:
dation systems and not hypothetical ones or only support tools for
recommendation systems.
Out of 353 search results, 260 were unique. We excluded 32 1. Alternatively, the development phase or developers action:
results that were not papers published in journals or conferences Development phase: the main development phase that the
(labeled not studies in Fig. 2). Afterwards, based on reading the tool supports, e.g., the implementation phase;
title and the abstract of the papers, we excluded 86, based on Developers action: a specic developers action that is used
the dened exclusion criteria (they all matched the rst exclusion independently from the phase or as a part of the phase, e.g.,
criterion). the search for a reusable component;
Out of 142 publications that were left for the full-text reading, we 2. Problems that developers are facing: issues that developers want
could not access 9. According to Kitchenham (2004), in a systematic to address;
literature review it is necessary to avoid language bias, but we could 3. Potential problem addressed by the system: the authors claim of
not satisfy this condition, since we were only able to analyze papers what will happen if the suggested tool is not used. This aspect is
in English. This resulted in exclusion of 2 other publications. related to the previous aspect 2;
After reading and analyzing 131 papers, we obtained 46 relevant 4. Recommendation and information types in output: the items pre-
papers, that contained descriptions of 43 unique tools. The duplicates sented to the developer;
M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113 105

Table 1
Output generated by the analyzed recommendation systems.

Output Publications Product Process Resource

Binary source code les Ashok et al. (2009)


Changes in the deployment environment Holmes and Walker (2010)
Design patterns Palma et al. (2012)
Digital documents that might be interesting for the software Andric et al. (2004)
engineer
Estimated artifact costs Holmes et al. (2009)
Estimated defect density Lopez and van der Hoek (2011)
Help to perform changes Christidis et al. (2012); Kobayashi et al. (2012); Cubranic et al.
(2004, 2005); Niu et al. (2012, 2013)
Messages posted on software engineering forums Bacchelli et al. (2012); Cordeiro et al. (2012); Cubranic et al.
(2004, 2005)
Method parameters present in the API Zhang et al. (2012)
Necessary source code changes due to a previous modication Gall et al. (2009); Lozano et al. (2011)
Possible experts, mentors, and collaborators Ashok et al. (2009); Canfora et al. (2012); Christidis et al. (2012);
Lopez and van der Hoek (2011); Moraes et al. (2010); Schuler
and Zimmermann (2008)
Problem reports Ghanam et al. (2012); Cubranic et al. (2004, 2005)
Reusable test steps Landhausser and Genaid (2012)
Source code churn Lopez and van der Hoek (2011)
Source code that might be interesting for the software engineer Antunes et al. (2012a, 2012b); Ashok et al. (2009); Barbosa et al.
(for different development environments, for different (2012); Danylenko and Lowe (2012); Denninger (2012);
development phases, for different types of source code, for Holmes (2009); Holmes et al. (2009); Ichii et al. (2009); Kim
different types of requirements, or for different types of users) et al. (2013); Malheiros et al. (2012); Mccarey et al. (2005);
McMillan et al. (2012); Murakami and Masuhara (2012); Pavn
and Gmez-Sanz (2003); Piorkowski et al. (2012); Shimada
et al. (2009); Cubranic et al. (2004, 2005); Wang et al. (2011);
Zagalsky et al. (2012)
Source code that needs to be changed Arcoverde et al. (2012); Dagenais and Robillard (2008);
Kawrykow and Robillard (2009); Juergens et al. (2009)
Source code transformations Dotzler et al. (2012)
Suggested improvements to the source code Muslu et al. (2012)
Suggested tools Viriyakattiyaporn and Murphy (2009)
Visited methods Piorkowski et al. (2012)
Words extracted from documentation and source code Piorkowski et al. (2012); Wang et al. (2011)

5. Base output form: how the recommendations are visualized. They 4. Results and analysis
can be visualized in different ways and they can be interactive. We
were interested only in the initial output; This section describes the results of our study; we discuss sepa-
6. Input: all artifacts needed to produce a recommendation; rately for each research question the answers we found in the identi-
7. Behavior: the behavior of the system is either reactive or proactive ed literature.9
(see Section 4.4);
8. Type of effort: prerequisites that a developer has to explicitly sat-
isfy, to make the system work or to obtain the recommendations8 ; 4.1. Research question 1: What output do RSSE systems present to the
9. Publication title; user and how?
10. Publication authors;
11. Publication year; We identied different types of information and recommenda-
12. Name of the tool. tions that RSSE present to their users and grouped them into 20 cat-
egories as in Table 1.
During the synthesis phase, we normalized the terms describing We further classied the found RSSE systems following the clas-
the same phenomenon and we continued to use the most common sication of Fenton and Peeger (1998) into product, process, or re-
term. We built a taxonomy using these terms. Since, due to time con- source RSSEs:
straints, each paper was read only by one author and, consequentially,
we could not evaluate extraction accuracy, we sent emails to all au-
If the different papers describe an RSSE that describes, criticizes,
thors of the papers, asking if they are willing to evaluate the accu- or recommends source code or any other type of artifact that is
racy of the (normalized) extracted data related to their paper. Ten handled during the software development process, we classied
authors of the papers were willing to collaborate, and, as discussed the RSSE as a Product RSSE. Product RSSEs generate recommen-
in Section 5, the accuracy was high. The subsequent merge of basic dations with the intention to use the recommendation during the
actions and phases, input and output types, and required type of ef- creation of source code.
fort into bigger groups was pair-reviewed by the two authors of this
If the RSSE describes, criticizes, or recommends a process to edit
paper. artifacts, we classied it as Process RSSE.
Finally, if the described RSSE describes, criticizes, or recommends
a human or technical resource that might be useful, we classied
the paper as Resource RSSE.

8
We are only interested in actions that are performed to use an RSSE. Actions that
9
would be performed regardless of a usage of a RSSE, e.g., programming, are not taken The used extracted raw data is publicly available at: http://sites.google.com/site/
into account. mgasparic/slr.
106 M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113

Table 2
Types of output produced by the identied systems.

Type of output Publications

Not explained in the paper Christidis et al. (2012); Schuler and Zimmermann (2008)
Change guide graph Kobayashi et al. (2012)
Dashboard Juergens et al. (2009)
Document Danylenko and Lowe (2012); Ghanam et al. (2012); Juergens et al. (2009)
Documents with source Kim et al. (2013)
code examples
List Andric et al. (2004); Antunes et al. (2012a, 2012b); Arcoverde et al. (2012); Ashok et al. (2009); Barbosa et al. (2012); Canfora et al.
(2012); Cordeiro et al. (2012); Dagenais and Robillard (2008); Denninger (2012); Dotzler et al. (2012); Gall et al. (2009); Holmes (2009);
Holmes et al. (2009); Ichii et al. (2009); Kawrykow and Robillard (2009); Landhausser and Genaid (2012); Lozano et al. (2011);
Malheiros et al. (2012); Mccarey et al. (2005); McMillan et al. (2012); Murakami and Masuhara (2012); Muslu et al. (2012); Niu et al.
(2012, 2013); Piorkowski et al. (2012); Shimada et al. (2009); Cubranic et al. (2004, 2005); Viriyakattiyaporn and Murphy (2009); Wang
et al. (2011); Zagalsky et al. (2012); Zhang et al. (2012)
No direct output Moraes et al. (2010)
One result Palma et al. (2012)
Sliced circular view Lopez and van der Hoek (2011)
Source code les Pavn and Gmez-Sanz (2003)
Table Bacchelli et al. (2012); Holmes and Walker (2010)

Product 38
Class

Process 14
Resource 8
0 5 10 15 20 25 30 35 40
Number of papers
Fig. 3. Distribution of obtained papers according to their classication into product, process, or resource.

Table 1 reports the output generated by the analyzed recommen- RSSEs is a simple recommendation list containing source code arti-
dation systems. According to our classication, the majority of RSSEs facts. However, we detected also some initiative in the community to
we found recommend source code, e.g., source code that needs to develop RSSE tools that use different output forms, as well as innova-
be changed or source code that might be interesting to reuse. Not tive representations.
only source code is recommended, also other les, either digital doc-
uments that might be interesting or binary les that represent de-
ployed source code.
4.2. Research question 2: Which benets does a software engineer get
Fig. 3 summarizes the distribution of papers according to their
from using RSSE systems?
classication into product, process, or resource (the classes are
overlapping, i.e., articles can belong to more than one class; we count
The authors of the systems claim that their systems can provide
papers only once per class).
different benets to software engineers. Since the systems are tar-
Fourteen papers describe recommendations about the process,
geting highly diverse problems and most of them are limited to the
e.g., lists that help to perform complex changes, reusable testing
specic kinds of actions or development phases, we analyzed the dif-
steps, or the visited methods.
ferences from three different aspects: which phase or action type is
Finally, eight papers recommend human or technical resources
supported by the system, which actual problem the software engi-
that might be useful to the software engineer. This includes experts,
neer is facing, and which potential problem is prevented by the sys-
mentors, or tools that can be consulted during development.
tem.
If we look at how recommendation systems present their results,
As can be seen in Fig. 4, the tools we identied mainly support for
30 tools present recommendations in a list, Bacchelli et al. (2012)
the implementation and maintenance phase (the classes are overlap-
and Holmes and Walker (2010) present them in a table, and Palma
ping, i.e., articles can belong to more than one class; we count papers
et al. (2012) show only one result. Five tools output documents; Kim
only once per class). As implementation we consider a development
et al. (2013) uses documentation documents and adds source code ex-
phase in which mainly new functionality is created, as maintenance
amples to them, Danylenko and Lowe (2012) creates a plan, Ghanam
a phase in which mainly existing code is modied, and as informa-
et al. (2012) and Juergens et al. (2009) create a report, and Pavn and
tion gathering we classied papers that support learning activities,
Gmez-Sanz (2003) generates a source code. Juergens et al. (2009)
help a newcomer joining the team, help to nd experts, code exam-
output also dashboards. Kobayashi et al. (2012) and Lopez and van der
ples, or to navigate through various pieces of information in the IDE.
Hoek (2011) output charts; Kobayashi et al. (2012) visualizes a change
We classied those papers that describe their tools as independent
guide graph, while Lopez and van der Hoek (2011) visualizes a sliced
from a specic development phase as phase independent.
circular view. Moraes et al. (2010) does not produce a direct output.
Twelve tools are specically targeting reuse actions, four are tar-
We could not determine an output form of Christidis et al. (2012)
geting debugging actions, and four are helping newcomers that are
and Schuler and Zimmermann (2008).
joining the team.
Table 2 shows the different types of outputs of the identied
According to the tools we identied, RSSEs mainly support reuse,
systems.
debugging, implementation, and maintenance phases/activities. The
identied RSSEs attempt to support the improvement of system qual-
4.1.1. Discussion ity, e.g., bug prevention and performance improvement, make devel-
From Table 2 we can see that the ve-year-old statement of opment more ecient or less expensive, lower the cognitive load, and
Robillard et al. (2010) still holds: the predominant output mode of help the software engineer to take better decisions.
M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113 107

Implementation 26

Class
Maintenance 15
Information gathering 9
Phase independent 3

0 5 10 15 20 25
Number of papers
Fig. 4. Distribution of obtained papers according to the development phase they support.

Table 3
Systems addressing a certain problem that the software engineer is facing.

Problem that the software engineer is facing Publications

Ad-hoc approach Niu et al. (2012, 2013)


All dependencies have to be tracked Kobayashi et al. (2012)
All requirements have to be met Danylenko and Lowe (2012)
Architecturally-relevant code anomalies are not detected Arcoverde et al. (2012)
soon enough
Changes have to be introduced Gall et al. (2009); Ghanam et al. (2012)
Changes have to be tracked Dagenais and Robillard (2008); Holmes and Walker (2010)
Clone removal is needed Juergens et al. (2009)
Software engineer does not have enough knowledge Ashok et al. (2009); Bacchelli et al. (2012); Cordeiro et al. (2012); Ichii et al. (2009); Kim et al. (2013);
Landhausser and Genaid (2012); Lopez and van der Hoek (2011); Mccarey et al. (2005); Moraes et al.
(2010); Murakami and Masuhara (2012); Palma et al. (2012); Pavn and Gmez-Sanz (2003);
Viriyakattiyaporn and Murphy (2009); Wang et al. (2011); Zhang et al. (2012)
Dicult to search due to big data sets Andric et al. (2004); Antunes et al. (2012a, 2012b); Mccarey et al. (2005); Piorkowski et al. (2012);
Cubranic et al. (2004, 2005)
Function was not designed to be reused as needed Holmes et al. (2009)
Insucient documentation Andric et al. (2004); Kim et al. (2013); Lozano et al. (2011)
Mentoring is needed Canfora et al. (2012); Malheiros et al. (2012); Cubranic et al. (2004, 2005)
No focus on exception handling Barbosa et al. (2012)
Possible solutions have to be compared Zagalsky et al. (2012)
Reoccurring similar actions Dotzler et al. (2012)
Reusable components missed Kawrykow and Robillard (2009); McMillan et al. (2012); Shimada et al. (2009)
Insucient help from existing tools Christidis et al. (2012); Denninger (2012); Ghanam et al. (2012); Ichii et al. (2009); Kobayashi et al. (2012);
Muslu et al. (2012); Pavn and Gmez-Sanz (2003); Schuler and Zimmermann (2008); Cubranic et al.
(2004, 2005); Zagalsky et al. (2012); Zhang et al. (2012)
Using multiple facts is more ecient Holmes (2009)

Table 4
Systems preventing a certain problem.

Potential problem addressed by a system Publications

Big cognitive load for software engineer Christidis et al. (2012); Danylenko and Lowe (2012); Holmes et al. (2009); Piorkowski et al. (2012)
Bugs are introduced in the system Barbosa et al. (2012); Ghanam et al. (2012); Kobayashi et al. (2012)
Complete redesign of the system is needed Arcoverde et al. (2012)
Software engineer leaves work environment Bacchelli et al. (2012); Cordeiro et al. (2012)
Current practice is expensive Malheiros et al. (2012); Niu et al. (2012, 2013); Zhang et al. (2012)
Current practice is inecient Andric et al. (2004); Antunes et al. (2012a, 2012b); Ashok et al. (2009); Dagenais and Robillard (2008); Denninger
(2012); Dotzler et al. (2012); Ghanam et al. (2012); Holmes (2009); Holmes and Walker (2010); Ichii et al. (2009);
Juergens et al. (2009); Kim et al. (2013); Landhausser and Genaid (2012); Lozano et al. (2011); Mccarey et al. (2005);
McMillan et al. (2012); Moraes et al. (2010); Murakami and Masuhara (2012); Muslu et al. (2012); Niu et al. (2012,
2013); Piorkowski et al. (2012); Shimada et al. (2009); Viriyakattiyaporn and Murphy (2009); Wang et al. (2011);
Zhang et al. (2012)
Lower quality of the system Kawrykow and Robillard (2009); Mccarey et al. (2005); Shimada et al. (2009)
Project delay or failure Canfora et al. (2012)
Wrong decision taken Lopez and van der Hoek (2011); Palma et al. (2012); Pavn and Gmez-Sanz (2003); Schuler and Zimmermann (2008)
System performance issues Barbosa et al. (2012)
Task is unfeasible Cubranic et al. (2004, 2005); Zagalsky et al. (2012)
Usefulness of the system decreases Gall et al. (2009)

In Table 3 we report the problems that RSSEs address or which 4.2.1. Discussion
trigger the activation of an RSSE. Looking at this table, we see that the Most of the systems are specialized to support debugging, main-
most common reason for a usage of a RSSE is a lack of knowledge by tenance, and implementation phases; reuse is well supported by ex-
the software engineer. Moreover, many tools were developed because isting tools as well. A comprehensive support for the testing phase is
the existing tools lack a required functionality or do not perform the not present; we nd it surprising, since it is one of the core phases
given functionality with the necessary quality. in traditional as well as in newer software development approaches
Table 4 shows which problems RSSEs aim to prevent. We can see such as Lean or Agile. Otherwise, the existing RSSEs address relatively
that the most of the tools aim at improving the eciency of the soft- diverse range of kinds of potential development problems, from the
ware engineers. process and product perspective.
108 M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113

Table 5
Inputs used by the analyzed recommendation systems.

Input Publications Product Process Resource

A list of known design patterns or components (Ichii et al., 2009; Kawrykow and Robillard, 2009; Palma
et al., 2012; Shimada et al., 2009)
A list of known tools (Viriyakattiyaporn and Murphy, 2009)
Communication between software engineers (Canfora et al., 2012; Christidis et al., 2012; Moraes et al.,
2010; Cubranic et al., 2004, 2005)
Issue tracking system data (Ashok et al., 2009; Christidis et al., 2012; Denninger, 2012;
Gall et al., 2009; Lopez and van der Hoek, 2011)
Log les (Ashok et al., 2009; Cordeiro et al., 2012)
Meta-data (e.g., annotations, meta models, documentation) (Andric et al., 2004; Christidis et al., 2012; Dotzler et al.,
2012; Juergens et al., 2009; Kim et al., 2013; McMillan
et al., 2012; Moraes et al., 2010; Pavn and Gmez-Sanz,
2003; Piorkowski et al., 2012)
Output generated by external systems (Kim et al., 2013; Muslu et al., 2012)
Publicly accessible source code examples (e.g., in blogs, (Bacchelli et al., 2012; Cordeiro et al., 2012; Mccarey et al.,
forums, repositories) 2005; McMillan et al., 2012; Murakami and Masuhara,
2012; Wang et al., 2011; Zagalsky et al., 2012)
Software development process (actions, their occurrence, (Antunes et al., 2012a, 2012b; Dotzler et al., 2012; Holmes,
sequence, and duration) 2009; Ichii et al., 2009; Kobayashi et al., 2012; Piorkowski
et al., 2012; Cubranic et al., 2004, 2005; Viriyakattiyaporn
and Murphy, 2009)
Source code (in the development environment or present in (Antunes et al., 2012a, 2012b; Ashok et al., 2009; Barbosa
software revision control systems, e.g., CVSa or SVNb ) et al., 2012; Canfora et al., 2012; Christidis et al., 2012;
Dagenais and Robillard, 2008; Denninger, 2012; Dotzler
et al., 2012; Gall et al., 2009; Ghanam et al., 2012; Holmes,
2009; Holmes and Walker, 2010; Juergens et al., 2009; Kim
et al., 2013; Landhausser and Genaid, 2012; Lopez and
van der Hoek, 2011; Lozano et al., 2011; Malheiros et al.,
2012; Mccarey et al., 2005; Moraes et al., 2010; Murakami
and Masuhara, 2012; Niu et al., 2012, 2013; Pavn and
Gmez-Sanz, 2003; Piorkowski et al., 2012; Schuler and
Zimmermann, 2008; Shimada et al., 2009; Cubranic et al.,
2004, 2005; Viriyakattiyaporn and Murphy, 2009; Zhang
et al., 2012)
Tasks (e.g., user stories, bug reports) (Landhausser and Genaid, 2012; Malheiros et al., 2012;
Cubranic et al., 2004, 2005; Viriyakattiyaporn and Murphy,
2009)
Test artifacts (Ghanam et al., 2012; Juergens et al., 2009; Landhausser and
Genaid, 2012)
The deployed system (Kawrykow and Robillard, 2009)
User input (e.g., search terms, a query, request for help, (Andric et al., 2004; Antunes et al., 2012a, 2012b; Arcoverde
settings, preferences) et al., 2012; Ashok et al., 2009; Bacchelli et al., 2012;
Canfora et al., 2012; Dagenais and Robillard, 2008;
Danylenko and Lowe, 2012; Ghanam et al., 2012; Holmes,
2009; Ichii et al., 2009; Kim et al., 2013; McMillan et al.,
2012; Niu et al., 2012, 2013; Palma et al., 2012; Pavn and
Gmez-Sanz, 2003; Cubranic et al., 2004, 2005; Wang
et al., 2011; Zagalsky et al., 2012)
a
http://www.nongnu.org/cvs/.
b
http://subversion.apache.org.

Product 44
Class

Process 30
Resource 27
0 10 20 30 40 50
Number of papers
Fig. 5. Distribution of obtained papers according to the input type they require.

4.3. Research question 3: What input data do RSSE systems use? handled during the software development process, as Process in-
puts if the recommendation system uses data about the software
To answer research question 3, we analyzed the identied papers development process, and as Resource inputs if the recommen-
to understand how the output was generated, i.e., which data is used dation system uses data about the employed human or technical
by the different approaches. resources.
The results are shown in Table 5. As for the outputs, we fol- Fig. 5 summarizes the distribution of papers according to their
lowed again the classication of Fenton and Peeger (1998) and classication into product, process, or resource (the classes are
classied the inputs as Product inputs if the recommendation overlapping, i.e., articles can belong to more than one class; we count
system uses source code or any other type of artifact that is papers only once per class).
M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113 109

Table 6
Systems using a certain behavior.

Behavior Publications

Proactive (Antunes et al., 2012a, 2012b; Cordeiro et al., 2012; Gall et al., 2009; Ghanam et al., 2012; Holmes and
Walker, 2010; Juergens et al., 2009; Kobayashi et al., 2012; Lopez and van der Hoek, 2011; Lozano
et al., 2011; Mccarey et al., 2005; Murakami and Masuhara, 2012; Piorkowski et al., 2012; Shimada
et al., 2009; Viriyakattiyaporn and Murphy, 2009)
Reactive (Andric et al., 2004; Arcoverde et al., 2012; Ashok et al., 2009; Bacchelli et al., 2012; Barbosa et al.,
2012; Canfora et al., 2012; Christidis et al., 2012; Dagenais and Robillard, 2008; Danylenko and
Lowe, 2012; Dotzler et al., 2012; Holmes, 2009; Holmes et al., 2009; Ichii et al., 2009; Juergens et al.,
2009; Kawrykow and Robillard, 2009; Kim et al., 2013; Landhausser and Genaid, 2012; Malheiros
et al., 2012; McMillan et al., 2012; Moraes et al., 2010; Muslu et al., 2012; Niu et al., 2012, 2013;
Palma et al., 2012; Pavn and Gmez-Sanz, 2003; Cubranic et al., 2004, 2005; Wang et al., 2011;
Zagalsky et al., 2012; Zhang et al., 2012)
Not explained in (Denninger, 2012; Schuler and Zimmermann, 2008)
the paper

Proactive 15
Class

Reactive 30
Not explained 2
0 5 10 15 20 25 30
Number of papers
Fig. 6. Distribution of obtained papers according to their behavior classication.

Extensive eort 10
Class

Low eort 19
None 12
Not explained 2

0 5 10 15 20
Number of papers
Fig. 7. Number of systems requiring a certain level of manual effort to function.

We identied fourteen different input types; the most common a reactive presents recommendations automatically; in this way
are source code and manual user input. Ten papers monitor the soft- software engineers do not have to realize themselves that they need
ware development process, i.e., actions that the software engineers recommendations (Robillard et al., 2014). Table 6 reports the classi-
perform, when these actions occur, the sequence of these actions, and cation of all papers into these two groups.
how long they take. Fig. 6 summarizes the distribution of papers of Table 6 (the classes
are overlapping, i.e., articles can belong to more than one class; we
4.3.1. Discussion count papers only once per class).
The type of input and the frequency with which it is read have an We could not identify the classication of Barbosa et al. (2012),
impact on the analysis of the collected data as well as on the devel- Denninger (2012), Schuler and Zimmermann (2008), but the author
opment effort of a particular RSSE. For instance, source code is easier of Barbosa et al. (2012) informed us that it is a reactive tool. In sum-
to parse and analyze than natural language. Almost all existing RSSEs mary, we identied 27 reactive tools, 13 proactive tools, and one tool
take project artifacts as an input data. Because RSSEs cannot crawl that can be used in both ways.
into developers head to obtain the exact context (Robillard et al., Conscius (Moraes et al., 2010) is the only tool that does not present
2010), they try to use the artifacts that are generated during develop- any recommendation directly. When a software engineer submits a
ment. Few systems from our review also observe developers actions, message that has to be addressed by an expert, the tool automatically
e.g., navigation patterns or the time and duration of certain actions, forwards it to a recommended expert. Since software engineers are
or reactions, e.g., users feedback. In this way, RSSEs can only create aware of effects, we can treat their action as an explicit launch of a
general estimations of context, but other approaches would likely re- recommendation process, which makes the system reactive.
quire additional developers effort. In Fig. 7 we visualize what kind of effort is required by the soft-
ware engineer; extensive effort would, in our opinion, require a
4.4. Research question 4: Which effort does a RSSE system require by change in development process, and low effort requires submitting
the software engineer? an explicit request or lling out a short form, e.g., query submission.
We could not identify the required effort of seven systems, but we
We identied the behavior of the identied systems and we as- concluded that ve of them require at least running a program, which
sessed the level of manual effort that the systems require to function we consider equivalent to submitting a request. Another ve tools re-
as expected. quire to submit a request, and one requires either to submit a request
A RSSE is reactive if the user has to explicitly make a call to ob- or to formulate a query. Seven tools require only to submit a query,
tain the recommendations. A proactive system the opposite of and one tool requires to submit a message, which is delivered to the
110 M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113

2003 1

2004 2

2005
1
2006

2007
1
2008
6
2009 4
1
2010 1
1
2011 2
15
2012 6
1
2013

0 2 4 6 8 10 12 14 16
Proactive Reactive
Fig. 8. Distribution of the developed RSSEs over the years.

recommended recipient. Twelve tools, which are all proactive, do not manual effort; others expect that the user issues a request (we call a
require any effort. All other tools require a non-negligible effort, com- request the click on a button or the execution of a menu command)
pared to a query submission. or formulates a query (we call a query a request for information
A prerequisite of the tool described in Danylenko and Lowe (2012) which contains hints on how to retrieve the information) to the RSSE.
is the denition of formal contexts. Pavn and Gmez-Sanz (2003) It is not surprising that most of the systems are reactive, since the
and Dotzler et al. (2012) require manually written annotations in design and the implementation are generally easier; it is dicult not
source code; (Pavn and Gmez-Sanz, 2003) also demands the gen- to cross the line between providing helpful recommendations and
eration of a predened report that translates each automatically spamming. Additionally, standalone tools are reactive almost by the
generated diagram into a PROLOG predicate. A merging scope has denition; one exception is described in the previous section. What
to be dened when scoreRec (Niu et al., 2012, 2013) is used. In we nd surprising is that reactive systems are increasingly popular,
SCOOP (Arcoverde et al., 2012) the user has to set thresholds and de- this can be seen also in Fig. 8. We expected that an advancement in
tection strategies that are used by the tool. Conscius (Moraes et al., the research will encourage the development of proactive systems,
2010) expects the software engineer to submit a message. In De- but our expectations were wrong.
bugAdvisor (Ashok et al., 2009) an expert has to write a feature parser
and submit a query to generate recommendations. The tool described 5. Validity threats
in McMillan et al. (2012) reacts when a product description is in-
serted. In DPR (Palma et al., 2012) the user has to give answers to We used the list of threats dened by Wohlin et al. (2012), which
questions formulated by the tool. Christidis et al. (2012) expects issue includes four types of validity threats, namely, conclusion, internal,
insertion or search execution. construct, and external validity threats.
A RSSE can be proactive even if it requires manual effort to func-
tion as expected. However, it is uncommon: out of 14 proactive tools,
we identied two such systems. In ASPEN (Ghanam et al., 2012), 5.1. Conclusion validity
a user has to dene a model, which is later used together with a
source code and test artifacts during a recommendation process. Nev- Conclusion validity in qualitative studies represents reliability of
ertheless, it is the only standalone proactive system we identied: it conclusions drawn from the collected data (Wohlin et al., 2012). In
mainly runs in a background, and because it generates recommenda- our case, threats are related to potentially too small samples and vio-
tions without an explicit request (after it is started), we classied it lations of assumptions.
as proactive. In Spyglass (Viriyakattiyaporn and Murphy, 2009) a user The used methodology (Kitchenham, 2004) already assumes that
has to select an icon to visualize recommendations. This is another not all relevant primary studies that exist can be identied. As tac-
exceptional case, since the submission of a request is a representative tics, we lowered the effect of this validity threat with the selection
aspect of reactive tools. However, Spyglass generates recommenda- of a query with which we achieved a high recall, at the cost of a low
tions automatically and with a change of an icon only informs the precision. Nevertheless, due to publication bias, positive results are
user that recommendations exist. more likely to be published than negative, which may cause that cer-
tain relevant primary studies were not published. We were not able
4.4.1. Discussion to address this threat.
Less than a third of the tools demand non-negligible effort by the According to Kitchenham (2004), duplicated reports have to be
software engineer, while more than a quarter do not demand any avoided since they could bias the results; the most recent report has
M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113 111

to be taken into account. In one case, we obtained three pairs of pub- the 95% condence, we can say that the extracted data validity is not
lications describing the same tool. All of them have the same authors below 85%.
and the extracted data coincided. The author of one pair of papers ap- We later merged the extracted and normalized data into bigger
proved that they are describing the same tool, but the others did not groups. Since the authors of the papers did not review the merge, we
reply to the survey. We did not exclude the older publication, but we pair-reviewed the taxonomy and the merges, to satisfy Kitchenhams
merged the results and treated them as one publication. requirement.

5.2. Internal validity 5.4. External validity

Internal validity threats are related to possible wrong conclusion External validity threats are related to the ability to generalize the
about causal relationships between treatment and outcome (Wohlin results of the experiment to industrial practice (Wohlin et al., 2012).
et al., 2012). In the case of a literature review, internal validity de- In the case of a literature review, the external validity depends on the
scribes how well the ndings represent the true opinion expressed identied literature: if the identied literature is not externally valid,
in the literature. The primary objective of a systematic literature re- neither is the synthesis of its content. By the choice of our exclusion
view, such as the present one, is to minimize internal validity threats. criteria, we excluded papers that did not already demonstrate a prac-
To address validity threats, researchers use tactics, i.e., actions that tical implementation of the presented idea. This tactics supports ex-
minimize the validity treat. The tactics adopted in a systematic lit- ternal validity.
erature review are those suggested by the approach developed by
Kitchenham (2004) and implemented in this review. Wherever we 6. Conclusions
deviated from the approach of Kitchenham, we mentioned this ex-
plicitly in the related section. When it comes to the outputs of RSSEs, the authors rely on rela-
tively similar types of visualizations, i.e., most of the tools are using
simple lists and there are others which are using tables or charts. We
5.3. Construct validity
believe that investing more research effort in analyzing different and
novel user interfaces could be benecial. Furthermore, it is necessary
Construct validity is related to generalization of the result to the
that researchers consider providing more sophisticated recommen-
concept or theory behind the study execution (Wohlin et al., 2012). In
dations than most of the systems included in this review provide at
our case it is related to potentially subjective analysis.
the moment, i.e., source code artifacts or other raw data, e.g., other
According to Kitchenhams guidelines (Kitchenham, 2004), data
project documents or names of collaborators. It seems that most of
extraction should be performed independently by two or more re-
the authors of the tools are satised with only displaying certain arti-
searchers, which was not the case in our study. On the other hand, not
facts and expecting that the developer will gure out by herself what
all relevant publications contained all the relevant data, and where
to do with this information. That could be one of the main reasons
the behavior of the system was evident, even if it was not mentioned
why RSSE systems developed by the scientic community are almost
explicitly, we used our assessment. Furthermore, during the synthe-
non-existent in practice. We strongly encourage the researchers to
sis phase, we normalized the terms describing the similar phenom-
put more effort into building RSSE systems that can explain recom-
ena using generic terms, from which we also built a taxonomy.
mendations and (to some extent) tell developers what to do.
We minimized these validity threats by asking the authors of pub-
In our opinion, largely unexploited opportunities lie in other kinds
lications to conrm the extracted and normalized data. We received
of potentially relevant information, for example, process related in-
replies related to ten RSSEs. We briey summarize their reactions be-
formation, which could help developers to comply with team or en-
low.
terprise best practices, or improve their own development process.
CloneDetective (Juergens et al., 2009) is useful throughout the im-
We are already investigating some of them, namely, we are analyzing
plementation and maintenance phases, and not only for a clone re-
individual perceptions of software engineers to discover how they
moval, as we presumed; the base output form is a document with a
perceive the usefulness of visualizations of unnished tasks, effort
report, but also dashboards are supported. Moreover, we overlooked
spent per artifact, last software engineers that were engaged in a de-
that models, natural language documents, test cases, and XML les
velopment of a module, etc. Gasparic et al. (2013).
are part of the input used by the system. The authors of Barbosa et al.
Moreover, there is no comprehensive support for the testing
(2012) informed us that their system is reactive. In Piorkowski et al.
phase: we only detected one tool, which supports reuse of test steps.
(2012) we modied problems that software engineers are facing from
Since software testing is a popular research eld at the moment, we
software engineer doesnt have enough knowledge to dicult to
expect that in the future there will be a migration of theoretical nd-
search due to big data sets; we also added associated Javadoc for
ings into practice. Certainly the tool support will be required and we
each recommended method, words most likely to be relevant in the
think that a promising platform for a new kind of RSSE system is on a
recommended methods and Javadoc, and whether the software engi-
horizon.
neer has visited the recommended method in the past to informa-
On the other hand, debugging, maintenance, and implementation
tion and recommendation types in the output. Rascal (Mccarey et al.,
phases and reuse actions are well-exploited research topics in the do-
2005) does not support only code reuse actions, but an entire imple-
main of RSSE systems. We also identied some approaches to sup-
mentation phase; we also had to remove current practice is expen-
port a learning process, search for help or an expert, and merging of
sive from potential problems addressed by the system.
branches in a revision control.
The sample contained ten tools from which we extracted eight
The identied RSSEs are very task specic, but rarely take into ac-
data types.10 Out of 80 elds eight were wrong. The average validity
count broader context that could improve the behavior of the sys-
of the sample is 90% and the samples standard deviation is 0.3. The
tem. For example, we identied some tools that are targeting dis-
sample was quasi-random, based on a voluntary participation of au-
persed development teams or newcomers, but none of them is using
thors. Due to its size we could use the rules of a normal distribution,
an enough sophisticated user model that would allow the adapta-
even though we do not know the variance of the population. With
tion of recommendation visualizations to a specic user, for example;
on the other hand, advanced user models are indispensable in rec-
10
In this set we did not include the basic data, i.e., publication title, authors, year, and ommender systems applied in other domains, e.g., e-commerce. Fur-
name of the tool. thermore, the RSSE systems take into account very limited contextual
112 M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113

information related to the specic situation. For instance, some sys- Bacchelli, A., Ponzanelli, L., Lanza, M., 2012. Harnessing stack overow for the IDE. In:
tems adapt the recommendations based on the source code artifact Proceedings of International Workshop on Recommendation Systems for Software
Engineering.
under development, but none is considering current IDE environ- Barbosa, E., Garcia, A., Mezini, M., 2012. A recommendation system for exception han-
ment. dling code. In: Proceedings of International Workshop on Exception Handling.
To generate more relevant recommendations we suggest exten- Basili, V.R., Caldiera, G., Rombach, H.D., 1994. The Goal Question Metric Approach. John
Wiley & Sons.
sive inclusion of context information, since it improved the quality of Basili, V.R., Trendowicz, A., Kowalczyk, M., Heidrich, J., Seaman, C., Mnch, J.,
recommendations already in other RS domains (Adomavicius et al., Rombach, D., 2014. Aligning Organizations Through Measurement: The
2011). None of the systems included in this review takes into account GQM+Strategies Approach. The Fraunhofer IESE Series on Software and Sys-
tems Engineering. Springer International Publishing.
broader context, which would include different types of information,
Beck, K., Andres, C., 2004. Extreme Programming Explained: Embrace Change.
such as information about the environment, timing, user, etc. In addi- Addison-Wesley.
tion to more targeted and personalized recommendations, such infor- Canfora, G., Di Penta, M., Oliveto, R., Panichella, S., 2012. Who is going to mentor new-
comers in open source projects? In: Proceedings of ACM SIGSOFT International
mation could also enable detection of time intervals when the devel-
Symposium on the Foundations of Software Engineering.
oper can be interrupted with proactive recommendations or it could Christidis, K., Paraskevopoulos, F., Panagiotou, D., Mentzas, G., 2012. Combining
enable creation of new types of RSSEs. activity metrics and contribution topics for software recommendations. In: Pro-
Currently, we are analyzing which context factors may inuence ceedings of International Workshop on Recommendation Systems for Software En-
gineering.
perceived usefulness and perceived ease of use of IDE command rec- Cordeiro, J., Antunes, B., Gomes, P., 2012. Context-based recommendation to support
ommendations. In our initial research, Gasparic and Ricci (2015) iden- problem solving in software development. In: Proceedings of International Work-
tied four types of context: current practice, environment, interac- shop on Recommendation Systems for Software Engineering.
Cubranic, D., Murphy, G.C., Singer, J., Booth, K.S., 2004. Learning from project history: a
tion, and recommendation presentation context. case study for software development. In: Proceedings of ACM Conference on Com-
If it is still too early for the inclusion of advanced context models in puter Supported Cooperative Work.
RSSE systems, we encourage researchers to at least consider the de- Cubranic, D., Murphy, G.C., Singer, J., Booth, K.S., 2005. Hipikat: a project memory for
software development. IEEE Trans. Softw. Eng. 31, 446465.
velopment of context specic RSSEs. For example, the development Dagenais, B., Robillard, M.P., 2008. Recommending adaptive changes for framework
for smart phones is increasingly popular, but there are no RSSE tools evolution. In: Proceedings of International Conference on Software Engineering.
that would support the optimization of source code for energy con- Danylenko, A., Lowe, W., 2012. Context-aware recommender systems for non-
functional requirements. In: Proceedings of International Workshop on Recom-
sumption.
mendation Systems for Software Engineering.
Nevertheless, even if the new RSSE systems will start to collect ad- Denninger, O., 2012. Recommending relevant code artifacts for change requests using
ditional data, we think that the authors of RSSEs should relieve soft- multiple predictors. In: Proceedings of International Workshop on Recommenda-
tion Systems for Software Engineering.
ware engineers of manual effort, even when the required effort is low.
Dieste, O., Juristo, N., 2011. Systematic review and aggregation of empirical studies on
For example, RSSEs expecting a submission of a query can deduct it elicitation techniques. IEEE Trans. Softw. Eng. 37, 283304.
from the context and request a conrmation or propose a generated Dotzler, G., Veldema, R., Philippsen, M., 2012. Annotation support for generic patches.
query as a default value, instead of requiring manual input. In: Proceedings of International Workshop on Recommendation Systems for Soft-
ware Engineering.
Furthermore, we believe that proactive recommending has much Elberzhager, F., Kremer, S., Munch, J., Assmann, D., 2012. Guiding testing activities by
greater potential than reactive, even though reactive RSSEs are more predicting defect-prone parts using product and inspection metrics. In: Proceed-
common than proactive, since software engineers are usually un- ings of EUROMICRO Conference on Software Engineering and Advanced Applica-
tions.
aware that they are wrong, when they are; and in such a case, they Fenton, N.E., Peeger, S.L., 1998. Software Metrics: A Rigorous and Practical Approach.
will probably not consult a reactive system. Proactive RSSEs can in- PWS Publishing.
stantly redirect software engineers from a wrong path, but they Gall, H., Fluri, B., Pinzger, M., 2009. Change analysis with evolizer and changedistiller.
IEEE Softw. 26, 2633.
should not distract them. We hope that the development of proactive Gasparic, M., Janes, A., Hericko, M., Succi, G., 2013. Metrics-based recommendation sys-
RSSEs was not completely discouraged by the failure of the infamous tem for software engineering. In: Proceedings of Collaboration, Software and Ser-
Clippy, which was not only due to the proactive behavior (Whitworth, vices in Information Society.
Gasparic, M., Ricci, F., 2015. Modeling context-aware command recommendation and
2005).
acceptance in an IDE. In: Proceedings of International Workshop on Context for
This SLR provides information about existing RSSE tools and im- Software Development.
plemented approaches that were presented in scientic journals and Ghanam, Y., Maurer, F., Abrahamsson, P., 2012. Making the leap to a software platform
strategy: issues and challenges. Inf. Softw. Technol. 54, 968984.
conferences, and reveals some research gaps still present in the RSSE
Hall, T., Beecham, S., Bowes, D., Gray, D., Counsell, S., 2012. A systematic literature re-
eld. It can be used as a starting point for new researchers, as well view on fault prediction performance in software engineering. IEEE Trans. Softw.
as a motivation for new developments in this increasingly popular Eng. 38, 12761304.
research eld. Happel, H.-J., Maalej, W., 2008. Potentials and challenges of recommendation systems
for software development. In: Proceedings of International Workshop on Recom-
mendation Systems for Software Engineering.
References Holmes, R., 2009. Do developers search for source code examples using multiple facts?
Adomavicius, G., Mobasher, B., Ricci, F., Tuzhilin, A., 2011. Context-aware recommender In: Proceedings of Workshop on Search-Driven Development-Users, Infrastructure,
systems. AI Mag. 32, 6780. Tools and Evaluation.
Aleti, A., Buhnova, B., Grunske, L., Koziolek, A., Meedeniya, I., 2013. Software architec- Holmes, R., Ratchford, T., Robillard, M., Walker, R., 2009. Automatically recommending
ture optimization methods: a systematic literature review. IEEE Trans. Softw. Eng. triage decisions for pragmatic reuse tasks. In: Proceedings of IEEE/ACM Interna-
658683. tional Conference on Automated Software Engineering.
Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K., 2010. A systematic review of Holmes, R., Walker, R., 2010. Customized awareness: recommending relevant external
the application and empirical investigation of search-based test case generation. change events. In: Proceedings of International Conference on Software Engineer-
IEEE Trans. Softw. Eng. 36, 742762. ing.
Andric, M., Hall, W., Carr, L., 2004. Assisting artifact retrieval in software engineering Ichii, M., Hayase, Y., Yokomori, R., Yamamoto, T., Inoue, K., 2009. Software
projects. In: Proceedings of ACM Symposium on Document Engineering. component recommendation using collaborative ltering. In: Proceedings of
Antunes, B., Cordeiro, J., Gomes, P., 2012a. An approach to context-based recommen- Workshop on Search-Driven Development-Users, Infrastructure, Tools and Eval-
dation in software development. In: Proceedings of ACM Conference on Recom- uation.
mender Systems, pp. 171178. Juergens, E., Deissenboeck, F., Hummel, B., 2009. Clonedetective a workbench for
Antunes, B., Cordeiro, J., Gomes, P., 2012b. SDiC: context-based retrieval in eclipse. In: clone detection research. In: Proceedings of International Conference on Software
Proceedings of International Conference on Software Engineering. Engineering.
Arcoverde, R., Macia, I., Garcia, A., Von Staa, A., 2012. Automatically detecting Kawrykow, D., Robillard, M., 2009. Improving API usage through automatic detection
architecturally-relevant code anomalies. In: Proceedings of International Work- of redundant code. In: Proceedings of IEEE/ACM International Conference on Au-
shop on Recommendation Systems for Software Engineering. tomated Software Engineering.
Ashok, B., Joy, J., Liang, H., Rajamani, S.K., Srinivasa, G., Vangala, V., 2009. DebugAdvi- Kim, J., Lee, S., Hwang, S.-W., Kim, S., 2013. Enriching documents with examples: a
sor: a recommender system for debugging. In: Proceedings of Joint meeting of the corpus mining approach. ACM Trans. Inf. Syst. 31, 127.
European software engineering conference and the ACM SIGSOFT Symposium on Kitchenham, B., 2004. Procedures for Performing Systematic Reviews. Keele University,
the Foundations of Software Engineering. Keele, UK.
M. Gasparic, A. Janes / The Journal of Systems and Software 113 (2016) 101113 113

Kitchenham, B., Charters, S., 2007. Guidelines for Performing Systematic Literature Robillard, M., Walker, R., Zimmermann, T., 2010. Recommendation systems for software
Reviews in Software Engineering. Technical Report. Software Engineering Group, engineering. IEEE Softw. 27, 8086.
School of Computer Science and Mathematics, Keele University. Robillard, M.P., Maalej, W., Walker, R.J., Zimmermann, T., 2014. Recommendation Sys-
Kitchenham, B., Mendes, E., Travassos, G.H., 2007. Cross versus within-company cost tems in Software Engineering. Springer, Berlin, Heidelberg.
estimation studies: a systematic review. IEEE Trans. Softw. Eng. 316329. Rodrguez-Gonzlez, A., Lagares, A., Lagares, M., Gomez-Berbis, J., Alor-Hernandez, G.,
Kobayashi, T., Kato, N., Agusa, K., 2012. Interaction histories mining for software change Cortes-Robles, G., 2010. Recometh: using CBR and characteristic weights to rec-
guide. In: Proceedings of International Workshop on Recommendation Systems for ommend a software development methodology in software engineering. In: Pro-
Software Engineering. ceedings of IEEE International Conference on Software Engineering and Service
Landhausser, M., Genaid, A., 2012. Connecting user stories and code for test develop- Sciences.
ment. In: Proceedings of International Workshop on Recommendation Systems for Salleh, N., Mendes, E., Grundy, J., 2011. Empirical studies of pair programming for CS/SE
Software Engineering. teaching in higher education: a systematic literature review. IEEE Trans. Softw. Eng.
Lopez, N., van der Hoek, A., 2011. The code orb: supporting contextualized coding 37, 509525.
via at-a-glance views (NIER track). In: Proceedings of International Conference on Schuler, D., Zimmermann, T., 2008. Mining usage expertise from version archives. In:
Software Engineering. Proceedings of International Working Conference on Mining Software Reposito-
Lozano, A., Kellens, A., Mens, K., 2011. Mendel: source code recommendation based ries.
on a genetic metaphor. In: Proceedings of IEEE/ACM International Conference on Shimada, R., Hayase, Y., Ichii, M., Matsushita, M., Inoue, K., 2009. A-score: automatic
Automated Software Engineering. software component recommendation using coding context. In: Proceedings of In-
Malheiros, Y., Moraes, A., Trindade, C., Meira, S., 2012. A source code recommender ternational Conference on Software Engineering - Companion Volume.
system to support newcomers. In: Proceedings of IEEE Annual Computer Software Unterkalmsteiner, M., Gorschek, T., Islam, A.K.M.M., Cheng, C.K., Permadi, R.B., Feldt, R.,
and Applications Conference. 2012. Evaluation and measurement of software process improvement a system-
Mccarey, F., Cinnide, M.O., Kushmerick, N., 2005. Rascal: a recommender agent for atic literature review. IEEE Trans. Softw. Eng. 38, 398424.
agile reuse. Artif. Intell. Rev. 24, 253276. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D., 2003. User acceptance of informa-
McMillan, C., Hariri, N., Poshyvanyk, D., Cleland-Huang, J., 2012. Recommending source tion technology: toward a unied view. MIS Q. 27, 425478.
code for use in rapid software prototypes. In: Proceedings of International Confer- Viriyakattiyaporn, P., Murphy, G., 2009. Challenges in the user interface design of an
ence on Software Engineering. IDE tool recommender. In: Proceedings of Workshop on Cooperative and Human
Moraes, A., Silva, E., da Trindade, C., Barbosa, Y., Meira, S., 2010. Recommending ex- Aspects on Software Engineering.
perts using communication history. In: Proceedings of International Workshop on Wang, L., Fang, L., Wang, L., Li, G., Xie, B., Yang, F., 2011. APIExample: an effective web
Recommendation Systems for Software Engineering. search based usage example recommendation system for java APIs. In: Proceedings
Murakami, N., Masuhara, H., 2012. Optimizing a search-based code recommendation of IEEE/ACM International Conference on Automated Software Engineering.
system. In: Proceedings of International Workshop on Recommendation Systems Whitworth, B., 2005. Polite computing. Behav. Inf. Technol. 24, 353363.
for Software Engineering. Wohlin, C., Runeson, P., Hst, M., Ohlsson, M., Regnell, B., Wessln, A., 2012. Experimen-
Muslu, K., Brun, Y., Holmes, R., Ernst, M., Notkin, D., 2012. Improving IDE recommen- tation in Software Engineering. Springer.
dations by considering global implications of existing recommendations. In: Pro- Zagalsky, A., Barzilay, O., Yehudai, A., 2012. Example overow: using social media for
ceedings of International Conference on Software Engineering. code recommendation. In: Proceedings of International Workshop on Recommen-
Niazi, M., Ikram, N., Bano, M., Imtiaz, S., Khan, S., 2013. Establishing trust in offshore dation Systems for Software Engineering.
software outsourcing relationships: an exploratory study using a systematic liter- Zhang, C., Yang, J., Zhang, Y., Fan, J., Zhang, X., Zhao, J., Ou, P., 2012. Automatic parameter
ature review. IET Softw. 7, 283293. recommendation for practical API usage. In: Proceedings of International Confer-
Niu, N., Yang, F., Cheng, J.-R., Reddivari, S., 2012. A cost-benet approach to recom- ence on Software Engineering.
mending conict resolution for parallel software development. In: Proceedings of
International Workshop on Recommendation Systems for Software Engineering. Marko Gasparic is a computer science Ph.D. Student at the Free University of Bozen-
Niu, N., Yang, F., Cheng, J.-R., Reddivari, S., 2013. Conict resolution support for parallel Bolzano. His research interests lie in recommendation systems for software engineer-
software development. IET Softw. 7, 111. ing with a particular interest in improving the productivity of software developers by
Palma, F., Farzin, H., Gueheneuc, Y., Moha, N., 2012. Recommendation system for design recommending useful IDE commands. Mr. Gasparic holds a master degree in computer
patterns in software development: an DPR overview. In: Proceedings of Interna- science from the University of Maribor, Slovenia.
tional Workshop on Recommendation Systems for Software Engineering.
Pavn, J., Gmez-Sanz, J., 2003. Agent oriented software engineering with INGENIAS. Andrea Janes is a researcher at the Free University of Bolzano-Bozen. His research
In: Proceedings of Central and Eastern European Conference on Multi-agent Sys- interests include Lean software development, value-based software engineering, and
tems. empirical software engineering. Dr. Janes holds a doctorate degree in informatics from
Petticrew, M., Roberts, H., 2005. Systematic Reviews in the Social Sciences: A Practical the University of Klagenfurt, Austria.
Guide. Blackwell Publishing.
Piorkowski, D., Fleming, S.D., Scadi, C., Bogart, C., Burnett, M., John, B.E.,
Bellamy, R.K.E., Swart, C., 2012. Reactive information foraging: an empirical inves-
tigation of theory-based recommender systems for programmers. In: Proceedings
of SIGCHI Conference on Human Factors in Computing Systems.

Das könnte Ihnen auch gefallen