Sie sind auf Seite 1von 76

Master Thesis

June 19, 2014

Personal Analytics

Two Studies on Software Developers’ Perceptions of Productivity

on Software Developers’ Perceptions of Productivity André Meyer of Uster, Switzerland (09-736-398) supervised

André Meyer

of Uster, Switzerland (09-736-398)

supervised by

Prof. Dr. Thomas Fritz

software evolution & architecture lab
software evolution & architecture lab

Master Thesis

Personal Analytics

Two Studies on Software Developers’ Perceptions of Productivity

Two Studies on Software Developers’ Perceptions of Productivity André Meyer software evolution & architecture lab

André Meyer

software evolution & architecture lab
software evolution & architecture lab

Master Thesis

Author:

André Meyer, info@andre-meyer.ch

URL:

www.andre-meyer.ch

Project period:

01/14/2014 - 07/14/2014

Software Evolution & Architecture Lab Department of Informatics, University of Zurich

Acknowledgements

I would like to thank all the people who were involved in this thesis. First of all, I thank Prof. Dr. Thomas Fritz for giving me the opportunity to write this thesis at the software evolution and architecture lab (s.e.a.l.) and for his excellent support. Moreover, I want to thank Prof. Dr. Gail Murphy (University of British Columbia) and Dr. Thomas Zimmermann (Microsoft Research) for their valuable comments, suggestions, help, and ideas. Further appreciation goes to the members of the s.e.a.l. group for always being open for a discussion or being pilot participants. I also would like to express many thanks to my parents, my girlfriend, and my friends who gave me insights and comments from other non IT-related perspectives and motivated me throughout my work.

Disclaimer

Parts of this thesis work were worked on in collaboration with Prof. Dr. Gail Murphy (Univer- sity of British Columbia) and Dr. Thomas Zimmermann (Microsoft Research). In particular, the chapters on related work (Chapter 2), results for the first study (Chapter 3), results for the second study (Chapter 4), and parts of the discussion (Chapter 5). Together we submitted a paper to the FSE14 conference [MFMZ14], which was accepted in June 2014.

Abstract

The better the software development community becomes at creating software, the more software the world seems to demand. One way to address the gap between software demand and supply is to try to increase the supply, by optimizing the productivity of software developers. Although there is a large body of research about measuring and investigating productivity from an orga- nizational point of view, there is a paucity of research about how software developers, those at the front-line of software construction, think about, assess, and try to improve their productivity. To investigate software developers’ perceptions of software development productivity, we con- ducted two studies: a survey with 379 professional software developers to help elicit themes and an observational study with 11 professional software developers to investigate emergent themes in more detail. In the survey we found that software developers perceive their days as produc- tive when they complete many or big tasks, without significant interruptions or context switches. Yet, the observational data we collected shows our participants performed significant task and activity switching, while still feeling productive.

We analyze such apparent contradictions in our findings and use the analysis to propose ways to better support software developers in a retrospection and improvement of their productivity through the development of new tools and the sharing of best practices. Based on the finding that there might not be a single and/or simple measurement for a software developer’s productivity, we discuss a new idea to provide a software developer with a meaningful retrospective analysis of his workday and workweek and its possible impact on a developer’s productivity.

Zusammenfassung

Je besser die Software Entwicklungs-Branche bei der Herstellung von Software wird, desto mehr Software scheint die Welt zu fordern. Eine Möglichkeit um die Lücke zwischen Software Nach- frage und Angebot zu verkleinern, ist das Optimieren der Produktivität von Software Entwick- lern. Obwohl es eine Vielzahl an aktueller Forschung zum Messen und Untersuchen von Produk- tivität aus organisationsorientierter Sicht gibt, gibt es einen Mangel an Forschung die beschreibt, wie Software Entwickler, jene an der Front des Software Entwicklungs-Prozesses, über Produktiv- ität denken, ihre Produktivität einschätzen, und was sie tun, um ihre Produktivität zu verbessern. Wir haben zwei Studien durchgeführt, um die Auffassung der Produktivität eines Software En- twicklers zu untersuchen: Eine Umfrage mit 379 professionellen Software Entwicklern um Leit- motive zu eruieren, und eine Beobachtungs-Studie mit 11 professionellen Software Entwicklern, um diese Leitmotive im Detail zu erforschen. In der Umfrage haben wir festgestellt, dass En- twickler sich produktiv fühlen, wenn sie viele oder grössere Aufgaben ohne erhebliche Unter- brüche und Kontext-Wechsel erfüllen können. Die in der Observations-Studie erfassten Daten zeigen, dass Teilnehmer zwar signifikante Wechsel zwischen Aufgaben und Aktivitäten gemacht haben, sich aber trotzdem produktiv fühlen.

Wir analysieren diese scheinbaren Gegensätze in unseren Ergebnissen und nutzen die Erken- ntnisse um Wege aufzuzeigen, wie Software Entwickler besser bei der Retrospektive ihrer Arbeit und Verbesserung ihrer Produktivität unterstützt werden könnten. Dazu beschreiben wir beste- hende und mögliche neue Anwendungen, sowie bewährte Praktiken unserer Studien-Teilnehmer. Basierend auf der Erkenntnis, dass es möglicherweise kein einzelnes und/oder einfaches Mass für die Produktivität eines Software Entwicklers gibt, beschreiben wir eine neuartige Idee, um einem Software Entwickler eine aussagekräftige retrospektive Analyse seines Arbeitstages und seiner Arbeitswoche anzubieten, und diskutieren mögliche Auswirkungen auf die Produktivität des Software Entwicklers.

Contents

1 Introduction

1

2 Related Work

3

3 Study 1: Survey

7

3.1 Participants and Method

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

7

Results

3.2 .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

8

3.2.1

A Productive Workday

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

8

3.2.2

Productive and Unproductive Activities

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

9

3.2.3

Assessing Productivity

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

10

3.2.4

Measuring Productivity

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

11

3.2.5

Influence of Goal Setting on Productivity

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

13

3.3 Threats to Validity

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

15

4 Study 2: Observations

 

17

4.1 Participants and Method

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

17

Results

4.2 .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

20

4.2.1

Tasks

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

20

4.2.2

Activities

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

22

x

Contents

 

4.2.3 Work Flow

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

23

4.2.4 Goal Setting and Planning

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

24

4.3

Threats to Validity

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

25

5 Discussion And Future Work

 

27

5.1 Setting Goals

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

27

5.2 Retrospective Analysis

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

29

5.3 Reducing Context Switches

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

35

6 Conclusion

 

39

Contents of the CD-ROM

47

Online Survey Questionnaire

 

49

Observation Log

 

55

Interview Protocol

57

Contents

xi

List of Figures

3.1

Developers’ Productivity Satisfaction.

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

10

3.2

Metrics for Assessing Productivity.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

12

4.1

Developers’ Activity and Task Switches in the Two Observation

 

21

5.1

Overview of a Possible Retrospective Analysis of a Developer’s Workday.

.

.

.

.

.

.

30

5.2

Design Prototype of a Possible Implementation of the Retrospection

 

34

List of Tables

 

3.1

Sample Survey Questions.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

8

3.2

Top 5 Reasons for a Productive Workday.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

8

3.3

Top 5 Productive and Unproductive Activities.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

9

3.4

The Information Participants Want to Measure.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

13

3.5

How the Participants Monitor their Goals.

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

14

4.1

Observation Study

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

18

4.2

Sample Observation Log Entries.

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

18

4.3

Activity Categories for the Observations.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

19

5.1

Selection of Possible Tracked Measurements for the

 

32

5.2

Possible Higher-Level Measurements for the Retrospection.

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

33

xii

Contents

Chapter 1

Introduction

“When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.”

Lord Kelvin (William Thomson), 1883.

There is a common refrain that repeats itself in similar forms every few years: the inability for enough software to be produced to satisfy the needs of the world. In 1968, attendees at the first NATO software engineering conference coined the term software crisis [NR69]. In 1972, Dijkstra wrote about “programming [becoming a] gigantic problem” [Dij72]. In 1987, Brooks described how technological improvements can help but will not solve all (essential) problems of software, as there is no one silver bullet [Bro87]. In the same year, Boehm wrote about the growing demand for software [Boe87]. In 1996, Gibbs wrote about the chronic software crisis [Gib94]. And in 2011, Andressen wrote about software eating the world, expressing that the need for software keeps outstripping the ability to produce the software [And].

There are a couple of ways of addressing the gap between software demand and supply. We could try to reduce the demand, namely the world’s appetite for software. This approach seems unlikely to succeed. Or, we could try to increase the supply, namely our ability to produce soft- ware. In this thesis, we consider one way to address supply: how we might improve the produc- tivity of software developers.

A substantial amount of research into the meaning of software productivity has been under- taken over the past four decades (Chapter 2). Much of this research introduces particular defi- nitions of productivity, such as computing productivity based on the number of source lines of code per hour [DKMT96]. Another body of research considers organizational issues associated with productivity, such as the effect of the workplace on developer 1 performance [DL85]. There is also research focused at specific tools and approaches for improving productivity. For exam- ple, the Personal Software Process (PSP) aims to help software developers improve their skills and quality of work by tracking a number of measurements, including schedule data [Hum96a].

1 The term developer, participant and user can describe a female or male person. For simplicity reasons the male form was used throughout this thesis.

2

Chapter 1. Introduction

Surprisingly, there has been no work that we have been able to find that considers when software developers perceive themselves to be productive and when they perceive themselves as unpro- ductive. This information can help inform how productivity is defined, measured, assessed and supported by tools and best practices.

In this thesis, we investigate developers’ perception of productivity. In particular, we examine the following research question:

RQ1 What is software developers’ perception of productivity and how does it manifest itself in their work?

To answer this research question, we gather data about software developers’ perceptions of productive and unproductive work through two studies: a survey (Chapter 3) and an observa- tional study (Chapter 4). The survey we conducted had 379 responses from individuals with an average of 9.2 (±7.3) years of professional software development experience. Our analysis of the survey responses found, amongst other results, that developers think about productive days in terms of ones in which many small or one big task are completed without significant context switching or interruptions.

The observational study we conducted involved observing 11 professional software devel- opers with an average of 5.4 (±3.4) years of professional software development experience from three companies at work for four hours each. During the developers’ normal workday, we col- lected detailed logs of the tasks worked on and the programs used to perform work, gathering a total of 2650 log entries. We also performed semi-structured follow-up interviews of the partic- ipants in this study. From this data, we found that significant context switching between tasks and activities can occur with developers still perceiving themselves as productive. The number of task switches we observed, 13.3 (±8.5) per hour on average is similar to other reports in the literature [GM04, KAP + 07]. The number of activity switches (switches between activities such as programs or meetings) is larger than reported in previous studies at 47 (±19.8) times per hour on average. From this study, we gained insights into the complexities involved with helping developers assess their own productivity.

We conclude the thesis with a discussion of ways in which we might help developers retro- spect upon their productivity, reduce context switches, and share best practices (Chapter 5).

This thesis makes the following contributions:

1. It presents results from a survey of 379 professional software developers, describing their perceptions of when their work is productive and what measurements of productivity they might find useful,

2. it presents the results of an observational study of 11 professional software developers, showing that productive work can occur in the presence of some kinds of fast context switches, and

3. it synthesizes the results of these two studies to share best practices and suggest ways in which we might better support developers in reflecting upon their productivity and reduc- ing context switches.

Chapter 2

Related Work

In this chapter, we analyze and discuss related work with respect to three categories: definitions of productivity, studies on productivity, and retrospection.

Definition of Productivity

about efficiency, inputs and outputs. As one example, Card defines productivity as the ratio of output produced to the resources consumed (input) [Car06]. Often, the unit of input is time- based. Most research in software engineering defines productivity along similar lines; here are some examples:

Definitions of productivity share characteristics of typically being

number of modification requests and added lines of code per year [MFH02],

number of tasks per month [ZM10],

number of source lines of code per hour [DKMT96],

number of function points per month [Jon94],

number of lines of code per person month of coding effort [BSVW96],

amount of work completed per reported hour of effort for each technology [KMB + 96],

ratio of produced logical code lines and spent effort [HA05],

average number of logical source statements output per month over the product develop- ment cycle [LZ97],

total equivalent lines of code per person-month [NHB11],

resolution time defined as the time, in days, it took to resolve a particular modification request [CHC08], and

number of editing events to number of selection and navigation events needed to find where to edit code [KM06].

In this thesis, we do not attempt to define productivity for developers, but instead, we investigate how developers themselves think about productive versus non-productive work.

The majority of the work we could find about software development productivity focused on the organizational, not the personal, level. DeMarco and Lister found evidence that char- acteristics of the workplace and organization have significant influence on the performance of

4

Chapter 2. Related Work

programmers [DL85]. Boehm looked at large-scale possibilities for improving outputs, such as hiring well and using better languages and tools [Boe87]. Blackburn et al. correlated lines of code per person month to aspects such as program size and team size from survey data collected from Western Europe, Japan and US, finding that time-to-market correlated with higher productivity, while larger teams tend to have lower productivity [BSVW96]. Through a systematic literature review, Wagner and Ruhe distilled a list of the main factors influencing productivity, separating them into technical (e.g., product complexity) and soft (e.g., manager capability) factors [WR08].

There are much fewer studies on productivity at the level of an individual developer where the focus is on the study of productivity, not on the effect of introducing a new tool or process to improve productivity. One of the few recent studies is by Kamma and Jalote who recorded and analyzed the screens of highly productive developers (as identified by managers) to identify characteristics of what makes developers productive when performing model-based testing ac- tivities. They found differences in the behaviors of the highly productive developers that could be captured as best practices, such as that high productivity programmers moved all required information to a common place and avoided referring to multiple documents when writing test cases.

Studies on Productivity To address the previously described gap between software demand and supply, one could try to increase the supply, namely our ability to produce software. One way to address supply is to improve the productivity of software developers, as suggested with many approaches (e.g., Extreme Programming [BA04], Pair Programming [PSHH04], Getting Things Done [All03]). A few approaches have been aimed more specifically at improving productivity. The most notable of these is the Personal Software Process (PSP), which aims to help individuals improve their skills and quality of work by collecting (often manually) a set of basic PSP mea- surements such as time, size, quality (defects), and schedule data [Hum96a, Hum96b]. Johnson et al. [JKA + 03] observed a “PSP adoption problem,” which they attributed to the high overhead of PSP-style metrics collection and analysis, and the requirement that PSP users switch between product development and process recording. To eliminate overhead and context switching, they introduced Hackystat which fully automates both data collection and analysis. Our work is or- thogonal to PSP and Hackystat in that we explore what it means to developers to be productive and how they perceive productivity. This information can inform the design of new tools focused on aspects of productivity that developers care about the most.

In the last twenty years, an increasing number of studies was conducted about how devel- opers work. Perry and colleagues’ paper that collected time diaries from developers and per- formed direct observation of developers at work to see where they spent their time was one of the first [PSV94]. They found that work was typically performed in two hour chunks, progress on a particular development task could end up being blocked for various reasons, over half of the de- velopers’ time was spent in interactive activities other than coding and there were seven unique personal contacts per day on average. The data we present in this thesis, on the observational study we performed, also found situations where developers were blocked (also similar to Ko et. al. [KAP + 07]), but we saw much less time being spent on any one task. Other observational based studies have focused on specific areas of development, such as novice programming [BS08], pro- gram comprehension [RTKM12], software dependencies [dSR08], information needs [KAP + 07], and change tasks [SMDV06]. The work we report on in this thesis is unique in considering the condition under which developers perceive they are productive.

5

One aspect that might have a particularly big impact on a developer’s productivity are in- terruptions. A lot of work has been conducted to discuss interruptions, for example the influ- ence of instant messages [CCH00], alerts and notifications [IH07], or the fragmented nature of tasks [GM04,MGH05] on developers’ productivity. Researchers found that developers are inter- rupted more often the earlier it is in the day [MICJ14] and in open plan offices [DMG11], and that they are more easily distracted and interrupted, if they are doing boring or rote work [MICJ14]. The two studies we report on in this thesis help describe more general work practices about when developers see themselves as productive.

Self-Monitoring and Retrospection In his book on persuasive technologies, Fogg describes how self-monitoring technologies assist people in understanding their behavior and help them to achieve predetermined goals [Fog03]. This area of persuasive technology and quantified self is growing within research and industry, which can be seen on the example of the wide adoption of commercial wearable devices for tracking health- and fitness-related activities. Research showed how activity trackers, such as the FitBit [fit], Nike+ Fuelband [fue], or Jawbone Up [jaw], succeed in encouraging short-term behavior change, such as taking the stairs or going out more [FHMZ14, RRMC14], by monitoring ones goals. They found that simple and useful metrics combined with meaningful visualizations and positive feedback motivated users to maintain activity levels. For example, the creators of the Nike+ Fuelband tried to abstract the collected data into the notion of a proprietary “Fuel”, to indicate the user’s progress towards the predefined goals.

Can the success of these activity trackers be replicated to the area of software engineering? We found a couple of tools that visualize several metrics tracked of a developer’s workday, such as Codealike [cod], RescueTime [res], and the Eclipse Activity Report Statistics [ecl]. They all track the programs the developer uses on his computer or the actions he takes in the IDE and visualize them, often on a timeline. None of them supports specifying the measurements that are tracked or setting a personal goal to observe the progress towards it.

In common with all these forms of measuring and self-monitoring is that they need to of- fer flexibility with the selection on the measurements to be tracked [CHW04], respect privacy concerns [US08, FHMZ14, TS10], assure the integrity and accuracy of the data [US08, RRMC14, FHMZ14], and be aware of the fact that many users are dishonest and will try to “play” the mea- surements if they know what is measured [US08, BVvD12]. This thesis suggests a new approach to provide a developer with a retrospection on how work was performed, and might help him to change his behavior towards improving productivity.

Chapter 3

Study 1: Survey

To gain a broad sense of what productivity means to software developers, how developers assess their productivity, and to answer our research question, we conducted an exploratory study. The

results of this first study, an online survey, are presented in this chapter and discussed in chapter

5.

3.1 Participants and Method

The online survey had 28 questions: 8 on the participants’ background and experience, 7 on per- ceptions of productivity, 7 on goal setting and monitoring, 3 on techniques for improving and monitoring productivity, and 3 on the raffle participation and dissemination of results. 16 of the 28 questions had a closed set of answers from which a participant selected, while 12 of the ques- tions were open-ended. Table 3.1 shows an example of a closed and an open-ended question. None of the questions were obligatory and a participant was allowed to drop out at any time. Depending on a participant’s answers, some questions were filtered and not presented to avoid asking unnecessary questions. The complete online survey can be found in the Appendix. We an- nounced the online survey on Twitter and in several big online developer forums, including two big German-speaking IT news portals. 1 We also advertised within Microsoft, sending a person- alized email to 1500 Microsoft employees and directing them to a special internal posting of the survey. To incentivize participants, we held a raffle for the online participants to win two 200 US$ Amazon gift certificates and one for the Microsoft participants to win two 50 US$ Amazon gift certificates. We ended up with 379 valid responses: 185 from the general advertisements and 194 from within Microsoft. Of the 185 participants that completed the survey from the general adver- tisement, the majority of the participants were from Germany (32%) and Switzerland (16%). Of all 379 participants, more than 93.4% of the participants listed their job as software developer; the remaining 6.6% reported having experience in software development. The average professional software development experience per participant was 9.2 (±7.3) years.

8

Chapter 3. Study 1: Survey

Table 3.1: Sample Survey Questions.

Q10

Are you satisfied with your productivity last week? (very unsatisfied,

Q11

unsatisfied, undecided, satisfied, very satisfied) How did you assess if you were productive last week?

3.2

Results

We begin by describing when and under what conditions the participants perceived their software development activities as productive. We then describe how participants think they could assess or measure their productivity. Finally, the influence of goal setting on participants’ productivity is described.

3.2.1 A Productive Workday

In the survey, we asked each participant to complete the sentence “I have a productive workday

” in up to three different ways. Table 3.2 summarizes the top five reasons mentioned by

participants for having a productive workday. Most participants responded that their workday is productive when they complete tasks or make progress on tasks, as stated by 192, 53.2%. The second most mentioned reason for having a productive workday is getting into a “flow” without having many “context-switches” and with having no or few interruptions and distractions (182,

50.4%):

I have a productive workday when I have not been stuck in stupid errors [, when I] could work concentrated and focused. (Web30) Different kinds of distractions were described, ranging from interruptions from co-workers ask- ing questions, phone calls, infrastructure issues, such as waiting for a build to complete, to dis- rupting background-noise in the office. While only a few (8, 2.2%) participants stated that a pro- ductive workday can include meetings, having no meetings is a reason for a productive workday for 79 (21.9%) participants. The fourth most often mentioned reason for having a productive workday is having clear goals and specified requirements (72, 19.9%). 62 (17.2%) participants responded to have a productive workday when they plan their workday beforehand.

when

Table 3.2: Top 5 Reasons for a Productive Workday. (#: number of participants, %: percentage of partici- pants)

I had a productive workday when I

#

%

.

complete tasks or goals

192

53.2

.

have no or few interruptions and distractions

182

50.4

.

have

no meetings

79

21.9

.

have clear goals and/or the requirements are set

72

19.9

.

plan my workday beforehand

62

17.2

3.2 Results

9

3.2.2 Productive and Unproductive Activities

Other questions focused on developers’ perceptions of productivity on a finer-grained level. For example, we asked participants in two open-ended questions about the activities they consider particularly productive or unproductive in a workday, as summarized in Table 3.3.

Unsurprisingly, the activities mentioned most frequently as particularly productive were cod- ing related, including implementing new features, testing, bug fixing, and code reviews (236, 71.5%). Despite most participants considering coding activities as productive, some participants (47, 14.2%) did mention certain unproductive aspects; these participants mostly differentiated be- tween productive coding activities, such as implementing a new feature, and unproductive ones, mainly debugging, testing and code maintenance.

Many other activities had mixed responses as to whether they were productive or unproduc- tive. Most participants (191, 57.9%) think meetings are unproductive and a waste of time. How- ever, 57 (17.3%) considered formal and informal meetings as productive, particularly when the meetings include decision making, have a clear focus and goals, improve relationships between developers, help others, and when all meeting attendees are well-prepared. The main reasons stated for unproductive meetings were missing goals, lacking a preparation, too many partici- pants attending the meeting, or going over time.

Participants’ attitudes towards emails are also mixed. 62 (18.8%) participants considered time spent on email as unproductive, while a few (10, 3.0%) stated reading and writing emails as a productive activity. The amount of emails and some inefficiencies in using emails as a mean of communication appear to be reasons contributing to this activity being seen as unproductive:

Tracking and responding to tons of emails [and] email communications going back and forth for days to- gether with no closure. (MS23)

Table 3.3: Top 5 Productive and Unproductive Activities. (#: number of participants, %: percentage of participants)

 

#

%

Productive activities

coding (implementing new features, testing, bug fixing, code reviews) meetings planning documentation and reports designing or modeling an architecture (i.e. solution for a programming problem)

236

71.5

57

17.3

25

7.6

22

6.7

18

5.5

Unproductive activities

meetings emails unplanned work (solving problems, fighting fires, unplanned tasks, randomization) coding (testing, debugging, maintenance) documentation and reports

191

57.9

62

18.8

58

17.6

47

14.2

25

7.6

10

Chapter 3. Study 1: Survey

# of Par cipants

Other activities mentioned as being productive included planning (25, 7.6%), writing doc- umentation and (administrative) reports (22, 6.7%), modeling or designing an architecture (18, 6.7%), or reading (6, 1.8%). Planning was also seen as an unproductive activity by some (20, 6.0%), as it requires a lot of time and effort without directly making progress on the specific tasks. Unplanned work, such as solving unexpected problems and “fighting fires”, were mentioned as being unproductive by 58 (17.6%) of the participants. Other unproductive activities included some specific coding activities (58, 17.%), documentation and (administrative) reports (25, 7.6%), waiting for others (24, 7.2%), and surfing the web for private social networking or news reading (17, 5.2%).

200

150

100

50

0

workday last week
workday last week

workday

last week
last week
last week
last week
last week
last week

last week

last week

5 = very sa sfied

4 = sa sfied

3 = neutral

2 = unsa sfied

1 = very unsa sfied

Figure 3.1: Developers’ Productivity Satisfaction.

3.2.3 Assessing Productivity

When asked to rate their productivity on the previous workday and workweek on a Likert-scale from 1 (very unsatisfied) to 5 (very satisfied), most participants were satisfied with their work (see Figure 3.1, median of 4, mean of 3.42). Only a few participants mentioned being very satis- fied (7.7%) or very unsatisfied (2.1%). The participants reported assessing their productivity in a variety of ways. Most (242, 78.1%) assessed their productivity based on the tasks they completed or made progress on in the past workday or workweek. As task sizes vary, developers often work on a task for several days or weeks. To reflect about their productivity on these bigger tasks, they either split them into smaller sub-tasks or think about the progress on the task at hand. Several participants (86, 27.8%) mentioned other measurements, such as lines of code, number of com- mits, number of bugs found or fixed, number of test cases written and code reviews completed and number of emails sent. 19 (6.1%) of the participants compare different aspects of their work- day or workweek to assess their productivity, such as the time per task ratio, the executed versus the planned test cases, or the amount of work they did versus the work they could have done. Others (43, 13.9%) stated that they use their feelings to assess their productivity:

How I felt about my work. Usually I feel awesome if I was productive, otherwise I feel pretty terrible.

(MS131)

3.2 Results

11

3.2.4 Measuring Productivity

Following the previous questions, we asked participants about which measurements might be helpful to them to assess their productivity. In particular, we asked participants to rate 23 possible measurements on a five point Likert-scale, with 1 - they strongly disagree that the metric would help them to assess their personal productivity and 5 - they strongly agree. We identified the 23 measurements in an iterative process taking into account related work and responses from our pilot survey participants. The results are presented in figure 3.2.

The measurement with the highest rating is “The number of work items (tasks, bugs) I closed.” with a mean of 3.88 (±1.22). This supports our findings described above that participants assess their productivity often based on their tasks or work items completed. Overall, the means of all metrics lie between 2 and 4 with an overall mean of 3.14 (±0.35). 16 metrics were rated 3 or higher while only 7 metrics have values slightly below 3, showing that while the number of work items closed might be considered helpful by a wide range of participants, there is no single metric that is considerably better than others to assess a developer’s productivity, similar to [Car06]. Furthermore, participants usually rated several metrics as helpful, and differed in which metrics they considered more helpful, suggesting that measuring one’s productivity is an individual task that varies across developers.

To reduce the possibility of suggesting the wrong measurements for the previous question, we asked participants in an open-ended question what they would want to measure to meet their goals and improve their productivity. 221 (89.1%) of the participants who answered the question mentioned one or multiple measurements. Thereof, 57 (25.8%) mentioned two or more mea- surements, which again underlines that there might not be one single measurement to assess a developer’s productivity. The mentioned measurements varied a lot across participants. The top five mentioned pieces of information participants wanted to measure are presented in Table 3.4. 67 (27.0%) participants are interested in better knowing how they spend their time on their com- puter, in meetings, or taking breaks, as well as in comparing their time spent coding with the time spent in meetings, the time spent on personal tasks and the total work time. Similar to the find- ings from previous questions, 44 (17.7%) participants wish they could measure the actual work done or how they progressed on their tasks. Participants were also interested in the value of their work: 41 (16.5%) participants mentioned that performing useful, necessary, and interesting work and having the feeling of being necessary to the team or product is very important. They want to contribute to the success of a project and want their co-workers to be aware of their skills. Other measurements mentioned to meet ones goals or to improve ones productivity included the time per task ratio (39, 15.7%), the number of distractions and interruptions (36, 14.5%), and other specific metrics from issue trackers, testing tools, and code review tools (29, 11.7%).

27 (10.9%) participants explicitly stated that they would not want to be measured, as they either think measuring decreases their productivity, have privacy concerns, or think it is not pos- sible to accurately measure productivity:

I don’t think you could measure productivity. If I wrote 1000 lines of codes it could be less productive than 100 lines. Closing one complex ticket could be more productive as closing 10 simple tickets. And the eval- uation of complexity is very individual. I don’t think you could monitor productivity in the IT-business.

I don’t build cars on the assembly line and at the end of the day, I can’t see how much cars I produced.

(Web82)

12

Chapter 3. Study 1: Survey

Knowing the following would help me assess my personal produc vity.

1 = strongly disagree

2 = disagree

3 = neutral

4 = agree

5 = strongly agree

The number of test cases I wrote.

The number of API methods I learned each day.

The number of emails I wrote. The number of lines of code that I changed per day.

The number of code reviews I’ve contributed to. The number of work items I created that were fixed.

The me that I spent browsing the web for work related informa on.

The number of work items I created.

The me I spent in mee ngs. The number of code reviews I’ve signed off.

The me it took me on average to respond to email.

The number of code elements (e.g. packages or classes) that I changed.

The me that I spent wri ng code.

The me I spend reviewing code.

The number of work items (tasks, bugs) I closed.

The number of commits I made.

The me that I spent in each code project or package. The number of test cases I wrote that subsequently failed.

The number of mee ngs I a ended.

The me I have spent on each work item.

The number of code elements that I changed for the first me.

The me it takes me on average to sign off on code reviews.

The me that I spent browsing the web for personal ma ers during work.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
me that I spent browsing the web for personal ma ers during work. 0% 10% 20%
me that I spent browsing the web for personal ma ers during work. 0% 10% 20%
me that I spent browsing the web for personal ma ers during work. 0% 10% 20%
me that I spent browsing the web for personal ma ers during work. 0% 10% 20%
me that I spent browsing the web for personal ma ers during work. 0% 10% 20%

Figure 3.2: Metrics for Assessing Productivity.

3.2 Results

13

Table 3.4: The Information Participants Want to Measure. (#: number of participants, %: percentage of participants)

 

#

%

activities (how much time was spent where) achievements (actual work done, progress on tasks) value (usefulness of completed work, success of the feature, value to the customer) time per task ratio number of context switches and distractions

67

27.0

44

17.7

41

16.5

39

15.7

36

14.5

You can measure almost everything, the question is, is it effective to measure it? - Because it costs a lot

of time to measure it and to evaluate the results. (

because every project is so unique and the complexity to test all circumstances will explode. (Web95)

Participants also fear that a measuring tool could be misused to track their performance and result in adjustment of their salary or goals according to these measurements.

But I think those things can’t be easily automated,

)

3.2.5 Influence of Goal Setting on Productivity

One aspect that might have a particularly big impact on a developer’s productivity is goal setting and planning, as research found that it can motivate a person to work towards the defined goals, resulting in a higher chance to meet them [RRMC14, Coo14]. Goal setting and monitoring goals is widely practiced among our study participants and mostly leads to positive results, which are presented in the following paragraphs.

Of the 293 (77.3%) participants that regularly set goals for themselves, the majority set goals on a daily (187, 49.3%) and/or weekly (223, 58.8%) basis. These goals are usually fairly concrete, such as fixing a bug, coding a function or completing the documentation, and mainly differ in their granularity. While daily goals are often smaller items, such as fixing a bug or meeting a person, weekly goals are often bigger tasks, such as finishing or creating a feature or a new component, specified in the Scrum planning:

[Daily goals:] implement specific features, like adding a set of GUI elements, get clarity on an aspect of the problem, like find out how to implement something. [Weekly goals:] finish part of projects, like implement a service. (Web132) Goals give developers a direction for their work, help them to better react on unplanned tasks, and to reflect on their work progress. Only 60 (15.8%) participants stated to define long-term goals, such as increasing the code quality or learning a new programming language or framework. 86 (22.7%) mentioned that they do not set any goals at all.

Companies or teams set goals for 50 (13.2%) participants on a daily and for 191 (50.4%) partic- ipants on a weekly basis, often during daily Scrum or sprint meetings. Companies or teams also set long-term goals for 117 (30.9%) participants. These long-term goals are often concrete, but their implementation is relatively vague and developers have some freedom in how to approach them: release component x, learn something new, increase customer feedback and satisfaction, or improve overall quality. 105 (27.7%) participants mentioned that their company does not set any goals for them at all.

14

Chapter 3. Study 1: Survey

Table 3.5: How the Participants Monitor their Goals. (#: number of participants, %: percentage of partici- pants)

 

#

%

planning (e.g. calendar, task list, work done versus work left) issue or work item tracker (tools: e.g. Jira, Redmine, TFS, Atlassian, Bugzilla) code related goal monitoring (quality, successful test runs or builds, cont. integration) meeting (Scrum/standup meeting, meeting with manager) mind work (just thinking/reflecting about the goals)

134

53.2

68

27.0

30

11.9

19

7.5

12

4.8

Participants who set goals for themselves were then asked about their monitoring practice. The top five mentioned goal monitoring practices are presented in Table 3.5. 231 (91.7%) monitor their goals to see if they will achieve or already achieved them. Most participants (134, 53.2%) plan

their goals using a calendar or a simple task list, either in a digital form or on paper, and frequently check if they completed a task, check the status of a task, or check how much work is left to do.

68 (27.0%) of all the participants who are setting goals monitor them using an issue or work item

tracker, such as Jira, TFS, or Bugzilla. Similar to the task list, participants mentioned comparing

their planned open work items or issues with the completed ones. 30 (11.9%) participants use tools to get code specific information to determine the progress on their goals, such as monitoring their code’s quality or running unit or integration tests on their code.

Given the previously mentioned positive influence of goal setting and planning on develop- ers, we also investigated the possible effects of the participant’s goal monitoring practice and if they change their behavior based on what they learn from monitoring. Most participants (189, 77.1%) believe that monitoring their goals can have an impact on their productivity, while 22 (13.9%) believe that it has no effect. The answers from the remaining 22 (9.0%) were not clear.

88 (35.9%) participants believe that goal-setting does not necessarily help them to increase their

productivity directly. They stated that goal-setting provides them with an overview of their tasks, improves clarity on what they need to do, lets them prioritize their work, and better react in case

of unplanned tasks, which then helps them to increase their productivity. Defining and setting goals helps them to work more focused, get less easily distracted by other tasks, and to better know how to resume a task after a context switch. Thinking back and reflecting about their goals helps 64 (26.1%) participants to determine if they are still on track and to find productive and unproductive days:

Monitoring helps to define realistic goals. Consequently, the goals will be met more likely which increases satisfaction. Being more satisfied about the own work makes me more productive. (Web27)

49 (20%) participants believe that monitoring their goals helps them to focus better and work

more efficientOther positive impacts that goal monitoring might have on developers included in- creased satisfaction or mood. Contrary, goal monitoring might also have negative impacts, such as the additional workload that comes with goal setting and monitoring, mentioned by 11 (4.49%) participants), or increased pressure, mentioned by 4 (1.6%) participants.

3.3 Threats to Validity

15

3.3 Threats to Validity

Since the survey participants might not be representative of the general population of software developers, the generalizability of our survey results might be limited. To mitigate this risk, we advertised our survey through various channels to a large audience. Having gathered data from software developers from various countries, and with different levels of open-source and closed source experience, we believe that our sample is fairly representative of software developers and that it provides an interesting perspective. Participants could freely decide whether to participate in the study or not (self-selection). They were informed about the survey’s topics, an estimated duration for the participation and offered an incentive (raffle) for their participation. This could have biased the selection of participants as only participants who could spare enough time or were interested in the incentive might have participated. We tried to mitigate this risk by adver- tising through various channels and offering a very generic incentive.

Chapter 4

Study 2: Observations

The survey results raise many questions: For instance, is there a common meaning between devel- opers for what constitutes a context switch? How, when and what kind of email must developers process? When are meetings productive and unproductive to a developer? To investigate some of the survey results in more depth, we conducted an observational study of software developers at work. As part of this study, we interviewed developers about their work and work practices. To structure this chapter, we extracted four themes from the results:

1. the role of tasks in how developers work and how developers view productivity,

2. the role of goals and planning in their perception of productivity,

3. different views on the effect of various kinds of activities on productivity, and

4. the role of uninterrupted work to be in “the flow”.

4.1 Participants and Method

The observational study involved 11 professional software developers who were recruited from three international software development companies of varying size (Table 4.1). We used personal contacts and a recruiting email to solicit participation. Of the 11 participants, two were female, nine were male and all resided either in the US or Canada. Participants had an average of 5.4 years (±3.4, ranging from 0.5 to 8 years) of professional software development experience and an average of 13.3 years (±8.3, ranging from 1.5 to 29 years) of total development experience. The primary work area of all participants was development and the roles varied between individual contributor and lead.

The study had two parts. In the first part, we observed each participant for a total of four hours on a single workday: two hours before and two hours after lunch. In four cases, we had to adapt the two sessions to accommodate for the participant’s availability, still ensuring to have at least one hour in the morning or afternoon and all four hours on the same day. Before the first session, the observer, the author of this thesis, introduced himself to the participant as well

18

Chapter 4. Study 2: Observations

Table 4.1: Observation Study Participants. (Dev: Development, PM: Project Management, IC: Individual Contributor)

ID

Primary

Role

Development Experience (years) Professional Overall

Work Area

Company R

R1

Dev

IC

8.0

14.0

R2

Dev

Lead

7.5

8.5

R3

Dev

Lead

5.0

11.0

R4

Dev

IC

8.0

13.0

Company S

S1

Dev

IC/Lead

7.5

14.5

S2

PM/Dev

IC/Lead

4.0

22.0

S3

Dev

IC

0.5

1.5

S4

Dev

IC

12.0

29.0

Company T

T1

Dev

IC

2.0

22.0

T2

Dev

IC

2.0

5.0

T3

Dev

IC

3.0

6.0

as to any colleagues working nearby, asking all to ignore him as much as possible. Before ask- ing the participant to continue his workday, the researcher provided a short introduction to the study, placed himself behind the participant to prevent distractions, and ensured being able to read the contents on the participant’s screen. The observer noted in an observation log each time the participant switched a program on his computer, was interrupted by others or interrupted him or herself, or switched a task while being in the same program. All interactions with others were also documented, including details about the topic, duration, place, and persons involved. To ease the transcribing of these events during the observation, we used a small helper applica- tion developed by the author of this thesis, which automatically fills in the system’s time stamp, provides predefined text fields for the event and helps with the analysis of the logged data. The researcher was very careful to capture as much detail as possible. Whenever something was not clear, the researcher noted it and asked the participant at the end of the observation session or in the follow-up interview for clarification. Each entry in the observation log consists of a time stamp, a reason for the switch, the program that the participant switched to and the task that the participant was working on. Table 4.2 shows an example of three observation log entries. Due to confidentiality reasons, we are not able to share the original observation logs. An excerpt of the observation log with more sample data can be found in the Appendix. The preparation and process of the observation was inspired by Mintzberg’s protocol of a structured observation session [Min80].

Table 4.2: Sample Observation Log Entries.

Created

Task Du-

Task Description

Task

Application

Activity Category

ration

Number

09:29:54

00:00:06

builds project modifies code looks at the task (issue)

1

ConsoleApp1

Dev: TestingApplication Dev: Code

09:30:00

00:00:21

1

EditorApp7

09:30:21

00:00:12

1

PlanningApp1 Planning

4.1

Participants and Method

19

Table 4.3: Activity Categories for the Observations.

Category

% of time over all Observations

Development

Code

Reading/editing/navigating code (version control) Debugging Reading/accepting/submitting changes Testing application outside IDE Performing code reviews Other related to development Reading/writing emails Editing work items/tasks/todos; creating/changing calendar entries Reading/editing documents and other artifacts, e.g. pictures Scheduled meeting/call Ad-hoc, informal communication; e.g. unscheduled phone call / IM call, or colleague asking a question Internet browsing related to code/work/task Internet browsing work unrelated Anything else, such as break or changing music

32.3%

Debug

3.9%

VC

2.4%

TestApp

11.7%

Review

2.3%

DevOther

4.1%

Email

4.9%

Planning

7.9%

ReadWriteDoc

2.7%

MeetPlanned

4.9%

MeetInformal

13.1%

BrowsingRel

4.0%

BrowsingUnrel

0.4%

Other

5.4%

We defined a task as a piece of work with a specific goal or intention, such as fixing a bug or reviewing code changes for a specific work item. We inferred tasks from the active programs on the screen and the information considered in those programs, such as a work item consulted before beginning to code. Ten minutes into the first session, the observer briefly interrupted the participant to validate the tasks the participant was working on.

From the observation sessions, we collected transcripts of a total of 44 hours of work with a total of 2650 observation events over all 11 developers. During the data analysis, we categorized each program and participant’s actions into one of the activity categories listed in Table 4.3, which we inferred from the survey results and the observations using an open coding technique. The results are similar to the ones reported in [GM04, Spr84]. The logged time stamps were then used to calculate the duration spent on interruptions, activities and tasks, amongst others.

All observations were conducted by the same observer. For the first session of the first par- ticipant, we cross-checked the observations by having a second observer. Almost all events were noted by both observers. Of the total of 202 entries for the first session, less than 10 log entries (4.9%) were not contained in both observer’s logs, while all others matched, suggesting a high accuracy of the primary observer’s logs.

In the second part of the study, following the observations, we interviewed each participant using a semi-structured approach by using a set of prepared questions as general guidance. As part of this interview, the observer briefly discussed the tasks a participant worked on during the two observation sessions to verify the accuracy of the observation notes. The bulk of the interview focused on gaining further insights into the observations and to investigate themes arising from the survey results. In particular, the questions focused on participants’ perception and reflection of productivity, context switches, work items, code check-ins, meetings and emails and whether and how information on these might help to assess a developer’s productivity. Interviews took between 34 and 51 minutes per participant, with a mean of 43 minutes (±5 minutes), and were conducted in person. The interview guidance notes can be found in the Appendix.

20

Chapter 4. Study 2: Observations

4.2

Results

To present the observational study results, we use the four themes extracted from the survey results; the role of tasks, goals, activities and uninterrupted work on a developer’s perception of productivity. Figure 4.1 illustrates the work of each participant over the two observation sessions. Each participant and session is represented by a row. Each row is divided into segments, with each color-coded (grayscales) segment representing a particular task. The figure also shows the activities undertaken by each participant, discussed in section 4.2.2. The occurrence of the start of an activity is indicated with a colored shape. The lines emanating from the activity indicate the duration of an activity. For two participants, we split the 4 hours of observation into one three hour session and one one hour session. Our analysis of the data in this section considers all of the data collected over the four hours for each participant. A discussion and more details of the data represented in the figure are discussed in the following sections.

4.2.1

Tasks

During the observation sessions, participants worked on between 2 and 10 tasks (mean of 4.8, ±2.3) each, such as implementing a feature, fixing a bug, reviewing code, helping co-workers with a problem, or reading and writing emails. Each participant switched frequently between tasks with a mean task switch rate of 13.3 (±8.5) times per hour. The average time spent on each task was 6.2 (±3.3) minutes. The number of tasks worked on in the time observed (2 to 10) is similar in magnitude to the number of working spheres (10), a concept similar to our use of tasks, observed by Gonzales and Marks in a study that included a small number of software developers [GM04]. The rate of task switching is similar to that reported in both [GM04] and [KAP + 07].

From the observations and the follow-up interviews, we identified several reasons for why task switching occurred, including starting a new task after completing the current one, help- ing a co-worker to make progress (T3), unblocking themselves (S3, S4), similar to [KAP + 07]. In several instances, participants switched tasks without any observable reason, which might have happened due to boredom or even just wanting to switch away from the current task, described as task avoidance by Ko et al. [KAP + 07]. Participants also mentioned that task switches occurred when they were waiting, such as waiting for a build to complete. In these cases, the participants mentioned that switching to email, code reviews, or other small tasks can help increase their productivity.

The results of this study also raise questions about the relationships between tasks and track- able work items. From the survey results, the highest rated measure for assessing productivity (Figure 3.2) was the number of work items closed. From our observations, we found that the participants worked on many more tasks than trackable work items. For example, they have company-wide meetings, help others with their tasks, answer emails, or write reports, which is often not represented in the work items list. When the participants were asked during the inter- view how many work items they worked on during that day, answers ranged from one to 13 and all participants stated that the size of the work items varies. There could be days where you’ve been working on one massive bug or issue or identified other sub-issues into it that need to be resolved, but there could be other days where you just got 20 bugs, but they are all little things. (R4)

4.2 Results

21

Actvities and Task Switches

● ● ● ● R1/1 ● 96 activity switches ● ● ● ● ● ●
R1/1
96
activity switches
● ●
●●●
● ● ● ● ●
28
task switches, 4 distinct tasks
●●
● ● ●●
● ● ● ● ● ● ● ● ● ● ● ● ●●
● ● ●
● ●
●●
● ● ● ● ●
R1/2
64
activity switches
●●
● ● ●
● ●
● ● ●
● ●
40
task switches, 4 distinct tasks
● ●
● ● ● ● ● ●
102
activity switches
R2/1
● ●
61
task switches
● ● ●
● ●
● ● ●
● ●
6
distinct tasks
● ● ● ● ● ●
28
activity switches
R2/2
● ●
19
task switches, 5 distinct tasks
66
activity switches
● ● ●
R3/1
25
task switches
●● ●
● ●
● ● ● ● ●●
4
distinct tasks
5
activity switches
R3/2
2
task switches, 2 distinct tasks
● ●
● ●
108
activity switches
R4/1
●●● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ●
16
task switches, 5 distinct tasks
● ● ● ● ●
● ● ● ● ● ●
● ●
●● ●
● ●
● ●
112 activity switches
R4/2
● ● ● ●
● ●● ● ● ●
● ●
● ● ●
33
task switches, 8 distinct tasks
● ● ● ●
● ● ● ●
● ●
● ● ●
● ●
● ●
● ●●
148
activity switches
S1/1
● ●
● ●
● ● ●●● ●
●●● ●
● ● ●●
●●● ● ● ● ●
●●
●● ● ●
●● ●●
27
task switches, 4 distinct tasks
● ● ● ●
● ●
●●
● ● ● ● ● ● ● ● ● ● ●
Dev:VC
● ●
● ●
141 activity switches
S1/2
● ● ●
●●● ● ●
● ●
● ●
● ● ●
●●
● ● ●● ● ● ● ●● ● ●
Dev:Debug
● ● ● ●●● ● ● ●●
29
task switches, 5 distinct tasks
● ●
● ● ● ●
● ●
● ● ● ● ● ●
● ● ● ● ●
● ● ● ● ● ● ●
● ● ● ●
Dev:Code
Dev:Review
● ●
88
activity switches
S2/1
● ●● ● ● ● ● ● ● ● ● ● ●
● ●
17
task switches, 5 distinct tasks
Dev:TestApp
● ●
● ● ● ● ●
Dev:Other
● ●
151
activity switches
BrowsingRel
S2/2
●●
3
task switches, 6 distinct tasks
● ● ● ● ● ● ● ● ● ● ● ●
● ●
● ● ● ● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●
BrowsingUnrel
● ● ●
● ● ●
MeetInformal
●●
59
activity switches
S3/1
●●
MeetPlanned
20
task switches, 5 distinct tasks
●●
● ● ● ● ●
● ●
●●
● ● ● ●
● ● ● ● ● ● ● ● ● ● ● ●
● ●
Email
Planning
● ● ●
●●
79
activity switches
S3/2
● ●
● ●
● ●
● ● ●
ReadWriteDoc
● ●
7 task switches, 6 distinct tasks
● ● ● ● ●
● ●
Other
● ●
● ● ● ●●●
85
activity switches
S4/1
● ●
●●● ● ●
● ●
●● ●
●●
13
task switches, 4 distinct tasks
● ●
● ●
● ● ●
● ● ● ● ●
37 activity switches
S4/2
● ●● ● ● ● ● ● ●● ●● ●●
●● ●
6 task switches, 3 distinct tasks
● ●
● ●●
● ●
42 activity switches
S4/3
● ●
7 task switches, 6 distinct tasks
● ●
● ●
●●
230 activity switches
T1/1
●●
●● ●
● ●● ● ●
●●
● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ●
●● ● ● ●
● ●● ●
● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●
● ●
79
task switches, 4 distinct tasks
●● ● ● ●
●●
● ●● ●
● ●
● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●
●●●
●●
● ● ●
● ●
● ● ●● ● ● ● ● ● ●
129 activity switches
● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ●
T1/2
● ● ●
● ● ●
● ●
● ● ●
●● ● ● ● ●
● ●
● ● ●
● ● ● ●
53 task switches, 4 distinct tasks
● ●
● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ●
● ●
●●●
●●
166 activity switches
T2/1
●● ● ●
● ● ●● ● ● ●● ● ● ●
● ● ● ● ●
●●●
● ● ● ● ●
● ● ● ● ● ● ●
36 task switches, 3 distinct tasks
● ●
● ● ●
●●
● ●
125 activity switches
T2/2
●● ● ●
●● ● ●
● ●
●● ● ●●
●●● ● ●
● ●
●●
●● ● ● ● ● ●
● ● ●● ●
● ●
● ●
● ●● ● ●
● ●
17 task switches, 2 distinct tasks
● ● ● ●
●●
● ●
51
activity switches
T3/1
● ●
●● ●
● ●
● ● ●●
10
task switches, 3 distinct tasks
● ● ● ● ● ●●
59 activity switches
T3/2
●● ● ● ●
● ●
● ●● ●
● ● ●
● ● ● ●
● ●
● ● ●
●●
9 task switches, 2 distinct tasks
● ●
● ●
0
30
60
90
120
150
180
Time [minutes]
Subject/Session

Figure 4.1: Developers’ Activity and Task Switches in the Two Observation Sessions.

22

Chapter 4. Study 2: Observations

The varying size, granularity, and difficulty of work items likely introduces problems for using the measurement of the number of work items closed as a sole means of productivity assessment, at least for shorter time periods, such as a day. Participants did mention the use of the number of closed work items as one that is used to reflect on a team’s productivity, such as in meetings associated with the end of an agile sprint.

4.2.2

Activities

During the sessions, participants switched activities 47.0 (±19.8) times per hour, spending on average 1.6 (±0.8) minutes on an activity before switching. Table 4.3 shows the percentage of time across the observation periods that developers spent in each activity. Unsurprisingly, given our selection of participants, the largest amount of time was spent on coding (32.3%), with testing the application also taking an average of 11.7% of the observed time for each participant.

Based on a n-gram analysis of the activities recorded, we found that developers most fre- quently switched back and forth between coding and testing. Following coding, the next activity was testing in 28.5% of the cases; a similar result holds for switches from testing to coding. Cod- ing was also often superseded by an informal meeting (14.9%) either because someone asked a question or the participant asked someone else a question. After an informal meeting, develop- ers either went to coding (26.1%), testing (14.8%) or right into planning (18.3%). Finally, emails were mostly checked while coding (18.8%) or testing (19.4%) and developers generally returned to testing (20.7%), coding (17.2%) or planning (19.5%) after the email check.

Informal Meetings. Interestingly, the second highest activity in terms of amount of time spent (Table 4.3) was informal, ad-hoc meetings (13.1%), ranging from instant messages to a colleague interrupting and asking questions. During the interviews, participants described having between one and ten informal meetings per day, taking anywhere from one minute to two hours. The participants in this study agreed that unplanned, informal meetings are usually productive as they are generally short and efficient, and often succeed in helping to unblock the initiator of the conversation. Still, these unplanned meetings can be quite stressful, as both developers want to go back to their own tasks as fast as possible. Usually, participant’s reaction to such an interruption was quite positive, as they consider these informal meetings as an investment to help the co- worker and to move the team forward.

Planned Meetings. Unlike the survey respondents, participants in the observational study found all meetings generally productive, whether informal or formal. Participants in this study de- scribed having had 5 to 10 planned meetings in the past five workdays (usually one to three meetings per day), taking anywhere from 15 to 60 minutes each and uniformly stated that having more than two meetings a day decreases their productivity. Participants did describe formal meet- ings as more productive when only a few people are involved, there is a concrete outcome and the participant feels useful in the meeting. Participants stated that formal meetings are unproductive, if not all participants are needed or if the goal of the meetings is just to deliver information that could easily be distributed via email (e.g. sales meeting).

4.2 Results

23

Emails. Most survey respondents described email activities as unproductive (62, 18.8%) com- pared to productive (10, 3.0%). Given that participants in the observational study spent on aver- age just under 5% of their time handling emails (see Table 4.3), the influence of emails on their productivity might be small compared to other activities. We observed participants regularly checking their emails during the observation, 4.3 (±2.9) times per hour on average:

If someone is trying to contact me and they have a problem, I don’t want them to wait too long. There is this mail clock ticking. At some point you have to go check it. (T1) Participants stated that they do this because they either want to keep their inbox clean or do not want their co-workers to wait too long for answers to their questions. Without being specifically asked, 5 of 11 (45.5%) participants mentioned having issues with emails as they sometimes put

a lot of “pressure” on them, similar to [IH07]. In particular, lengthy email conversations that go back and forth with multiple respondents, and constant questions via email, followed by the urge for an immediate response. All three companies where we run our study handle emails differ- ently: In company T, for example emails are the main mean of communication, whereas company

S usually only uses emails to communicate with people outside the inner team. In many cases,

participants reported that emails are not the best mean of communication for that particular case, as they often prefer a face to face discussion or a quick call. To better handle the masses of emails they receive, participants reported between 5 and 100, many participants set up different filters and tags to clean their inbox and to find important emails faster.

4.2.3 Work Flow

We were surprised by the number of task and activity switches performed by the developers we observed. In particular, as the participants of both studies had expressed concerns that context switches lead to a loss of productivity, while 8 of the 11 (72.7%) participants observed felt that their sessions were productive, even if the made so many switches in the four hours of the observation. To understand the relationships between task switches, activity switches and context switches, we asked the participants in the observational study to define a context switch. The participants predominantly described a context switch as a change in thinking, as in:

When I have to stop thinking about one thing and start thinking about something else. (T1) Participants did go on to discuss the range of context switches from small ones, such as getting distracted due to background noise or switching between programs when working on the same task (R2, R3), to switching between a main task and small other tasks such as code reviews or writing an email (R4) to total switches, switching between two cognitively different tasks (R1).<