Attention and Engagement-Awareness in the Wild:

A Large-Scale Study with Adaptive Notifications

Tadashi Okoshi , Kota Tsubouchi , Masaya Taji , Takanori Ichikawa , and Hideyuki Tokuda
Graduate School of Media and Governance, Keio University
slash@ht.sfc.keio.ac.jp
Yahoo Japan Corporation

AbstractIn todays advancing ubiquitous computing age,
with its ever-increasing amount of information from various
applications and services available for consumption, the man-
agement of peoples attention has become very important. In
particular, the high volume of notifications on mobile devices
has become a major cause of interruption of users. There has
been much research aimed at detecting the opportune moment
to present such information to users with in a way that lowers
the cognitive load or frustration. However, evaluation of such
systems in the real-world production environment with real users
and notifications, and evaluation on users engagement to the
presented notification beyond simple responsiveness have not been
adequately studied. To the best of our knowledge, this study
is the first to investigate user interruptibility and engagement
using a real-world large-scale mobile application and real-world
notifications consisting of actual news content. We equipped the
Yahoo! JAPAN Android app, one of the most popular applications
on the national market, with our mobile-sensing and machine-
learning-based interruptibility estimation logic. We conducted a
large-scale in-the-wild user study with more than 680,000 users
for three weeks. The results show that in most cases delaying the
notification delivery until an interruptible moment is detected is
beneficial to users and results in significant reduction of user
response time (49.7%) compared to delivering the notifications
immediately. We also observed a higher number of notifications
opened in our system as well as constant improvement in user
engagement levels throughout the entire study period.
I. I NTRODUCTION
While the capacity of our attention as humans is constant,
the amount of information available for consumption has been
growing by several orders of magnitude. Concomitant with
advances in computing and multitasking operating systems,
more devices, and more applications and services, increasing
volumes of notifications that proactively convey information
to users are resulting in a greater number of interruptions.
Versatile applications and services in the cloud are being
developed and utilized in this ubiquitous computing age. These
software and services generate enormous amounts of various
types of information for users, such as big data analysis,
schedule reminders, messages from social media friends, the
weather forecast, breaking news, and status updates from
devices. Such information is delivered to users through devices
such as smartphones and other mobile devices, wearable
watches, and even through ambient devices embedded in a
users environment. For better timeliness and speediness, the
provision of such information has progressively become more
proactive, and it is often delivered through push notification
systems.
In this information-overload world, the constant and limited
capacity of human attention has become a new bottleneck [1]
in computing. Push notifications that pop up in the background
of a users attention at random times cause interruptions
and divided attention. There have been several reports on
the negative effects caused by divided attention in terms of
productivity, emotion, and mental state [2], [3], [4], [5].
Researchers have been investigating user interruptibility in
various ubiquitous and pervasive computing situations using
different techniques with the objective of ensuring that inter-
ruptive notifications do not unnecessarily steal users precious
attention resources. Breakpoint [6], the boundary between two
adjacent units of user activities, is known as a timing that
can lower the impact on users cognitive load. We previously
investigated the real-time detection of users breakpoints in
their device interactions and physical activities using mobile
sensing and machine learning techniques on smartphones [7]
and wearable watch devices [8].
However, we found that three significant issues remain
to be studied: (1) real-world evaluation of breakpoint-based
adaptive notification with actual product application and as-
sociated notifications, (2) software architecture design of such
interruptibility estimation for real-world deployment both on
the client and server sides, and (3) comprehensive evaluation
of user behavior in terms of not only interruptibility but also
users further engagement levels with the notification content.
In this paper, we present the results and findings from a
large-scale study conducted on smartphone users interrupt-
ibility and user engagement with a popular real-world smart-
phone application. We designed and implemented our real-time
breakpoint detection and notification scheduling mechanisms
inside the Yahoo! JAPAN Android application [9] (shown
in Figure 1), one of the most popular applications in the
(Front screen (left), Weather radar (center), Notification (right))
Fig. 1. Screenshots of Yahoo! JAPAN Android Application [9]
national application market. Considering several real-world
requirements related to simplicity, scalability, and efficiency,
our mechanism particularly focuses on the users physical-
activity breakpoints [8], relying on activity recognition APIs
on the smartphone platform. Using mobile machine learning
techniques, the detection mechanism embedded in the app
detects the users breakpoints in real time and shows incoming
notifications at such timings.
Our large-scale user study, conducted over three weeks
with a total of 687,840 users, revealed the efficiency of
our proposition. We found that, in most cases, notifications
delivered at delayed breakpoint timings improved the users
overall click timing (earlier). While the notification delivery
delay due to additional breakpoint detection (as opposed to the
conventional deliver immediately style approach) is trivial,
once the notification is delivered, a significantly reduced user
response time (49.7%) was observed in our approach. We also
observed a higher number of notifications opened in our system
as well as constant improvement in user engagement levels
throughout the entire study period.
The contribution of this paper is three-fold. First, we
present the design and implementation of our interruptibility
detection mechanism on a large-scale real-world smartphone
platform. Second, we discuss our large-scale in-the-wild user
study on user interruptibility and engagement conducted using
an actual product and associated notifications in a real-world
situation. Finally, we evaluate our work in terms of not only
interruptibility but also further user engagement with the
presented notification contents. The remainder of this paper is
organized as follows. Section II explains the interruption over-
load problem. Section III discusses related work. Section IV
clarifies our research goals. Section V specifies requirements
for our solution. Section VI presents the system design and
architecture of our system, AtteliaY. Section VII describes our
initial model training study. Section VIII reports on our large-
scale in-the-wild user study conducted with 687,840 users for
three weeks, in terms of our experimental design, methodology,
results, and analysis. Section IX discusses further research
opportunities arising from the user study. Section X concludes
this paper.
II. I NTERRUPTION OVERLOAD
Our current computing life suffers from interruption
overload caused by large numbers of notifications presented
in inappropriate ways. Interruption overload is one class of
a broader information overload problem discussed in the
literature [10], [11], [12]. More studies have recently been
conducted in the context of interruptions and multitasking [13],
[14], [15], [16], [17].
The main source of interruption overload is notifications
from computer system entities such as local operating systems,
messaging services connected to other users, and various
applications. The notification in computer systems was origi-
nally designed to provide newly available information to users
in a more timely and speedy manner (than polling by the
user). Since typical notification systems deliver notifications
immediately to users as soon as they are available, the users
end up facing numerous interruptive notifications from the
background of their current tasks at random timings, regardless
of their timing preference. When a notification is perceived
and recognized by a user, some amount of his/her attention
with limited capacity [18], [19] is allocated to the information
carried by the notification. This situation is called divided
attention [20].
Past studies have been revealing several types of negative
influence of interruptive notification, such as productivity [2],
[3], [4], [5], [21], [22], emotional and social attribution [21],
and psycho-physiological states [3]. Needs for computing sys-
tems that can adapt their behavior to human users attentional
resources have been gradually recognized, with an increasing
number of literatures particularly on sensing users attentional
states.
III. R ELATED W ORK
There are two main targets for sensing a humans current
attentional state: the users current cognitive load and inter-
ruptibility.
A. Sensing Users Cognitive Load
In cognitive psychology, the concept of cognitive load
is defined as the total amount of mental effort allocated to
working memory. Several different approaches for measuring
this load have been proposed, including (a) subjective rating-
based methods, (b) task performance-based methods, and (c)
physiological response-based methods.
Several studies on the subjective rating-based approach
have shown that the measurement of cognitive load through
post hoc self-reporting is a relatively reliable methodology for
mental effort assessment [23]. The most widely used tool for
assessing a users cognitive load is the NASA Task Load Index
(NASA-TLX) [24]. Although use of this method is widespread,
the post hoc nature of the approach makes it difficult to apply
to versatile ubiquitous computing systems where an assessment
needs to be completed in real time.
The measurement of a users task performance is used
to objectively assess the users cognitive load during task
execution. The users performance regarding their primary and
focal task is used in the primary task measurements, whereas
secondary task measurements exploit the performance of a
secondary task (often asked to be) executed simultaneously
with the primary task [23]. In this methodology, the variation
in reaction performance indicates variations in cognitive load.
However, this methodology may not be feasible in ubiquitous
computing situations where a user conducts multitasking with
frequent task switching between multiple tasks, making it
difficult to measure the response performances of the users
various types of tasks using uniform measurement criteria.
The psycho-physiological response-based method includes
several different techniques, such as tracking of eye movement
and pupil size [25], [26], [27], [28], readings from electro-
cardiograms (ECG), galvanic skin response (GSR) [26], [29],
[30], electroencephalogram (EEG) [29], [28], heart rate (HR),
and HR variability (HRV) [31], [32], [28]. Haapalainen et
al. [33] found that, in desktop computing, the combinational
use of an electrocardiogram and heat flux is the most accurate
at classifying low and high levels of cognitive load. Although
this approach looks promising in terms of detecting users
cognitive load in real time, the burden placed on the target
users is not trivial.
B. Sensing Users Interruptibility
Rather than sensing a users cognitive load to represent
that users internal mental status relatively directly, several
researchers have proposed detecting users interruptibility
from the viewpoint of the source of possible interruptions. This
class of research can be categorized into two main groups: (a)
estimation of interruptibility at a certain timing period based
on a users context, and (b) detecting the users breakpoint [6].
The breakpoint is the boundary between two adjacent actions
of a person, and was found to be the timing when interrupting
the user results in relatively lower frustration and cognitive
overhead [21], [34], [35].
Following early research in the desktop computing do-
main [36], [37], [38], [39], [40], more studies have recently
been conducted in the mobile field. Ho et al. used wireless on-
body accelerometers to trigger interruptions when users change
activities [41] and found that interruptions at these transition
times reduce user annoyance. The most recent studies have
been on widespread mobile and smartphone environments.
Fischer et al. identified breakpoints after phone calls and
text messages [42]. They found that users tend to be more
responsive to notifications after these activities than at other
random times. Hofte et al. used an experience sampling
methodology to collect information on location, transit status,
company, and activities in order to build a model of inter-
ruptibility [43], particularly for phone calls. Pejovic et al.
expanded the use of context to detect interruptible moments
on smartphones, including user activity, location, time of day,
and emotional states [44]. Recent studies have even detected
user boredom [45] as yet another opportune moment for
notifications and engagement level[46] as a further indicator
of users response to received information.
At the system level, current situation of fragmented inter-
ruptive notification delivery over mobile network is also known
to be inefficient in power consumption. Acer et al. [47] showed
that delaying notification delivery can yield power savings in
mobile devices.
Our previous works, Attelia I [7] and II [8], have followed
the same research trend in interruptibility on smartphones
and wearable watches (multi-device environment). Towards the
realization of opportune moment detection and on-the-fly adap-
tation in notification scheduling, we particularly emphasized
four design principles: (1) feasibility on users real mobile
and wearable devices, (2) supporting real-time detection, (3)
applicability to diverse types of applications, and (4) affinity
to all-day use. Attelia realizes real-time breakpoint detection
on smartphones and wearable watches without any external
sensors or modification to existing operating systems or appli-
cations.
C. Further Research Challenges
Although the studies cited above show that researchers are
actively working on interruptibility and notification scheduling
on smartphones and wearable devices, there remain significant
research challenges that need to be addressed:
1) To the best of our knowledge, no study has inves-
tigated and evaluated user interruptibility with real
(product-level) applications and the real notifications
issued from such applications. This is primarily ow-
ing to the current situation of major smartphone
platforms (e.g., Android and iOS) not having opened
their APIs to control notifications. Thus, past studies
mainly used a custom sample application and/or re-
lated custom-made notifications prepared in an ad hoc
manner for their research user study. Although some
studies [48], [49] focused on real-world smartphone
notifications, their main contributions pertained to
analysis of the current situations.
2) The system design for such interruptibility detection
and notification adaptation in real situations (with the
real-world applications used by real users) has not
been adequately studied.
3) User engagement for information content presented
via notifications, beyond the users initial responses
to the notifications (such as response time or click
rate), have not been adequately evaluated.
IV. R ESEARCH G OALS AND A PPROACHES
Considering the research issues outlined above, this study
aims to investigate smartphone users interruptibility against
notifications and further engagement against the notified con-
tent in systematically estimated opportune timings. Espe-
cially, this research features such investigation in a real-
world environment with a real application, real users, and real
notifications. To achieve these goals, we took the following
approaches after careful discussion.
A1. Embedding interruptibility estimation logic into a
market-leading smartphone application: We added our in-
terruptibility estimation logic into the Yahoo! JAPAN An-
droid application [9]. Yahoo! JAPAN has been popular in the
Japanese market since its launch in 1996, with a search engine
market share of 32% [50] (its share of Yahoo!s worldwide
market share is 3.4%). The Android app, shown in Fig. 1,
has an installed base of more than 10 million users, making it
one of the most popular smartphone applications. The app is a
portal-like application with several different features including
Web search, news reader, weather map, and links to a variety
of Yahoo! JAPAN services. To the best of our knowledge, we
are the first to conduct interruptibility research with a real-
world application that is utilized by such a large number of
users.
A2. Using real-world notifications: Along with the applica-
tion, we use real notifications issued from Yahoo! services on
the app to evaluate our interruptibility estimation approach.
Whereas most previous studies used an artificial notification
or ESM [51] as notifications, utilization of real notifications
from real information sources enables us to understand how
users behave in real situations.
A3. Investigating users engagement: Finally, we quan-
titatively measure user engagement levels for the content
presented through each notification in addition to several
immediate response criteria, such as response time and click
rate. Because users load Web content from Yahoo! JAPAN
servers when they click notifications, we decided to track
users browsing behavior from the server side by measuring
engagement-related criteria such as session length and revisit
rate. This evaluation facilitates our understanding of how
users behave and engage with the presented content beyond
immediate click to open behavior.
 V. R EQUIREMENTS FOR S YSTEM D ESIGN AND
D EVELOPMENT
In spite of the goals and approaches above, doing such
research on the real production environment has several dif-
ferences from that on a research-purpose environment. Here
we present the requirements for the system design and devel-
opment that emerged after our discussion in the beginning of
this project. Since we aim to port the original research software
Attelia into the Yahoo! JAPAN commercial production system,
we faced several real-world requirements related to acceptable
behavior and user experience of the product application, as
well as Yahoo!s business-oriented decision and restrictions.
R1. Users additional burden needs to be minimized.: When
placing the interruptibility estimation as an additional logic
to an existing application, depending on the sensor and API
types the logic uses, the users of the app will experience
additional burden, such as explicit confirmation to give ad-
ditional permissions (e.g., accessibility, location information)
to the application. Such burden should be minimized in order
to retain the existing user base of the application.
R2. Power overhead needs to be minimized.: As power is
always a precious resource in mobile devices, and because
mobile users are very conscious about an applications power
usage, our system design needs to be energy-aware and mini-
mize power overhead.
R3. Cross-platform generalizability needs to be consid-
ered.: Although our first step experimental system can be
implemented on a single platform for research purpose, the
fundamental system design needs to be aware of cross-platform
generalizability over both iOS and Android major platforms.
R4. Collection of sensitive data needs to conform to
the corporate policy and process.: Additional collection
of sensitive and/or privacy-related sensor data, such as fine-
grained location information, needs to be proposed, carefully
discussed, and approved in the corporate-wide business process
for assuring end users privacy protection. It means that this
process can take time and our system design may need to
start with a minimum set of data collection for the time-bound
research period.
R5. System design and development needs to conform
to the existing product management.: Yahoo! JAPAN
Android application includes lots of commercial level prod-
uct features of lots of Yahoo! JAPAN services. Its product
planning, design, and development processes are managed
in the business-oriented governance. Thus, the design and
development of our system naturally needs to fit such existing
processes. This also means that opportunity of the application
updates to the market (i.e., GooglePlay) is considerably limited
compared to single-purpose research prototype application
which often contains only bare-bone features and can be
pushed even nightly.
VI. S YSTEM D ESIGN
In accordance with the goals, approaches, and the require-
ments outlined above, this section explains our system design.
Fig. 2. Two Types of Breakpoints - Device Interaction and Physical Activity
A. Detection of Physical Activity Breakpoints
For our concrete interruptibility estimation design to embed
inside the app, we decided to use and extend our previous
work on real-time mobile detection of breakpoints developed
in Attelia [7], [8]. We previously placed breakpoints into
two classes, namely physical activity breakpoints and device
interaction breakpoints as shown in Figure 2. In the figure,
a user is sitting down and doing work on her tablet. After a
while, she decides to take a coffee break. She stands up, walks
to the kitchen, pours a coffee, walks back, sits down on the
couch, and enjoys her coffee while watching a video on her
smartphone. In essence, in our daily lives with smartphones
and wearable devices, there is a significant amount of time
when we simply carry or wear them but do not actively
use (manipulate) them, in contrast to the certain periods we
actually do use (manipulate) them. By detecting two different
types of breakpoints, our previous work detected interruptive
moments in users ubiquitous computing life comprehensively.
Meanwhile, in the present study, we pay special attention
to the utilization of physical activity breakpoints. This is
done for several reasons. Collecting UI interaction data on
smartphone needs users explicit permission for accessibility
API on the smartphone platform (against R1). Furthermore,
accessing such sensitive information of users take long time
in the product management process and it is even not clear if
such data collection gets approved (against R4). On the other
hand, detection of physical activity breakpoints has several
advantages. Physical activity breakpoints can solely cover a
users all-day computing life as long as the user is carrying
the device (even during the users active device use period).
Moreover, activity recognition API used for physical activity
breakpoint detection has been recently implemented in both of
the major mobile platforms [52], [53] (compatible with R3),
and those APIs are considered optimized in terms of efficiency,
accuracy, and power consumption (compatible with R2).
In our design, the system utilizes Google Play Services
Location API (activity recognition API) [53] provided by the
Android platform, reads its detected activity results (e.g., walk-
ing, still, on bike), and detects changes in the users activity
as candidate timings of opportune moments for notification.
B. Notification to Be Tested
Table I gives the definitions of several different notification
classes currently used in Yahoo! JAPAN services. For the first
step of our study, we decided to use the Recommendation
class, for three reasons. First, this class of notifications consists
mainly of updates from services such as sports and showbiz-
related news departments and does not include any emergency
content that needs to be presented to users immediately, so we
can explore various notification scheduling techniques. Second,
receiver users of Recommendation notifications, i.e., those
who enabled reception of this information class on the apps
preference screen, receive exactly the same content, so we can
focus on investigating a users behavior and interruptibility
without being influenced by any personalization aspect of the
notification content. Finally, this plan also fits the current
product planning of the Yahoo! JAPAN application (compatible
with R5).
TABLE I. N OTIFICATION C LASSES
Class Name Real-Time/Batched Content Examples
Breaking news real-time breaking news articles
Natural disaster real-time weather update, earthquakes
Recommendation batched sports & showbiz news, tips
Personal batched transit, email, travel, auction
The recommendation notifications have four basic notifi-
cation timing slots (8AM, noon, 6PM, and 9PM) every day.
The content of the notifications are posted in the batch content
push queue by the news provider department in advance and
then sent to each user at the scheduled timings. At each timing
slot, only one notification content will be pushed to the users.
C. System Architecture
Figure 3 shows the architecture of our production system.
Our implementation consists of a series of additional compo-
nents inside the Yahoo! JAPAN Android application as well
as the components on the server side.
Fig. 3. System Architecture
1) Notification Content Fetching: On the client side, News
Fetcher is a component that triggers whole breakpoint detec-
tion. News Fetcher maintains a connection to the server via
a TCP connection. When a new content (to be presented to
the user) is received from the server, News Fetcher notifies
the Controller component that initiates the core breakpoint
detection logic.
2) Life-cycle of Breakpoint Detection: Figure 4 shows the
life-cycle of the core breakpoint detection logic. The core logic
will be initiated upon the arrival of new notification content.
Once the logic starts, it repeatedly executes core detection
logic (sensing, feature extraction, and prediction) every time
a new sensor data is detected by Mobile Sensing. When a
breakpoint is detected by Predictor (with an installed model),
the system issues a new Android notification and finishes the
core detection logic.
Fig. 4. Life-cycle of Breakpoint Detection Logic
With this design, we can minimize the duration of on-the-
fly mobile sensing and online prediction and also minimize
the power consumption overhead (compatible with R2). We
actually tested the additional power consumption in quality
assurance department and confirmed that the additional over-
head was less than 3% compared to the power consumption
of the original application.
3) Mobile Sensing: The Mobile Sensing component ob-
tains several types of sensor data, including that from the
GooglePlay Service Location API ActivityRecognition
and other device-related data. Table II lists the sensor data
collected on the client. The data mainly consists of a series
of output values from the GooglePlay Service Location APIs
ActivityRecognition and other device-based events (e.g.,
screen on/off events). On the activity recognition results from
ActivityRecognition, we used data only with a confidence
value greater than 51 (the value can be between 0 and 100)
based on our empirical knowledge. The mobile sensing compo-
nent obtains all of these data through individually implemented
event handlers for each sensor. Any changes in sensor data,
such as when the users activity changes or when the device
volume is changed, will be sensed and logged by the system.
4) Feature Extractor: When new sensor data is detected,
the Feature Extractor and Predictor modules execute and
predict if the current moment is a breakpoint of the user.
Table III lists the features extracted from the sensor data. The
total number of features is 387. The system extracts seven
 TABLE II. C OLLECTED S ENSOR DATA
Sensor Name Values
Activity Recognition IN VEHICLE, ON BICYCLE, ON FOOT,
STILL, TILTING, UNKNOWN
Device Volume integer (0 to 7)
Network Mode off, Mobile, Wi-Fi, WiMAX, LTE
Screen Sleep normal, sleeping
Vibration off, on
types of features: timestamp (hour), activity type, volume,
device sleep/awake status, vibration status, silent mode setting,
and network connection type. In addition to these sensor
values, transition of the sensor data is introduced as an eighth
type of feature. For all possible From and To pairs of
sensor status transition, we prepared a dedicated feature value.
To detect breakpoints that occur in the timings of activity
change (e.g., tilting their phone from the still status),
these transition type features are helpful for the system to
characterize changes detected in each sensor.
TABLE III. F EATURES E XTRACTED F ROM T HE S ENSOR DATA
Event Types Explanation
Timestamp (Hour) Hour value extracted from time stamp of client
Collected sensor data Present status of the client device
Trigger Sensor type triggering this detection process
Transitions of From and To transition sets of each sensor
sensor values activity recognition (6x6),
volume(15x15), silent mode(4x4), network(6x6)
We referenced the detailed specifications of the Android
platform to specify the number of possible values in activ-
ity transition, volume transition, silent mode transition, and
network connection transition. We also confirmed that, to
ensure the desired cross-platform nature of the system design,
these feature designs are feasible and reasonable for future
implementation of the system on the iOS platform.
5) Predictor: Using the above features, the actual break-
point prediction is executed in Predictor with an installed
linear regression model. The model parameters are generated
on the server side and then downloaded to the clients.
It is possible that no breakpoint is detected in Predictor for
a long period after the core detection module has been initiated.
For example, when a user has placed the phone on the desk and
no sensor data update (activity recognition, volume change,
etc.) is detected, the core logic may not detect any breakpoints
until a change in a time-related feature value influences a
prediction result. In such cases, the system automatically issues
a notification after a specified timeout T. We set this value as
1 hour throughout this research.
6) Ground Truth Annotation: The users reaction to the
notification, along with the current sensor data, will be logged
on the client. When a user clicks a notification within 10
seconds of the breakpoint (that triggered the notification),
all the sensor data at that breakpoint timing will be logged
with an annotation of breakpoint ground truth. According to
Yahoo! JAPANs knowledge on notifications, empirically we
configured the 10 second timeout value for this study. The
log data is periodically sent to the server for further model
training.
7) Model Training and Update at The Server: The log data
along with the ground truth annotation will be sent to the
server nightly. At the server side, every night a new model
is built from all the data uploaded from the clients in the
past. The resulting models parameter will be available on the
Model Server and will be downloaded to the clients on a daily
basis. This scheme of the periodic model update at the server
and distribution of it to the clients nicely fits our requirement
R5, since the system can update the model frequently without
building and distributing the client application.
Because the size of the client log is expected to be huge,
we prepared a dedicated Hadoop [54] cluster with 32 workers
for model training. After the log data from the clients are
split into 32 buckets at the server side, each worker creates
a logistic linear regression on LIBLINEAR [55]. The 32
generated models will then be merged into a final unified
model considering the bias of the number of training data.
VII. I NITIAL M ODEL T RAINING
On the basis of the system design presented above, we
implemented a prototype version (a new version of the Ya-
Levels
hoo! JAPAN Android app) and conducted an initial study
24 to collect users response logs to notifications and train the
34 initial prediction model (that will be used in the production
3 user study later), as well as to check the basic behavior
326 of our implementation. We installed the app on 39 Android
smartphone devices belonging to members of our lab and
continued the study for 35 days.
During the study, notifications from the real Yahoo! service
were delivered at the conventional random timings. Users
response time to the notification along with the sensor data
in the timing of the user click were logged on the client and
sent to the server. As described in Section VI-C6, the sensor
data will have annotated as a breakpoint ground truth when
the user clicks a notification within 10 seconds and vice versa.
The prediction models, trained from all the log and an-
notation data over 35 days, showed an average accuracy of
94.8% in our 10-fold cross-validation. In this model, a total
of 104 features (all those with a value over 0) are selected for
predicting breakpoints. Table IV shows the top 5 influencing
features. Interestingly, the top feature is the hour value of
the clock being equal to 10. We speculate that this may be due
to the working style of the participants (our lab members). The
second to fifth features relate to the activity change occurrence
of different From and To activities. It is interesting to see
that IN VEHICLE (literally meaning that the user is in a
vehicle) appears twice in this ranking. According to Googles
API definition [53], this class means that the device is in a
vehicle but does not necessarily mean that the owner is driving
the vehicle. ON FOOT includes both walking and running.
TABLE IV. T OP 5 I NFLUENCING F EATURES IN THE I NITIAL M ODEL
Feature Name Value
Hour value=10 4.27
Activity (STILL to ON FOOT) 3.51
Activity (STILL to IN VEHICLE) 3.50
Activity (IN VEHICLE to STILL) 3.30
Activity (TILTING to STILL) 3.23
The API definition considers TILTING to be detected when
the device angle relative to gravity changed significantly. This
often occurs when a device is picked up from a desk or a user
who is sitting stands up. [53]. Although these five features do
not solely determine a breakpoint, we were convinced enough
by these results after a review discussion among the members
(including some subjective impressions).
Selection of the participating users was not visible to
the participants. The application release was checked and
confirmed by the corporate legal and compliance departments.
The study was conducted in conformity to the corporation
regulation and the end user agreement.
B. Experiment Procedure

VIII. E VALUATION The user study was conducted for three weeks (21 days)
in September 2016. To ensure the stability of the production
On the basis of these promising results from our initial application, the new version (including our implementation)
study, we conducted a large-scale in-the-wild user study was released to the production environment with a graduated
in the production environment with about 680,000 users for deployment scheme on the app store. After three days, the new
three weeks to better understand how our breakpoint-based version was made available for all users.
notification scheduling works in a real user environment. Our
evaluation criteria are as follows. The mechanism we utilized for this user study is the one
detailed in Section VI. Our logic was enabled only for the
1. Investigating users immediate response to the experimental group users and not enabled for the control group
breakpoint-scheduled notifications: We want to see how ef- users. Note that, except for the delivery timing difference of
fectively the breakpoint detection works from several different the specific Recommendation type notifications described in
points of view, such as relationships between activity change Section VI-B, the users in both groups experienced the same
and the conclusive detected breakpoints, and actual delay of notification content and delivery timings.
the notification by waiting for a breakpoint.
All users (of the experimental group) used the same model
2. Investigating users response to the breakpoint- for breakpoint prediction. At the beginning of the study, the
scheduled notifications: We want to observe how users react model trained in our initial study was installed in each clients
to the notifications scheduled at detected breakpoint timings. device. Once the study began, a new model was trained every
night at our Hadoop cluster from all clients log data and then
3. Investigating users (long-term) engagement to the pre- was downloaded to each client as a daily update.
sented contents and services: We want to investigate how
the users engagement level to the source Web service of the
C. Results and Analysis
notifications (Yahoo! JAPAN) will be influenced in the longer
term, beyond the users immediate reaction in a short period Through the nightly model update training over 21 days,
of time. our prediction model was gradually adapted to the users real
usage. After 21 days (i.e., 21 iterations of the model update),
A. Participants the latest model showed the average performance accuracy of
91.6% in the same 10-fold cross-validation methodology that
We selected 687,840 users (approximately 10% of the total we used in Section VII.
user base of the Yahoo! Android application) as participants in
this study. We used an existing A/B test infrastructure inside 1) Breakpoint and Activity Change: Table VI shows a
our application where a specific functional component of the breakdown list of detected breakpoints with true annotation
app can be enabled (or disabled) for a specific sub class of (i.e., breakpoints with a notification (presented based-on the
users based on the device ID. Using such system, we randomly basis of the breakpoint detection trigger) that was clicked by
selected 5% of the whole userbase as the experimental group the user within 10 seconds) into activity change pairs.
and the control group respectively. TABLE VI. B REAKDOWN OF T RUE - LABELED D ETECTED
Table V shows the demographics of the users. We split B REAKPOINTS INTO ACTIVITY C HANGES
these users evenly into two groups: (a) the experimental group To
(users to which our interruptibility detection and notification IN_VEHICLE ON_BICYCLE ON_FOOT STILL UNKNOWN TILTING
schedule are enabled), and (b) the control group (users to IN_VEHICLE 0.01 0.22 6.05 0.99 2.71
which our logic is not used) to compare the results and validate ON_BICYCLE 0.03 0.01 0.32 0.01 0.09
ON_FOOT 0.48 0.00 2.68 0.28 1.29
the effectiveness of our system. From
STILL 8.16 0.06 1.75 7.42 43.28
UNKNOWN 0.84 0.00 0.19 3.51 1.50
TABLE V. U SER D EMOGRAPHICS
TILTING 2.63 0.04 0.62 13.55 1.27
Number of Users 687,840
Gender Male 60.7% Very interestingly, the STILL to TILTING activity
Female 39.3% change showed the highest value. Again, as Googles API
0-19 3.6% document [53] mentions, timings such as when a device is
20-29 7.9% picked up from a desk or a user who is sitting stands up
30-39 22.3% are considered to have been opportune moments for the noti-
Age Group 40-49 35.2% fication receiver users. Moreover, activity changes to STILL
(Median: 44.0) 50-59 21.0% showed high numbers, such as TILTING to STILL (13.55%)
(Stdev.: 12.3) 60-69 8.4% and IN VEHICLE to STILL (6.05%). This matches our
70-79 1.4% previous hypothesis [8] that people would have breakpoints
80- 0.3%
when changing from a high energy state to a low energy state.
On the other hand, we see that STILL to IN VEHICLE
resulted in the third highest number, 8.16%. What types of
real-world situations are caught by this change is a possible
topic for future research.
2) Notification Delay Due To Breakpoint: Figure 5 shows
a cumulative distribution function (CDF) on the delay from
when a notification content arrives at the client to when a core
estimation logic detects a breakpoint and the actual notification
is posted. The graph on the right side shows the overview. We
configured the timeout to 1 hour (3,600 seconds), so the value
gets very close to 1 at 3,600 seconds. (We also observed a very
rare situation where notification was further delayed due to an
implementation issue.) When looking at the left side graph
(zooming from 0 to 100 seconds), we see that more than 70%
of notifications were posted within approximately 10 seconds.
The overall average delay is 236.8 seconds (approximately 4
minutes). The timeout occurrence rate is 1.11%.
Fig. 5. CDF on Delay from Content Reception To Notification Issue
3) Users Response to Notifications: One of the most
interesting results we obtained from this study is the users
response time to presented notifications. Figure 6 shows a
CDF on the users response time to the presented notifications
(from when a notification is posted to when a user clicks it)
in both user groups. The response times in the two groups are
significantly different: the time in the experimental group is
1,639.2 seconds (standard deviation: 1,690.2) while that in the
Fig. 6. CDF on Users Response Time To Notifications
control group is 3,258.1 seconds (standard deviation: 1,920.6).
Comparing the two groups, we see that user response time was
reduced by 49.7%, with statistical significance. Combined with
the fact that more than 90% of notifications were delivered
(Figure 5), these result mean that, in most cases, the users
clicks of notifications occurred earlier in our breakpoint-
based notification scheduling. We conclude that, in most
cases, delay in notification delivery due to the breakpoint
detection does not hurt and is even beneficial because the users
can click earlier.
Figure 7 and 8 shows CDFs of two user groups on the
response time for each of the four notifications issued every
day. As explained previously, the Recommendation class
news content becomes available four times a day: 8AM, noon,
6PM, and 9PM. When we plot each of them on the graph (blue:
8AM, red: noon, green: 6PM, grey: 9PM), we confirm that,
for all of them, breakpoint-scheduled notifications resulted in
clearly shorter response times. The average response times are,
in the experimental group, 1,490.0 sec (8AM), 1,527.8 sec
(noon), 1,824.2 sec (6PM), and 1,645.0 sec (9PM), and in the
control group, 3,306.9 sec (8AM), 2,995.7 sec (noon), 3,413.6
sec (6PM), and 3,400.6 sec (9PM).
Fig. 7. CDF on Users Response Time To Notifications (Experimental Group)
Fig. 8. CDF on Users Response Time To Notifications (Control Group)
 4) Users Content Browsing: Table VII shows the number TABLE VIII. U SER E NGAGEMENT L EVEL IN B OTH G ROUPS
of opened notifications on a daily basis. Including the initial
three days when we gradually released the new application to
Timestamp Control Grp Experimental Grp gain (%)
GooglePlay, we measured the numbers for 24 days in total.
TABLE VII. U SER B ROWSING ON N OTIFICATION C ONTENT Before Study 3.757 3.758 0.030
Study Day 7 3.753 3.756 0.077
Day DOW Control Group Experimental Group gain(%)
Study Day 14 3.766 3.770 0.097
(Deploy Day 1) (Tue) 1,837 1,880 +2.34
Study Day 21 3.828 3.832 0.104
(Deploy Day 2) (Wed) 5,459 5,715 +4.69
(Deploy Day 3) (Thu) 9,096 9,280 +2.02 As seen in the table, the differences between the experi-
Study Day 1 (Fri) 29,837 30,386 +1.84 mental group and the control group continue to increase from
Study Day 2 (Sat) 27,747 27,918 +0.62 Before Study (0.030%) to Day 7 (0.077%), Day 14
Study Day 3 (Sun) 36,528 36,927 +1.09 (0.097%), and Day 21 (0.104%). We believe that this gain,
Study Day 4 (Mon) 39,990 41,253 +3.16 as much as 3.46 times in three weeks, is a promising result
Study Day 5 (Tue) 31,237 31,792 +1.78 in terms of long-term user engagement beyond simple Web
Study Day 6 (Wed) 33,869 34,320 +1.33 access session-wise user response to notifications.
Study Day 7 (Thu) 48,419 49,184 +1.58
Study Day 8 (Fri) 53,191 53,874 +1.28 IX. D ISCUSSION
Study Day 9 (Sat) 49,008 49,379 +0.76
Now we discuss the further research opportunities that this
Study Day 10 (Sun) 45,544 45,717 +0.38
research enables. First, we are interested in utilizing more
Study Day 11 (Mon)* 50,630 51,142 +1.01
types of sensors and features for exploring even better system
Study Day 12 (Tue) 39,136 41,266 +5.44
performance. In this study, the types of sensors were limited
Study Day 13 (Wed) 69,301 72,747 +4.97
due to time limitations related to both corporate approval
Study Day 14 (Thu)* 37,845 38,345 +1.32 and advance notification to the users. Once we have cleared
Study Day 15 (Fri) 42,394 42,971 +1.36 these requirements, injecting more types of sensor information
Study Day 16 (Sat) 47,824 48,051 +0.48 into the estimation model will be our future research focus,
Study Day 17 (Sun) 45,256 46,330 +2.37 while always maintaining our protection of users privacy. In
Study Day 18 (Mon) 33,128 33,678 +1.66 Figure 5, more than 70% of notifications were posted within
Study Day 19 (Tue) 39,089 39,479 +1.00 approximately 10 seconds. This can mean that, in some cases, a
Study Day 20 (Wed) (No data due to temporal system down) breakpoint was immediately detected at the first iteration of the
Study Day 21 (Thu) 42,569 43,159 +1.39 logic execution. We consider this comes from relative simple
(*): National public holiday in Japan set of features we used for the current system. For example, if
a trained model contains a relatively bigger weight for device
The interesting finding here is that, for 24 days in a row, volume set to value x, then the first execution immediately
the numbers in the control groups continued to get a greater detects a breakpoint for notification firing. Even so, we are
number of accesses. The average gain over the 21 days of the excited to our result in this study, with that limited amount of
user study period was 1.91% (standard deviation: 1.35). We sensor and feature types.
believe that this amount of gain is significant for Web services
Use of multiple models, including model personalization,
when compared with the industry practice. In addition, we
is another search avenue. In our user study, we observed an
can observe that we have more gain on weekdays than other
interesting difference in users responses between weekdays
days including weekends and holidays. The average gain of
and other days (Section VIII-C4). Further investigation on this
the weekdays is 2.39% (standard deviation: 1.45) whereas that
difference, as well as the difference between days (day-to-day
of weekends and holidays is 1.00% (standard deviation: 0.64).
difference) are obviously our immediate future work.
This difference may stem from differences of users behaviors
with regard to their physical activities (commuting to work vs. We are also interested in an even longer period of user
being at home) and also from net access activities on weekdays study in the production environment. During the present study,
and other days. we did not observe any statistical significance in differences
between user clicks and user engagement. However, we see
5) Longer-Term User Engagement: Finally, we analyzed
clearly reduced p values in some cases (on user engagement,
users longer-term engagement level to the Yahoo! JAPAN
p value decreased from 0.67 (before study) to 0.20 (day 21)).
Web service. As the evaluation criteria, we used a 5-level user
This motivates us to pursue further investigation over an even
engagement level that Yahoo! JAPAN typically uses for its
longer period.
user engagement analysis. This 5-level system was originally
designed to rank each user as 5 if she/he visits the Yahoo! Evaluation on the relationship between the notification
Web site 6-7 days a week, 4 in the case of 4-5 days, 3 in types (and content) and users preference and primary task
the case of 2-3 days, 2 in the case of 1 day, and 1 in the is also another future work. In this paper, for our first step,
case of 0 days. Introducing this metric, we summarized the we focused on the generic user response thus the evaluation.
score of 680,000 users on a group-by-group basis. Table VIII influence of notifications from other applications, including
shows the average level of users in each group. We first popular messenger applications, is yet to be studied. During
summarized the data a day before the study began and then our user study, we hypothesized that they are random external
did the same summary for days 7, 14, and 21 of the study. variables that can randomly occur in both user groups.
 Finally, qualitative analysis from the end users perspective
is also among our future work. We could not do such evaluation
on the application itself during our user study, since our
experiment uses the real product application. A user interview
can be a concrete method of such evaluation.
X. C ONCLUSION
In todays advancing ubiquitous computing age, where
increasingly proactive notifications have been causing inter-
ruption overload, we addressed the problem of finding users
interruptible moments in the mobile environment in order to
provide them with better notification experiences featuring
lower frustration, more responsiveness, and more engagement
to the content. We developed a real-time interruptibility esti-
mation logic based on breakpoint detection inside the Yahoo!
JAPAN Android app, one of the most popular smartphone
applications on the national market. Our large-scale in-the-
wild user study in the production environment with more
than 680,000 real users for 21 days clearly demonstrated the
effectiveness of the system. We found that, in most cases,
notification delivery delay due to breakpoint detection does not
hurt and even improves a users overall click timing (earlier),
with significantly reduced user response time (49.7%). We also
observed a continuous increase in content click numbers and
user engagement level over the entire study period.
R EFERENCES
[1] D. Garlan, D. Siewiorek, A. Smailagic, and P. Steenkiste, Project aura:
toward distraction-free pervasive computing, Pervasive Computing,
IEEE, vol. 1, no. 2, pp. 22 31, april-june 2002.
[2] J. G. Kreifeldt and M. E. McCarthy, Interruption as a test of the user-
computer interface, in JPL Proceeding of the 17 th Annual Conference
on Manual Control, 1981, pp. 655667.
[3] F. R. Zijlstra, R. A. Roe, A. B. Leonora, and I. Krediet, Temporal
factors in mental work: Effects of interrupted activities, Journal of
Occupational and Organizational Psychology, vol. 72, no. 2, pp. 163
185, 1999.
[4] C. Speier, J. S. Valacich, and I. Vessey, The influence of task
interruption on individual decision making: An information overload
perspective, Decision Sciences, vol. 30, no. 2, pp. 337360, 1999.
[5] M. Czerwinski, E. Cutrell, and E. Horvitz, Instant messaging: Effects
of relevance and timing, in People and computers XIV: Proceedings of
HCI, vol. 2. British Computer Society, 2000, pp. 7176.
[6] D. Newtson and G. Engquist, The perceptual organization of ongoing
behavior, Journal of Experimental Social Psychology, vol. 12, no. 5,
pp. 436450, 1976.
[7] T. Okoshi, J. Ramos, H. Nozaki, J. Nakazawa, A. K. Dey, and
H. Tokuda, Attelia: Reducing users cognitive load due to interruptive
notifications on smart phones, in Proceedings of IEEE International
Conference on Pervasive Computing and Communications 2015, ser.
PerCom 15, 2015.
[8] T. Okoshi, J. Ramos, H. Nozaki, J. Nakazawa, A. K. Dey, and
H. Tokuda, Reducing users perceived mental effort due to interruptive
notifications in multi-device mobile environments, in Proceedings
of the 2015 ACM International Joint Conference on Pervasive and
Ubiquitous Computing, ser. UbiComp 15, 2015, pp. 475486.
[9] Yahoo! japan android application, https://play.google.com/store/apps/
details?id=jp.co.yahoo.android.yjtop, Yahoo Japan Corporation, Sep.
2016.
[10] H. A. Simon, Designing organizations for an information-rich world,
Computers, communication, and the public interest, vol. 37, pp. 4041,
1971.
[11] A. Toffler, Future shock. Bantam, 1990.
[12] J. T. Milord and R. P. Perry, A methodological study of overloadx,
The Journal of General Psychology, vol. 97, no. 1, pp. 131137, 1977.
[13] Attention management in ubiquitous computing environments (amuce
07), http://www.ubicomp.org/ubicomp2007/index-14.htm.
[14] S. Gould, D. Brumby, A. Cox, V. Gonzalez, D. Salvucci, and N. Taat-
gen, Multitasking and interruptions: a sig on bridging the gap between
research on the micro and macro worlds, in CHI12 Extended Abstracts
on Human Factors in Computing Systems, 2012, pp. 11891192.
[15] B. Poppinga, M. Pielot, N. Henze, N. Oliver, K. Church, and A. S.
Shirazi, Smarttention, please! intelligent attention management on
mobile devices, in Proceedings of the 17th International Conference
on Human-Computer Interaction with Mobile Devices and Services
Adjunct, ser. MobileHCI 15, 2015, pp. 10661069.
[16] D. Weber, A. S. Shirazi, S. Gehring, N. Henze, B. Poppinga, M. Pielot,
and T. Okoshi, Smarttention, please!: 2nd workshop on intelligent
attention management on mobile devices, in Proceedings of the 18th
International Conference on Human-Computer Interaction with Mobile
Devices and Services Adjunct, ser. MobileHCI 16, 2016, pp. 914917.
[17] A. Voit, B. Poppinga, D. Weber, M. Bohmer, N. Henze, S. Gehring,
T. Okoshi, and V. Pejovic, Ubittention: Smart & ambient notification
and attention management, in Adjunct Proceedings of the 2016 ACM
International Joint Conference on Pervasive and Ubiquitous Computing,
ser. UbiComp 16, 2016, pp. 15201523.
[18] D. Kahneman, Attention and effort. Prentice-Hall, Inc., 1973.
[19] R. M. Shiffrin and W. Schneider, Controlled and automatic human
information processing: Ii. perceptual learning, automatic attending and
a general theory. Psychological review, vol. 84, no. 2, p. 127, 1977.
[20] R. J. Sternberg and K. Sternberg, Cognitive Psychology, 6th ed. Cen-
gage Learning, 2012.
[21] P. D. Adamczyk and B. P. Bailey, If not now, when?: the effects of
interruption at different moments within task execution, in Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems,
ser. CHI 04, 2004, pp. 271278.
[22] B. P. Bailey and J. A. Konstan, On the need for attention-aware
systems: Measuring effects of interruption on task performance, error
rate, and affective state, Computers in Human Behavior, vol. 22, no. 4,
pp. 685 708, 2006.
[23] F. Paas, J. E. Tuovinen, H. Tabbers, and P. W. Van Gerven, Cognitive
load measurement as a means to advance cognitive load theory,
Educational psychologist, vol. 38, no. 1, pp. 6371, 2003.
[24] S. G. Hart and L. E. Staveland, Development of NASA-TLX (Task
Load Index): Results of empirical and theoretical research, in Human
Mental Workload, ser. Advances in Psychology, P. A. Hancock and
N. Meshkati, Eds. North-Holland, 1988, vol. 52, pp. 139 183.
[25] J. Beatty and B. Lucero-Wagoner, The pupillary system, Handbook
of psychophysiology, vol. 2, pp. 142162, 2000.
[26] C. S. Ikehara and M. E. Crosby, Assessing cognitive load with physi-
ological sensors, in System Sciences, 2005. HICSS05. Proceedings of
the 38th Annual Hawaii International Conference on, 2005, pp. 295a
295a.
[27] S. T. Iqbal, P. D. Adamczyk, X. S. Zheng, and B. P. Bailey, Towards an
index of opportunity: understanding changes in mental workload during
task execution, in Proceedings of the SIGCHI conference on Human
factors in computing systems, 2005, pp. 311320.
[28] G. F. Wilson, An analysis of mental workload in pilots during flight us-
ing multiple psychophysiological measures, The International Journal
of Aviation Psychology, vol. 12, no. 1, pp. 318, 2002.
[29] K. Ryu and R. Myung, Evaluation of mental workload with a combined
measure based on physiological indices during a dual task of tracking
and mental arithmetic, International Journal of Industrial Ergonomics,
vol. 35, no. 11, pp. 9911009, 2005.
[30] Y. Shi, N. Ruiz, R. Taib, E. Choi, and F. Chen, Galvanic skin response
(gsr) as an index of cognitive load, in CHI07 extended abstracts on
Human factors in computing systems, 2007, pp. 26512656.
[31] T. K. Fredericks, S. D. Choi, J. Hart, S. E. Butt, and A. Mital, An
investigation of myocardial aerobic capacity as a measure of both
physical and cognitive workloads, International Journal of Industrial
Ergonomics, vol. 35, no. 12, pp. 10971107, 2005.
[32] L. Mulder, Measurement and analysis methods of heart rate and
respiration for use in applied environments, Biological psychology,
vol. 34, no. 2, pp. 205236, 1992.
[33] E. Haapalainen, S. Kim, J. F. Forlizzi, and A. K. Dey, Psycho- [50] StatCounter, Inc., StatCounter, http://statcounter.com/, Jul. 2016.
physiological measures for assessing cognitive load, in Proceedings [51] M. Csikszentmihalyi and R. Larson, Validity and reliability of the
of the 12th ACM international conference on Ubiquitous computing, experience-sampling method. The Journal of nervous and mental
ser. Ubicomp 10, 2010, pp. 301310. disease, vol. 175, no. 9, pp. 526536, 1987.
[34] S. T. Iqbal and B. P. Bailey, Investigating the effectiveness of mental [52] Apple Inc., CMMotionActivityManager, https://developer.
workload as a predictor of opportune moments for interruption, in CHI apple.com/library/ios/documentation/CoreMotion/Reference/
05 Extended Abstracts on Human Factors in Computing Systems, ser. CMMotionActivityManager class/index.html, 2014.
CHI EA 05, 2005, pp. 14891492.
[53] Google Inc., Making your app location-aware - Android Developers,
[35] S. T. Iqbal and B. Bailey, Leveraging characteristics of task structure https://developer.android.com/intl/ja/training/location/index.html.
to predict the cost of interruption, in Proceedings of the SIGCHI [54] The Apache Software Foundation, Apache Hadoop Project, http://
Conference on Human Factors in Computing Systems, ser. CHI 06, hadoop.apache.org/.
2006, pp. 741750.
[55] Machine Learning Group, National Taiwan University, LIBLINEAR
[36] E. Horvitz and J. Apacible, Learning and reasoning about interruption, A Library for Large Linear Classification, https://www.csie.ntu.edu.
in Proceedings of the 5th International Conference on Multimodal tw/cjlin/liblinear/.
Interfaces, ser. ICMI 03, 2003, pp. 2027.
[37] S. Hudson, J. Fogarty, C. Atkeson, D. Avrahami, J. Forlizzi, S. Kiesler,
J. Lee, and J. Yang, Predicting human interruptibility with sensors:
A wizard of oz feasibility study, in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, ser. CHI 03,
2003, pp. 257264.
[38] J. B. Begole, N. E. Matsakis, and J. C. Tang, Lilsys: Sensing unavail-
ability, in Proceedings of the 2004 ACM Conference on Computer
Supported Cooperative Work, ser. CSCW 04, 2004, pp. 511514.
[39] E. Horvitz, P. Koch, and J. Apacible, Busybody: Creating and fielding
personalized models of the cost of interruption, in Proceedings of the
2004 ACM Conference on Computer Supported Cooperative Work, ser.
CSCW 04, 2004, pp. 507510.
[40] S. T. Iqbal and B. P. Bailey, Oasis: A framework for linking notifica-
tion delivery to the perceptual structure of goal-directed tasks, ACM
Transactions on Computer-Human Interaction, vol. 17, no. 4, pp. 15:1
15:28, Dec. 2010.
[41] J. Ho and S. S. Intille, Using context-aware computing to reduce the
perceived burden of interruptions from mobile devices, in Proceedings
of the SIGCHI Conference on Human Factors in Computing Systems,
ser. CHI 05, 2005, pp. 909918.
[42] J. E. Fischer, C. Greenhalgh, and S. Benford, Investigating episodes
of mobile phone activity as indicators of opportune moments to deliver
notifications, in Proceedings of the 13th International Conference on
Human Computer Interaction with Mobile Devices and Services, ser.
MobileHCI 11, 2011, pp. 181190.
[43] G. H. H. ter Hofte, Xensible interruptions from your mobile phone, in
Proceedings of the 9th International Conference on Human Computer
Interaction with Mobile Devices and Services, ser. MobileHCI 07,
2007, pp. 178181.
[44] V. Pejovic and M. Musolesi, InterruptMe : Designing Intelligent
Prompting Mechanisms for Pervasive Applications, in Proceedings
of the 2014 ACM International Joint Conference on Pervasive and
Ubiquitous Computing, ser. UbiComp 14, 2014, pp. 395906.
[45] M. Pielot, T. Dingler, J. S. Pedro, and N. Oliver, When attention is not
scarce - detecting boredom from mobile phone usage, in Proceedings
of the 2015 ACM International Joint Conference on Pervasive and
Ubiquitous Computing, ser. UbiComp 15, 2015, pp. 825836.
[46] A. Mathur, N. D. Lane, and F. Kawsar, Engagement-aware computing:
Modelling user engagement from mobile contexts, in Proceedings
of the 2016 ACM International Joint Conference on Pervasive and
Ubiquitous Computing, ser. UbiComp 16, 2016, pp. 622633.
[47] U. Acer, A. Mashhadi, C. Forlivesi, and F. Kawsar, Energy efficient
scheduling for mobile push notifications, in Proceedings of the 12th
EAI International Conference on Mobile and Ubiquitous Systems:
Computing, Networking and Services on 12th EAI International Confer-
ence on Mobile and Ubiquitous Systems: Computing, Networking and
Services, ser. MOBIQUITOUS'15, 2015, pp. 100109.
[48] A. Sahami Shirazi, N. Henze, T. Dingler, M. Pielot, D. Weber, and
A. Schmidt, Large-scale assessment of mobile notifications, in Pro-
ceedings of the 32nd annual ACM conference on Human factors in
computing systems - CHI 14, 2014, pp. 30553064.
[49] M. Pielot, K. Church, and R. de Oliveira, An In-Situ Study of
Mobile Phone Notifications, in Proceedings of the 16th International
Conference on Human-Computer Interaction with Mobile Devices and
Services - MobileHCI 14, 2014, pp. 233242.