Sie sind auf Seite 1von 36

AN EVALUATION OF PATIENTS’

EXPERIENCED USABILITY OF A
DIABETES MHEALTH SYSTEM USING A
MULTI-METHOD APPROACH

Erika Trisliana – 13414030


Aditya Binowo - 13414057
OUTLINE
2

▸ Introduction
▸ Materials and methods
▸ Results
▸ Discussions
▸ Conclusions
3

1.
Introduction
4
Mobile Health Technology
▸ Support systems for diabetes self-management using
Information and Communication Technology (ICT)

▸ Successful improving glucose management and aided in


regulating lifestyle changes

▸ Self-coaching by allowing users to interact with algorithm-


based systems that provide individually tailored messages
and feedback messages
5
Mobile Health Technology
▸ Need for improved usability to allow patients to get comfy in
their interaction with mHealth applications.

▸ Most studies lack methodological rigor and use a sigle


method for data collection

▸ Qualitative data analyses lack a structured approach

▸ Methodological improvements could result in more


comprehensive analysis and reproduciblity, also improved
user interaction and performances
6
Multi-Method Approach

▸ Observation, field notes


and interview first
identified by Patton

▸ Data collection used user-


based testing, Think Aloud
protocol, open-ended
interviews and a short
answer questionnaire
7
Structured Approaches for Data Analysis
▸ Framework Analysis (FA)

▸ Purpose: to assist with the analysis of descriptive, textual


source data to produce reliable and valid qualitative reports

▸ Consists of five stages (1) familiarizing oneself with the


data, (2) identifying a thematic framework to be used, (3)
indexing and applying the framework to data, (4) charting
the data, (5) mapping and interpelation
8
Structured Approaches for Data Analysis
▸ Usability Problem Taxonomy (UPT)

▸ Purpose: Providing a clear structure in usability problem


definition using classification scheme and framework for
characterizing usability programs according to their
dimensions

▸ Two perspectives: The task-artifact approach and The


Object-Action Interface Model
9
Structured Approaches for Data Analysis
10
Study Aim

▸ Using a multi-method approach for data collection


during a mHealth usability evaluation

▸ Applying structured approaches to data analysis


by using the FA method and UPT
11

2.
Materials and
Methods
12
The Evaluated System
• Interactive, SMS-based movile
intervention
• Combined mobile phone
solution and web service
• Patients interact by either
sending in or being prompted
to send in their Type 2
Diabetes, self-management
values
• Using the web service, they
can enter and/or review their
result
13
Study Sample and Setting
▸ 2317 patients met the criteria for the study

▸ 10 patients picked randomly by computer randomization and


invited to take part in usability evaluation

▸ Inclustion criteria:
(1) Patients diagnosed with Type 2 diabetes
(2) No cognitive impairment
(3) Familiarity and some knowledge and use of computer, the
Internet, and cell phone
(4) The ability to speak and understand English
14
Data Collection Methods
▸ User-related method:
(1) Think Aloud: Usability assessment method commonly
employed to determine users’ thoughts and opinions while they
peform a list of specified tasks with a system
(2) Tasks for this study were based on common patient user
interaction with the system that validated by a panel consisting
of a physician and a nurse whose specialities were diabetes
•Uploading glucose values •Interpreting a blood pressure measurement
•Interpreting a glucose measurement •Setting personal tracking goals
•Correcting recorded glucose measurement •Setting medication reminders
value •Setting a physician appointment reminder
•Exporting glucose measurement to PDF
15
Data Collection Methods
▸ User-related method:
(3) Respondents completed open-ended interview and a post-
experience questionnaire

The three questions in the interview asked patients to comment on


sections that were inadequately designed and any further
comments they might have about system usability

In post-experience questionnaire, first section consisted of short


answer question about IT, mobile phone, internet used. The second
section included open-ended questions about the system
16
Study Procedure
▸ Patients were asked to informed consent
▸ Evaluation procedure and asked if they hand any question
▸ Standardized training to simulate an actual patient educational
process in a health clinic
▸ The Think Aloud and Usability evaluation (Interaction were
digitally audio and video recorded using Morae)
▸ Patients were interviewed about their experiences an perceptions
about the system
▸ Completed post-test questionnaire on their perceptions about the
usability of the system
17
Data Analysis
18

3.
Results
19
Sample’s Characteristics
▸ Gender composition: Women (6) and Men (4)

▸ Aged between 40 – 59

▸ Used technology daily


Definition of used technology in here was used computer,
internet, and mobile phones to make and receive calls

▸ Majority agreed that using technology in health care is a


positive development and positive to their diabetes self-
management
20
Usability Problems
▸ A total 117 initial usability problems were detected using 3
methods (usability test, post interview, and questionnaire)
and consolidated into a unique list of 19

▸ The average severity for the whole site was 2.47 (serious)

▸ The views with highest number of problems were those with


several steps in a task
21
Example of Identified Usability Problems
22
Critical and Severe Usability Problems

▸ The most critical usability problems were located in Glucose


Reading View and Glucose Diary View

▸ This problems commonly caused by


▹ confusing sequence to finish the task
▹ multiple function of the button
▹ no navigation system
▹ display (location or label)
23
Results for Methodological Approaches

▸ The usability test


detected most of the
usability problems
both on its own and in
combination with
other methods
24
Problem Classification

▸ This figure indicates


distribution of problems for
each UPT classification level
and the depth of classification

▸ Critical problems were


classified into the artifact
components. Task
classifications involved
functionality concern.
25
Problem Classification (2)

▸ This figure
indicates whole
system problems
related to
presentation,
visualization,
system design and
structure (not
match with user
perspective).
26

4.
Discussion

The usage of multiple methods
resulted in more comprehensive
understanding of patients’
interactions with the system and
provide triangulation on severe
usability issues.
28
Usability Problem Types and Classifications

▸ The majority of problems were located in the Glucose


Readings View and two other views, the Glucose Diary View
and Blood Pressure View, had the highest severity ratings.

▸ Usability problems included issues with patients’ data (delete


and enter) and information about how to save a file and many
more as have already said in section 3.

▸ The results pointed to difficulties patients had completing


common tasks (frequent basis for diabetes self management)
so these concerns need to be resolved quickly.
29
Data Collection Methods

▸ Using a multi-method approach is beneficial to identifying


usability problems especially initial usability problems
because each method provide different point of view

▸ Usability testing (think aloud) identifies more problems,


recurring usability problems and defines the underlying
causes.

▸ In-depth interview did not contribute any unique consolidated


because may be the patients echoed only the issues they
voice during testing so they did not have any other insights
30
Data Collection Methods (2)

▸ Used open ended questions to provide an opportunity for user


to express their thoughts freely and perspective about
problems that may not surface during testing.

▸ The post questionnaire can verified significant usability


problems. But, in this study this method only generated few
issues due to study sequencing. (Questions seem similar to
users, fatigue at the end of 2 hours sessions, only mention
severe problems to them)
31
Structured Data Analyses

▸ The method used in this paper provided a beneficial way of


structuring data and coding large amounts of data.

▸ The weaknesses of UPT : vague problem descriptions and


problem could not be fully classified (due to insufficient
descriptions)

▸ The strengths of UPT : detailed, could group problems of a


similar nature easily
32
Contributions to The Literature

▹ Overall, this study adds to the body of literature because it


provides an example of a multi-method approach for
collecting data on usability problems , for standardizing data
analyses, and contributing to the reproducibility of results in
qualitative usability studies.
33
Lessons Learned and Recommendation

▸ FA analysis can generated a large amount of data even with


small number of samples but the coding need laborious
process

▸ This method is recommended for studies employing with 2


researchers

▸ UPT is easy to use for individuals familiars with HCI, usability,


and familiar with the system.
34
Limitations and Future Research

▸ Small sample size


▸ Convenience sample of patients with diabetes may not
generalize findings to the population

▸ Recommendations for future research


▹ Use purposeful sampling to select users with different characteristics to
maximize variability
▹ Involved users early and in development process
▹ Use larger samples
35
Conclusions

▸ This study provides an example of a multi-method design for


both data collection and data analyses. Multiple data
collection methods resulted in a more comprehensive set of
usability problems and helped triangulate data. The
structured data analyses allowed reproducible steps and data
validation (triangulation), a method of determining the most
severe problems for users. We recommend the use of
multiple data collection methods to uncover a variety of
problem types and severity levels. We also recommend the
use of the FA and UPT methods during data analyses.
36

THANKS!
Any questions?