Beruflich Dokumente
Kultur Dokumente
Sound to Sight
TEAM RAINBOW
Ankita Narvekar • Angela Zhang • Clara Gilligan
Fanrui Sun • Simin Gu • Yibing Qian
DM-UY 2213 User Experience Design | Spring 2018
Professor Dana Karwas | Graduate Assistant Will Hsu
TABLE OF CONTENTS
INTRODUCTION ____________________________2
Team Bios __________________________________________________________________4
Design Challenge ___________________________________________________________7
Brainstorming _______________________________________________________________ 8
10
Disability Analysis & Design Process __________________________________________
Team Rainbow
5
Chromati
INTRODUCTION
7
DESIGN CHALLENGE
Our diverse team of six came together
from backgrounds ranging from China,
India, and New Jersey. Comprised of both
undergraduate and graduate students equipped
with skills ranging from coding to visual design
to 3D devlopment, Team Rainbow was formed to
solve a challenge that was curated by Microsoft
for Dana Karwas' User Experience Design course.
"Design a product, service, or solution to
solve for exclusion in a deskless workplace.
Think about software, hardware, varying
interaction modalities, AI & Cognitive
augmentation, and/or, a physical workspace. Your
design must meet a clear need and be extensible
to wider applications. Your project may be near-
term practical or blue sky, but the idea must
be innovative, technically feasible, and have a
realistic chance of adoption if instantiated."
“Design a product, service, or solution to
solve for exclusion in the workplace.”
Chromati
8
BRAINSTORMING
Designing for inclusivity opens up our experiences and
reflects how people adapt to the world around them. What
does this look like in the evolving deskless workplace?
- Spring 2018 Microsoft UX Design Challenge
For our first round of brainstorming, we used Later, we were advised to scrap the visually-
post it notes and categories to sort disabilities and impaired Uber driver idea. We had to do a second
deskless jobs. We also created a chart of different round of brainstorming to figure out a different
people with disabilities and their jobs, and wrote spectrum and how we wanted to go about designing
for the user. We came to a conclusion that we wanted
down what would be necessary for them to succeed
to stick to the field of people with disabilities who
at their job. This led to our first idea which was the
want to work in transportation. For a brief while, we
vision impaired uber driver. When brainstorming for
discussed how we could assist hearing-impaired or
solutions regarding the vision impaired driver, we
visually-impaired pilots.
created sketches of what it would look like inside
Eventually, we chose to look into a surgeon who
a car with our device. We also did research about
communicates using American Sign Language (ASL)
different visual impairments and how they affect
and we developed our idea from there. Through our
driving.
research, we learned that ASL is most widely used by
people who have difficulty hearing, so we changed our
idea to include those who have a hard time hearing.
Team Rainbow
9
1. Microsoft. “Empower, engage and connect your deskless workers with their team and company” YouTube, 27 Sept. 2016, youtu.be/gK645_7TA6c.
Chromati
10
Team Rainbow
11
DESIGN PROCESS
Once we came up with a profession and disability we began brainstorming.
We discussed the different options for bridging the gap between
communication. Once we came up with a solution we wrote out scenarios
and drew sketches of the device.
Sketches of possible features that would utilize vision, A scenario incorporating the functions of the AI, which
including eye tracking and laser pointing would interpret actions via input and output
Team Rainbow
13
Chromati
14
Team Rainbow
15
SOLUTIONS
The Chromati is in the form of glasses that facilitate communication between a
hearing-impaired person and those around them through speech and text. It can be
Chromati
Left) Our team visited the Center for Hearing
and Communication in NYC to gain insight
and learn about the disability we were
working with.
17
USER RESEARCH
You really need to know your audience —
emapthy is a key factor in inclusive design.
- Nick Zimmerman, Microsoft
Chromati
18
PROBLEM STATEMENT
Because the realm of hearing impairment is so vast, we divided our
problem statement into three parts. How can we create use visual base
to create assistance for those who...
Team Rainbow
19
A) Frequency
Sounds are described in terms of
their frequency, known as “pitch,” and
intensity, known as “loudness.” High
frequency sounds needs to be louder to
be heard.
4. Types of Hearing Loss. Centers for Disease Control and Prevention. Retrieved from https://www.cdc.gov/ncbddd/hearingloss/sound.html. Chromati
20
B) Distance
Team Rainbow
21
5. How Our Ears Sense the Direction of Sounds. E-Home Recording Studio. Retrieved from https://ehomerecordingstudio.com/binaural-hearing/.
6. Inverse Square Law. Sweetwater. Retrieved from https://www.sweetwater.com/insync/inverse-square-law/. Chromati
22
USER CONTEXT
For a surgeon, it is important to be able to communicate. A surgeon needs to
communicate with their staff during the surgery, and they need to communicate with
their patients to make sure they get the correct care. However, for a hearing-impaired
surgeon, all of the important communication they have to do is hindered in some way.
Communication, both verbal and non-verbal, are Information must be relayed properly to a patient
crucial in an environment involving a team. from a medical professional and vice versa.
Team Rainbow
23
STORYBOARDS
Storyboards are illustrations that represent The second storyboard portrays a
shots that ultimately represent a story.7 In customer service executive, who mainly handles
storyboarding different scenarios based on our user phone calls on a regular basis. Whether she is
personas (p. 29-30), we aim to bring alive the purpose hard of hearing or not, she can use the Chromati
and functionality of the Chromati in respective to interpret the customer's words and display
contexts through potraying scenarios the users would them on the interface for her to easily read. It
face on a daily basis. also shows that the device can sync to a cellular
The first storyboard is based on the the line so that the user can conveniently use the
deskless worker we originally designed the Chromati Chromati interface which is as fluid and cohesive
for, the hearing-impaired surgeon. Communication as that of a smartphone or smartwatch.
in the procedure room is vital for everyone involved.
For a surgeon who is hard of hearing and may also
have her hands full at times is capable of using the
Chromati's eye tracking feature to navigate through
the device's interface and select key phrases that are
synced into her device instead of typing out responses,
which would require hand movements she cannot
perform.
Team Rainbow
25
Chromati
26
PERSONA SPECTRUM
Team Rainbow
27
USER PERSONAS
USER FLOW KEY
To help us further narrow our broad range of
ideas that we came up with during the brainstorming
sessions, we came up with three user personas. ENVIRONMENT
The first persona is of a medical student who
has been hard of hearing since birth and is also mute.
We chose this persona specially because we wanted to
include those of the hard of hearing community who
use ASL to communicate. Another persona is designed USER ACTION
for a customer service executive, who is constantly
taking calls, and thus has developed a bit of hearing
loss. The last persona is designed for a professor, who
requires assistance dealing with multiple voices in a
crowd
CHROMATI ACTION
We also created the user flows for the respective
personas, to further demonstrate how the Chromati
works for these varying users and contexts.
Team Rainbow
29
DAISY
Daisy is 23 years old. She has been hard
of hearing since she was born. She
aspires to be a doctor—more specifically,
a surgeon. She studies medicine at
New York University's Medical school,
and she is currently in her junior year.
She faces problems when she is in a
crowd and wants to focus on what one
person is saying, while others also talk
simultaneously. Lip reading is not that
entirely accurate either, so it is difficult
for her to soley rely on it. She also faces
difficulties when she has to interact with
“During practical/lab other people as they don’t know sign
sessions, I cannot use lip language.
reading to understand
who is saying what, as Goals:
everyone wears masks to • To be able to perform successful
cover their mouths.” surgery.
• To be able to communicate with the
Age: 23
patients.
Work: Student
• To have a stress-free communication
Family: Parents, older
with nurses.
sibling
Location: New York
Frustrations:
Character: Hard working,
• Soliciting attention from nurses to help
intelligent
her with instruments.
• Conveying decisions during emergency.
• Being notified if there is some anomaly
during surgical procedure.
Chromati
30
TINA
Tina is a customer service executive..
She is 27 years old. She works at a call
centre and has to attend calls all week
long. She spends time with family on
weekends. She has been working as a
customer service executive for almost
five years and has therefore acquired
some hearing loss. On some days, she
cannot even hear when her son calls
out to her from the living room!
Goals:
“[Taking calls] would • To perform efficient requests
sometimes become processing at the customer service
pretty taxing and stressful centre.
on the ears, using the • To not make mistakes, for her job is
headphones all the time.” very detail-oriented.
Age: 27 Frustrations:
Work: Customer Service • Unable to concentrate on the voice
Executive on headphones.
Family: Wife, one child • Has acquired a little hearing loss
Location: New York due to years of constant exposure to
Character: Fit, caring. constant calls.
• Has difficulty communicating with
personal relationships, such as with
family.
Team Rainbow
31
SUSAN
Susan is a lecturer of Machine Learning
at CUNY. She is 67 years old. She has
been in the profession since last 25
years. She is one of the best professors
in the field now, and often delivers
lectures at a lot of other universities
and conferences.
Goals:
• To give lectures and help students
understand concepts.
“When someone raises • To reach out to students in class, help
a doubt or asks me a them clear their doubts.
question, its really difficult
to locate him/her quickly Frustrations:
in a the lecture hall.” • If the lecture hall is too huge, cannot
hear the questions that students ask.
Age: 67 • Cannot focus on one student’s doubt
Work: Lecturer if a lot of students are talking.
Family: Husband, one
child
Location: New York
Character: Kind,
approachable, intelligent
Chromati
We compared our product with other
existing technologies. For carrying out this
research, we visited the Center for Communication
and Hearing. We attended the sessions that were
giving out information on assistive technologies
available for people who are hard of hearing. But
while talking to representatives there, we realized
that there were not many solutions available
to help out the section of the hard of hearing
community which uses ASL to communicate with
others.
Taking a step forward to include them in our
design, we decided on some unique features which
are mentioned in the column names. We have
given binary ratings to denote what is included
in the products mentioned at the beginning of
every row. This gives accurate information about
the experience our product proposes to offer. We
give a clear and straightforward experience while
having a device that is innovative and easy to use.
We have given special accommodation to people
who use ASL by providing text to voice and AI
responses.
COMPARATIVE ANALYSIS
Chromati
35
PROTOTYPING
STAGE_1
-THE PAPER PROTOTYPE-
We took the notes and ideas that were drafted during brainstorming and our initial
design process and translated them into coherent ideas that we could implement into
the Chromati. Our iteration process was divided into two parts for each step: the physical
mockup of the Chromati and the interface mockup of the Chromati's AI.
Chromati
36
INTERFACE DESIGN
-PAPER PROTOTYPE-
Initial steps to create the interface
design involved some brainstorming, as
we had to narrow down the functions
that our device would be capable of
performing. It was easy to come up
with different features to include in its
performance, but it was a challenge to
navigate how we were going to narrow
down the functions in order to make the
Chromati a well-designed, simplified
interface for any person using it.
Team Rainbow
37
CHROMATI MODEL
-PAPER PROTOTYPE-
Clara modeling a tangible paper prototype of Sketch made on Illustrator by Yibing of the
the Chromati. frames.
Chromati
39
PROTOTYPING
STAGE_2
-THE LOW FIDELITY PROTOTYPE-
Because our paper prototype was not exactly testable for users to engage in
and instead more focused on the general concept of the Chromati, when we began
creating the low-fidelity mockup of the interface design, we worked on making the
prototype with various functions that we could present to users to navigate.
For our prototypes we decided to go two ways. First, we created a marvel app
that would emulate the interface, then we created videos that would demonstrate
what the glasses are like in real life. . Therefore; it is important that it is easy to access
every function with minimal movements, and with minimal discomfort.
Chromati
40
For the visuals we used sketches in the beginning, and took inspiration from 3D Mockups of the frames of
other glasses, like the google glass. We wanted to use these glasses because the Chromati. Modeled main
it is important that they do not take up too much of the face, and are not features, such as lenses, legs,
too uncomfortable. The google glass had that in mind as well, so it is was and piece on top of frames for
an important inspiration. We had many different types of sketches, that use the camera lens.
different types of designs.
Team Rainbow
41
CHROMATI MODEL
The first mockups of the physical
-LO-FI PROTOTYPE-
prototype of the Chromati.
Design is very clunky and meant
to just show concept of the
shape of glasses. Examples of
AI phrases the interface could
provide.
Chromati
Bottom) The interface
appears in a type of mixed
reality AI, so the user can still
experience the authenticity
of their surroundings
43
INTERFACE DESIGN
-LO-FI PROTOTYPE-
Our second stage of lo-fi prototyping for the interface focused more on showing
the actual interactions. Mockups were initially made in Axure first, then images and
detailed text were added in and compiled into a prototyped Marvel app that shows an
examples of the interface. We had users test the Marvel app to see how convenient it
was for them to locate the different functions.
Chromati
44
Team Rainbow
45
Chromati
46
Team Rainbow
47
Chromati
48
Team Rainbow
49
Chromati
50
Team Rainbow
53
PROTOTYPING
STAGE_3
-THE HIGH FIDELITY PROTOTYPE-
Based on user feedback and testing, we took our lo-fi prototype and assessed
which functions of the Chromati we wanted to keep or scrap. For instance, initially
we had a function for interpreting music for conductors or those who play in a band
or orchestra, but we realized that the frequencies in music are on another spectrum
from the human voice, so we decided to focus on soley human interaction for the
Chromati's different modes. We also took our user feedback to fix the simplicity of the
design in terms of how easy it would be to locate different functions.
Chromati
55
CHROMATI MODEL
-HI-FI PROTOTYPE-
Chromati
On the Chromati interface, the user can select from options On the Chromati interface, the user can select from options
Speech to Text, Friends, My Account, and Settings. Speech to Text, Friends, My Account, and Settings.
57
INTERFACE DESIGN
-HI-FI PROTOTYPE-
Based on our user tests and low-fi prototype, we made our hi-fi prototype.
Users can use eye tracking or hands movement to select and swipe elements on
the interface. It has 4 modes for people to choose based on different situation: one
on one mode, small group mode, crowd mode and phone call mode.
On top of that, users can either type the dialogue they wish to say or they
can have their voice carried out through the AI voice or choose relative phrases
based on the AI system database and translate those to voice. This function helps
users who use sign language to articulate verbal words so that even people who
are not fluent in sign language can understand them with ease.
Chromati
60
PROJECT OUTCOMES
In designing and devloping the Chromati, our team learned a lot through the many iterations
and changes we made based on user research and user feedback. It is interesting to look back on how
we initially planned to design for a colorblind driver. We took a step back and re-evaluated to realize
that we were designing for a deskless worker who is not necessarily excluded from the workplace
today. When we took a step back to look at the bigger picture and held a fresh brainstorming session,
we took a more empathetic approach and instead delved into the medical industry and upon research,
we found that people do face judgment and exclusion from a field that is so dependent on precision
and communication. And from there, we found out that what we wanted the Chromati to feature is
so vital to everyday life: fluid human interaction. Initially designing for a hearing-impaired surgeon, we
discovered that there exist many situations, whether they be permanent, situational, or temporary, that
exclude people from what is considered the 'norm'.
With such different minds from different backgrounds, it is inevident that we all had different
interpretations and solutions we wished to create for such a multi-functioning device. What bridged
us together was a cohesive goal to create an experience for those who need assistance with using
the senses they possess to guide the ones they were not confident with; in Team Rainbow's case, we
wanted to convert the limitless colors of sound into a rainbow through the user's eyes and translate
those colors from sound into sight.
Team Rainbow
61
CHROMATI IN CONTEXT
-One on One-
Chromati
62
CHROMATI IN CONTEXT
-Small Group-
Team Rainbow
63
CHROMATI IN CONTEXT
-Phone Call-
Chromati
64
FINAL THOUGHTS
Team Rainbow
65
FUTURE STEPS
DESIGN RESEARCH
The high-fidelity prototype and the visual As mentioned earlier in the book, we were not able to
representation capture the functions that the Chromati get in touch with as many references or contacts to
delivers, but with further prototyping and iterations of fund our research as we would have hoped. With more
the Chromati's interface, we can truly expand it and time and efforts, we would gain more insight from
showcase its full-functioning capabilities. centers and individuals who work with those who are
hard of hearing. We would also branch out to try and
DEVELOPMENT learn about those who are speech-impaired, as our
Because our group members did not have background device does supplement those who have trouble with
in 3D printing, we were not able to develop a tangible verbal communication as well.
representation of the Chromati. Given the skills and the
time, we would like to eventually create iterations of a TESTING
final physical device. We believe there is no such thing as having too many users
to test any iteration of a design—we would anticipate
DEMONSTRATION reaching out to not just students at school but also users
The video that we created to demonstrate the who are hard of hearing to gain an outside perspective.
Chromati's functions only shows one mode, which is Getting a range of different ages and exclusions from
the small group setting and one scenario. We wish varying environments is important to get a spectrum of
to develop the demonstration video to portray all the responses from user testing.
interaction modes, as well as polish up aesthetics.
Chromati
THANK YOU.
Thank you to our Professor Dana & Graduate Assistant Will for guiding
& Thank you to all the students and USER testers , along with our