HOME VENTURE DESIGN LEARN/TEACH THE INTERDISCIPLINARIAN ABOUT ME

THE CUSTOMER DISCOVERY

HANDBOOK
Table of Contents
Persona & Problem Hypothesis
ValueHypothesis
Customer Creation Hypothesis
Usability Hypothesis
Creating Effective Screeners
Reference A: Enable Quiz Usability Test Plan
Reference B: Usability Test Plan forSmall Business Social Media Automation

Views: 7,594

32 90

This customer discovery handbook draws on long-standing practices from design

research and usability engineering. I think youll find it a useful complement to
your work indesign thinking, Lean Startup, Customer Development, and the
related topic of business model generation.

Discovery/design research spans everything a business does around making

things and selling things. The first step to success is structuring your ideas. Ive
organized thebalance of the handbook around the following five areas:

These areas are tightly related to the Venture Design process and I reference
those tutorials, templates and workshops below.
Persona & Problem Hypothesis

Ive paired these under a single

heading since most of the related
activities are done together. The key
questions youre trying to answer for
yourself are:

Who is my customer and do I

understand what makes them tick?

Have I identified one or more problems that are important to them?

What alternatives are my target personas using to deliver on the problem

today?

The persona is a humanized view of your customer, be they buyer and/or user of
your product. For background, see the tutorial here on Personas. By problem, I
mean an underlying job or need that exists for your persona(s). Anchoring to the
right problem (vs. presupposing a solution) is central to the practice of
design/design research. The aforementioned tutorial describes personas &
problem scenarios.

Your goal at the end of this is to have clear working answers to the above,
supported by notes that are vivid, actionable, testable for you and your
collaborators.

Preparation
High quality observation learning offers the only good
answers to the questions above. That said, I highly
recommend pushing yourself to 1) brainstorm a set of
personas and problem scenarios and then 2) pick the
most important and draft them.

Why? Ive found my students & advisees get the most

mileage out of the mentally and emotionally taxing work of
finding subjects and interviewing them when they have a
focused view of what they want to know. This section of the Venture Design page
has tutorials, examples, templates, and workshops you can use to draft your first
set of personas & problem scenarios: Tools for Creating Personas.

Once youre comfortable you know what you want to know, its time to move to
execution.

Execution
Execution
Ready to go discover who and where you can innovate and deliver value? Here
we go!

0. Know What You Want to Know

Before you go out and execute discovery, youll want to have a clear idea of what
you want to know. See the previous section on this.

1. Create a Screener

You have specific personas you want to learn about. What simple, factual question
can you ask a potential subject to be sure theyre relevant?

This is more important than it may seem. We have a natural bias to go with
subjects that are convenient & comfortable, which can dramatically limit
actionable learning. Dont blame yourself, but do screen yourself!

For example, lets say were interested in problem scenarios around some aspect
of network management, with the idea of possibly building an application for
network engineers to manage transport elements like routers and switches. We
have a persona(s) for the end user that we want to develop and validate. A good
screener would be: How many times last week did you log into a switch or
router?.

2. Design an Interview Guide

Not a survey and not a script, this a tool for you to organize your questions across
your objectives and available best practices, many of which youll learn and
incorporate as you go.

Do not worry about sticking to the script every time. You wont create a useful
persona by attempting statistically significant results.

Do worry about covering your bases on the persona and problem scenarios. Do
worry about updating the guide as you learn whats working for you and isnt (yes,
its OK to update the guide as you go along; it doesnt invalidate the data).

Heres an example of an interview guide for Enable Quiz, a startup-up thats

looking at problem scenarios around how HR managers screen technical recruits
for engineering jobs.

PERSONA HYPOTHESIS

Question Format Example Questions (Enable Quiz)

Tell me about [yourself in the role Tell me about being an HR manager?

of the persona]? How did you choose that line of work?
Why?
What do you most, least like about the job?
What are the hardest, easiest parts of the
job?
Ive heard [x]- does that apply to you?

Tell me about [your area of Do you do screen new candidates? If not,

interest]? who?
Can you tell me about the last time? What
was the trigger?
Who else was involved? What was it like?

Tell me your thoughts about Where do you learn whats new? What
[area]? others do?
Who do you think is doing it right?
How did you make your last decision?

What do you see in [area]? Where do you learn whats new? What
others do?
Who do you think is doing it right?
How did you make your last decision?

How do you feel about [area]? What motivates you? What parts of it are
most rewarding? Why?
Tell me about the last time?
What would it be like in your perfect world?

What do you do in [area]? Would you show me your interview guide?

Example notes?
What the vetting process was like on the
last few candidates?

PROBLEM HYPOTHESIS

Question Format Example Questions (Enable Quiz)

What are the top [5] hardest things about What are the top 5 most difficult
[area of interest]? things about making good tech
hires? Why?

How do you currently [operate in area of How do you currently screen for
interest- if you dont have that yet]? OR technical skill sets?Who does what?
Heres what I got on [x]- is that right? How does that work?

Whats [difficult, annoying] about [area of Whats difficult about screening

interest]? technical candidates?How do you
validate they have the right skill set?
How are the actual outcomes?
Examples?

What are the top 5 things you want to do What are the top 5 things you want
better this year in [general area of to do better in technical recruiting
interest]? and hiring?

Why is/isnt [your specific area of interest Why is/isnt screening for technical
on that list]? candidates on that list?

You may notice that the two parts of the interview guide overlap. Thats OK-
particularly if you observe that youre getting new and different responses
between the different forms of the question. Subjects will remember things they
wanted to tell you over the course of the interview and they will never understand
all the questions exactly the way you intended (thats just the limitations of human
communication).

For a template with these items, see Interview Guide (Venture Design Template).

3.Find Subjects

Are you finding that getting to the right interview subjects is difficult, messy, and
time-consuming? Good- youre probably doing it right.

If I knew an easier way, dear reader, I would tell you straight away. The reality is
that soliciting online and even using high-priced agencies tends to deliver atypical
subjects, semi-professional participants that are not a reliable or useful source of
actionable learning.

Find your subjects by hook or by crook and use the screener to make sure
everyones investing their time appropriately- thats the best way. Is your persona
an oncology nurse? Call your uncle across the country who does the books at a
hospital see who he knows. Half of those people will probably tell you never to
call them again (thats hyperbole). Take solace in the fact that quality learning at
this stage will have geometric benefits downstream.

Should you compensate them? Yes, in some fashion. Theyre generously giving
up their time for your venture. If youre a startup you probably cant afford to
compensate them at market rates. You should nevertheless present them some
token of your appreciation (Starbucks gift card, etc.), acknowledging that its just a
token and that youve appreciated their generosity with their time. Not only is it
appropriate and polite, but it will make it much more likely that they help you find
other subjects.
4. InterviewSubjects

Never, ever, ever, pitch subjectsyour product, or even advocate a particular point
of view. Your job is to learn something from thesubject that will allow you to scale
an offering to 100 then 1,000, then 10,000, etc. customers. Its not to sell them
something (or worse yet have them tell you what you want to hear).

That said, most subjects will not understand your approach or objective and that
will make them uncomfortable. Being forthright with them about what youre
doing is key. Explain to them what you want to learn. That is your objective (not
selling). For example, telling an HR manager Were trying to learn about whats its
like for HR managers to recruit technical talent. or telling a nurse Were trying to
learn about what its like for an oncology nurse to file paperwork. is fine and
useful.

Recording the session will make them uneasy at first, but most subjects will forget
about it in 5 minutes or so. (If you do record them, be sure to abide by applicable
laws and regulations.) That said, youll likely find you have little time to review
these transcripts. Most important is to make notes as you go along and to sit
down immediately after the interview and make notes on what you learned, ideas
that came to mind.

Avoid interrupting the subject (this will dam up the natural flow of information) but
balance that with the time you both have available and the topical coverage you
want.

Freely update your interview guide as you learn whats working. Dont worry
about it making the interviews non-comparable: your goal is not statistical
significance. The reason I use a Google Doc for a template is that it eases ad hoc
collaboration between team members.

4.Update Your Personas & Problem Scenarios

Theyre never done. Not to harp on the Google Docs angle (and its certainly not
the only place to put all this), but personas are the innovators spreadsheet.
Everything changes constantly, and the successful innovators uses their personas
(and problem scenarios, etc.) to keep track.

How do you know when youre in relatively good shape? The closing section in
this area offers a checklist.

Validation (Invalidation)
Youre never done, but the following are good indicators that for any given project
its time to move on to the next step.

CHECKLIST: PERSONA HYPOTHESIS

 Sub-Hypothesis
This persona exists (in non-
trivial numbers) and you can
identify them.
You understand this persona What kind of shoes do they wear?Are you
well.
Do you understand what
they Think in your area of
interest?
Do you understand what
they See in your area of
interest?
How do they Feel about
your area of interest?
Do you understand what
they Do in your area of
interest?
CHECKLIST: PROBLEM HYPOTHESIS
Sub-Hypothesis
Youve identified at least one
discrete problem ( job, desire, Do others get it?
etc.)
The problem is important
You understand current
Experiment
Can you think of 5-10 examples?
Can you set up discovery interviews with
them?
Can you connect with them in the market at
large?
hearing, seeing the same things across your
discovery interviews?
What do you they mention as important?
Difficult? Rewarding?
Do they see the work (or habit) as you do?
What would they like to do better? To be
better?
Where do they get their information? Peers?
Publications?
How do they decide whats OK? Whats
aspirational?
What are their triggers for this area?
Motivations?
What rewards do they seek? How do they
view past actions?
What do you actually observe them doing?
How can you directly or indirectly validate
thats what they do?
Experiment
Can you describe it in a sentence?
Can you identify current alternatives?
Do subjects mention it unprompted in
discovery interviews?
Do they respond to solicitation (see also
value and customer creation hypotheses)?
Have you seen them in action?
 alternatives Do you have artifacts (spreadsheets,
photos, posts, notes, whiteboard scribbles,
screen shots)?

ValueHypothesis
You cant ask a customer whether theyd like a
product youre thinking of creating (or even that
they may be thinking of buying). Theyll always
say sure. For more on this see the Story of the
Yellow Walkman.

This reality creates a stark boundary between the type of

work you can do to enhance your Persona Hypothesis and Problem Hypothesis
and the type of work that will help you test a Value Hypothesis.

For an actionable result on Value Hypothesis, you must incite a material exchange
of value. If you ask a subject Would you buy this yellow Walkman?, they will say
Sure. They want to be nice and they dont want to argue you with you. But if you
offerthem eithera black or a yellow walkman, one of which they actually get to
take home, then you have a material exchange of value.

Heres another example with a smaller absolute exchange of value but one which
can still be material. If you spend an hour talking with a subject and then, standing
over their shoulder, ask if theyd like to sign up for your email newsletter, thats not
a material exchange of value. The awkwardness of refusing far outweighs the cost
of unsubscribing or just deleting the emails. But what if someone youve never
met learns about your offer on the Internet and subscribes? Yes, then thats a
material exchange of value.

Preparation
I highly recommend having a working view of your Persona Hypothesis & Persona
Hypothesis before you approach this area (see above). The outputs will give you
what you need to create a working Product Hypothesis:

Investing time in understanding your focal personas and problem scenarios

avoids excessive waste in formulating and testing your Value Hypothesis.

I also recommend a quick think on your engines of growth (a term coined by Eric
Ries). The proposition here is that there are three principal engines of growth and
a new ventures knows which one is most important:

Viral customers/users tell each other about the offer. Crucial here is some
measurement of viral coefficient, the propensity of one customer/user to share in
some fashion the offer with others. In this case, your ability to drive sharing/word-
of-mouth is crucial.

Paid you have a certain cost of customer acquisition based on the use of
marketing and/or sales resources. Here, ascertaining the cost of acquisition and
the value of a customer are key to understanding the validity of your unit
economics.

Sticky the lifetime value of customers is very high because the relationship will
deepen over time. Here, testing your ability to retain and maximize the lifetime
value of a customer relationship is key.

The validity of the fundamental Product Hypothesis and your point of view on a
principal engine of growth are important focal items for the testing of a Value
Hypothesis.

Execution
How do you engineer an exchange of value that tests your Product Hypothesis
and (ideally) engine of growth without wasting money building someone that no
one wants? Thats what the discipline of Lean Startup is about, specifically the use
of an MVP.

The MVPis an execution that is viable to definitively test your Value Hypothesis
with a minimum of resources. The product term is a kind of necessary misnomer:
if you can possibly help it, the point of Lean Startup is that you should avoid
building a full-blown product.

This is a tricky, inexact science and it rubs against the grain of permanence and
durability that runs strong in most of us.The good news is that there are several
well established patters for creating an MVP, minimum viable product. Here are a
few:

MVP Notes
Pattern

The Wizard Create a realistic demo recording/rendering of your product, post

of OZ MVP it online, drive some traffic to it, and see how you do on sign-ups
and sharing. This is a pattern that DropBox initially used.

The Google This is a popular pattern for both new products and features- see if
AdWords you can generate click-throughs and sign-ups from a Google
MVP AdWords campaign. This is especially useful if your engine of
growth has certain assumptions about paid CPA (cost-per-
acquisition).

The Pre- Theres really nothing as definitive as getting customers to pre-pay

Sales MVP (or even sign a letter-of-intent; its still an exchange of value) for
your product, assuming the price point is viable. Make sure your
pitch and the transaction are clear.

The Have the customer submit inputs on the front end and you
Concierge manually execute whatever it is your product would do on the
MVP back end.We did this at a photo-social startup I was advising:
(Strong rather that creating an app to do something (hypothetically)
Form) exciting with a set of photos, I challenged the team to execute
hypothetically exciting action by hand. If the output was popular on
social media, thats a validation signal. If not, it isnt.See also the
Howe & Associates example on the case studies.

The Using consulting as a prelude to (hypothetically) building a piece

Concierge of software is a great way to discover and validate all your
MVP hypothesis areas.Leonid Systems used consulting as a vehicle to
(Consulting) solve customer problems while evaluating problem areas that
looked ripe for automation and standardization with software.

These are all described in more detail on the Lean Startup tutorial, case studies
section.

Validation (Invalidation)
The thing that separates a hypothesis from a notion is that it has a testable
formation, one that allows a structured experiment to prove or disprove it. Your
venture (whether its a startup or an internal project) is likely in a rush and starved
for resources. Ive been there.

Its important to make a habit of thinking your MVP experiments all the way
through, and youll probably find that once you get used to it, it really doesnt add
that much time on the front end. Before you invest in executing the MVP, think
through how youll interpret the results for decision making.
What metrics (quantitative or qualitative) constitute validation? Invalidation? What
will you do on that basis? (Generally, invest in scaling up the initiative if it validates
and re-formulating+re-testing if it doesnt). Even if its a simple email, write up the
report youll issue on the experiment (with placeholders for results) beforehand.

Customer Creation Hypothesis

These hypotheses only make sense after youve got
fundamental validation on your core Value Hypothesis (see
above). You cant market/growth hack your way around a
lack of product/market fit.

Once youve got that basic product/market fit, its time to

start observing, hypothesizing, and experimenting against
some kind of customer acquisition funnel. I like the AIDAOR
model (attention-interest-desire-action-onboarding-retention):

Why is that funnel so important? Because if you dont break down the
question/problem, youll almost certainly end up mired in confusion. Your funnel is
your anchor point for both qualitative and quantitative data. I recommend starting
with qualitative ideas since that will help you with the why? and drive better
hypotheses. One technique I like for that is storyboarding. You might have
multiple takes on this, which is fine/great. Heres an example from Enable Quiz, a
fictional company that makes online quizzes that HR managers can use to screen
engineering candidates:
For more on doing this, see the corresponding section of the storyboarding
tutorial here- Storyboarding for Growth.

Now its time to form hypotheses. An example for attention might be something
like:

If we deliver [a certain Google AdWord ad] against [a certain set of keywords],

well see a click-through-rate of [x%]. You might have a similar hypotheses later in
the funnel around pricing or certain offers. You might have others for content
marketing. One really important thing is to take an integrated view of your
marketing mix/brand experience. My favorite tool for that is the Growth Hacking
Canvas:
Its a lot of stuff- I know! But were moving toward a pretty firm realization that just
like siloed product development doesnt work (and so we need agile), siloed
marketing doesnt work either. For more on using the Canvas, check out this
tutorial: Growth Hacking Canvas.

Growth is hard and in my experience, everyone thinks theyre the only ones that
dont get it and everyone else is killing it. Its not true. Just take a disciplined
approach, keep experimenting, keep observing, and youll get there.

Usability Hypothesis

Reusing existing best-practice interface models and

maintaining a test-driven orientation are probably the best
ways to consistently succeed on usability. The usability of
any interface changeis unknown, even with the best
designers. Treating each change as hypothetical avoids
waste and expensive mistakes.

In the early phases of a big change (or new product), push yourself to diverge,
developing multiple possible directions and completing preliminary testing on
your favorites. The successful innovator is also a good economist and the
economy of broad prototyping in the early phases is clear:

Ive organized the balance of section against three progressiveapproaches to

testing: exploratory, assessment, and validation. These arent mutually exclusive-
commonly, youll move from one to the other.

Exploratory testing advances the discovery process with the benefit of early user
contact. Youll start with an explicit prototype and test plan, but freely update them
on a test by test basis as you learn whats working and your observations
encourage new ideas. Specific measurements are unimportant. Popular tools are
paper prototypes, PowerPoint/Keynotes prototypes, and various other facile tools
that discourage overemphasis on details and encourage experimentation and
variation.Exploratory testing drives to a decision about the fundamental approach
on aninterface.

Assessment testing focuses on measuring the fundamental efficacy of a new

interface (or interface element). As opposed to validation testing, its less
concerned with detailed measurements on specific items, but instead on whether
or not most users can accomplish topline goals, noting the rough spots.
Assessment testing typically provides inputs tofurther interface refinement (if its
basically successful) or pivot and revision (if it is not).

As the name suggests, validation testing is for once youre relatively sure youve
got an acceptable interface, and you want a) a final check on that and b) some
intuitive sense about the nature of what youll see with measurements you take
out in the wild (through Google Analytics, etc. with users you dont have the
chance to meet and observe directly). Try hard to leave yourself enough time to
make at least a few minor changes before releasing to the public- youll likely
identify a few rough spots well worth fixing.

You can also execute all three of these tests on a comparative basis (comparison
between alternatives).

Preparation
Relative to customer discovery interviews, user testing requires a relatively
detailed set up. As with any detail-orientedtask, its easy to get lost in the weeds.
Resist the false gratification that task completion provides: the important thing is
connecting what youve learned back to your core innovation engine.

User testing occurs relatively late in the overall discovery process. At this point,
you should have validated personas, problem scenarios, and value propositions,
something you can summarize in a Product Hypothesis. The diagram below
describes all this in terms of the convergence-divergence pattern from design
thinking. Your work on the Persona & Problem Hypothesis should ultimately
converge on a focal problem set, to which you attach your value propositions. All
this you can summarize in a Product Hypothesis to connect problem with solution.

The Product Hypothesis will include a Value Hypothesis and thatwill have certain
assumptions attached to it.You should test those assumptions with thequickest,
least expensive MVP product proxies you think are viable to provide a definitive,
actionable result (see section on Value Hypothesis for popular patterns). Once
youve converged on which problems you can provide adequately valuable
solutions, encapsulate your solution ideas in agile user stories. Then prototype,
test, refine, and test some more!
Every substantial interaction you plan to test should have a user story attached to
it. The user stories will anchor your executions in the customer narrative and
validated learning youve developed. For every material interface element, you
should be able to answer these questions:

What user story did we design & execute this against?

What value proposition was that story delivering against? Why do we believe
the user wants to do this?

What problem are we solving for the user with this value proposition?

Who is this user? Do we understand what makes them tick? What shoes they
might wear?

This will not only help you converge onmore valuable solutions; it will also help
you zero in on the right place to start revisions when your execution doesnt go
perfectly (and it never does!). Also, successful innovation is a loop, not a line. Ive
presented most of the material as a sequence for clarity, but the successful
innovator is constantly looping back through the hypothesis areas. People
change, problems change (a little), and every solution is temporary.

Execution
0. Prepare

User testing will deliver useful results on the investment just about all the time.
Dont feel like you absolutelycantdo it if you havent completed the items in the
preceding section. That said, a little investment in those other areas (personas,
problem, MVP testing) will go a long way.

1. DecideTest Type
Is your primary test objective exploratory, assessment, or validation? You can have
elements of each, but I strongly advise deciding in advance your principal focus.
The implications for preparation, execution, and decision (post-test) or
substantially different between the three. Clearly explain to the rest of your team
the focus- that will help bring the right focus to the testing.

Heres a brief assessment, if youre having trouble deciding

Youre in the early phases of creating a new interface/interface element:

Exploratory

Youve drafted one or more directions that are well supported by validated
learning and a review of comparables: Assessment

You think youre done: Validation

2A. Prepare Research Design

This has several sections in addition to the test plan- I call use the term research
suite to designate the whole package for a given set of testing. At first, it may
look like a lot of stuff, but every time Ive done user testing Ive found all this
preparation well worthwhile (and the reverse is also true). The example and
template I reference are in the Venture Design Template/Appendix B. Notes on
preparing the various sections of the research suite follow:

:: Objectives & Methods

This is for you to use internally. Describe in the clearest possible terms what you
want to have happen as a result of the testing and your principal testing focus
(exploratory, assessment, or validation), linking that to your objectives. Particularly
if youre early in a project, you may want to couple additional customer discovery
interviews (persona, problem hypothesis) with your user testing. This is the place
to explain that.

Heres an example from Enable Quiz:

There are three general types of tests:- Exploratory: for learning about customer
problem scenarios in greater detail, sometimes with a paper or working
prototype
Assessment: for testing the usability of an early direction on product
implementation

Validation: for later stage final testing of an implementation

This test suite is exploratory and were preceding the user testing with customer
discovery interviews from Appendix A [this is the appendix for customer
discovery interviews in the template]to deepen and align our view of personas
and problem scenarios with the exploratory test results.

::Product Version

Note the version youre planning to use for testing.See the next step (2B)on
defining a test release.Preview, though, so you dont get stuck:this need not be
actual software or a real release- it could be paper prototypes or a prototype you
make in PowerPoint or Keynote.

:: Subjects

Define your target subject count in terms of the personas youve identified (see
above or tutorial on personas). Youre not trying to achieve statistical significance
in most cases (and almost always in the early phases), so 5-10 subjects is perfectly
OK.

Heres an example from Enable Quiz:

Since enabling the HR manager persona to be more effective is central to our

value proposition, our target weighting of subjects should reflects that. An ideal
total and mix of subjects would be:
Helen (or Hank!) the HR Manager 4Frank the Functional Manager 1-2The
screening question for both these subjects type is:
How many technical job candidates did you interviewin the last month?

:: Research Composition

Here you will summarize everything that happens to a subject and how long you
think it will take. Heres an example from Enable Quiz:

Duration
#Item Notes
(min.)

Here we will explain the objectives of the test and the

Intro. &
1 5 parameters of their participation. Well also obtain the
Explanation
designated release & consent form*.

Using the interview guide, well spend a few minutes to

Discovery
2 20 discovery to improve our personas, problem scenarios
Questions
and user stories.

3Test Tasks 15 Well introduce the test scenario and then ask them to
 complete the Test Items.

Post-Test Make sure we ask if it is OK to follow-up with additional

4 5
Debrief questions.

::Pre-Session Checklist

Theres nothing worse than starting of a test with something not being ready or
general not in the right state. This is a simple checklist to help you and/or your
collaborators make sure they dont have any false starts. Heres an Example from
Enable Quiz:

#Item Notes

Make sure have written versions of discovery and

1
test questions to refer to

log in- make sure app is

2Make sure test instance is up and functional
on starting page

Make sure recording equipment* is up and

3
functional

:: Session Design

This includes the intro youll do with subjects as well as the test items.

I strongly advise writing up the intro and practicing it- it takes work to put yourself
in the subjects shoes and as things get busy (and repetitive) youll easily miss
things.

On the test items, youll notice each row has four items

Enumeration (#): This is just for reference

Research Objective: This will help keep you focused. Each item should have a
research objective (otherwise, why is it there?). If youre running an exploratory or
assessment test, your user stories can provide a great anchor for the objective
(see example below).

Estimated vs. Actual Time: This is for setting expectations on duration as well as
evaluation. If youre running an exploratory or assessment test, youll be less
concerned with actuals.

Notes: This is where you set up and design the testing. I like to break each of
these into a set of notes for the moderator and a set of target outputs. The
outputs should closely and obviously tie to the research objective.

Heres an Example from Enable Quiz:

Intro
Thanks for making time to take part in our study. My names [name] and this is
[observer]. [Explain participation and deal with consent form/obtain written
consent]*Well be using a test guide through the rest of this, so I hope you wont
mind me referring to that.Were here to learn about [an early version of a
solution that allows HR managers to assess the technical skill set of a job
candidate through an online quiz].Im going to ask you some questions and give
you some tasks. Feel free to think out loud if youre comfortable with that. Were
not here to test you and there are no wrong answers. Our results just help us
better understand our product and its audience.The session will take roughly
[40-60] minutes.Do you have any questions before we start?Test Items

Est. v.

#Research Objective ActualNotes

(min.)

MODERATOR GUIDE

Lets say your job is to create one of

these quizzes for an open position.

Heres a description of the position

[Provide them sample job description

and let them review.]. Let me know when

1 Exploratory Intro 5
youve finished reviewing it and if you

have any questions.OUTPUT

Validation that the subject understands

their goal and the job description,

roughly as well as they would in their

current position.

2 Assess primary navigation for 2 MODERATOR GUIDE

new quiz creation Lets say you want to create a new quiz.
What would you do?OUTPUT
 Assessment of primary navigation for
new quiz creation

How are we doing on this user
story:As an HR manager, I
want to match an open
positions required skills with
3 quiz topics so I can create a
quiz relevant for candidate
screening.
?
MODERATOR GUIDE
Tell me what you think youre seeing
here?Lets say you wanted to choose a
set of quiz topics for the open position
you just reviewed. Show me how youd
5
do that?OUTPUT
An assessment of the users relationship
to the available affordances and their
appropriateness to the current user
narratives and tasks.

NOTES ON TAKEAWAYS

Personas & Problem Scenarios []

UI and User Stories []

:: Post-Test Debrief

Do you really need this? Yes, probably. This is an after the fact checklist to make
sure you cover your bases: seeing if follow-up questions are OK, compensating
the subject (in whatever way you plan), seeing if they have other thoughts, seeing
if they have ideas on other subjects. Heres an example from Enable Quiz:

Thanks so much. Well be using this to make the product and solution stuff like
documentation better.
Would you mind if we send you follow-up questions?
(if youre giving them some kind of tangible thank you, make sure that gets
done)
*:: Note on Recording and Compliance

I dont supply legal advice on this site and I dont warrant these notes as fit for
legal compliance. As well it should be, recording individuals is subject to various
laws and regulations depending on who you are, who they are, where you are,
and how the recording will be used and stored (among other factors). Its
important that you get advice from your legal counsel and maintain transparency
and applicable compliance with your subjects. At a minimum, this means securing
written releases for the recordings and making sure that the recordings are stored
and accessed securely (if you store them at all). Regarding releases and consent,
your specific compliance requirements will vary, but here are a few sample
consent forms from US institutions:

Usability.gov

Indiana University

2B. Prepare Test Release

If youre working on an actual piece of software, test whats current but dont
make yourself crazy (and probably your subjects) by cutting it too close.

If you want to do an exploratory or assessment test against a prototype, say a

Balsamiq mockup, there are alternatives to building working software.

:: Paper Prototype

Yes, you can actually get meaningful test results from playing with pieces of paper.
Its hard to believe until you do it.

To start, you need a prototype. The Balsamiq prototying process will serve you
well, assuming your subject is software. You will essentially prepare a set of
screens on paper and ask the user to interact with them- clicking (by pointing) and
typing (by writing with a pen).

Modularity and layering will serve you well in your preparation. I recommend
having a few templates that are regular paper layered on cardboard or a similar
substrate. This will make it easier to physically handle the prototypes and
exchange them with the user. The base template should look like your target
device- phone, tablet, laptop, etc.

Then layer basic screens on top of those (with light paste or spray adhesive which
you can buy at any craft store). On top of those you can layer additional controls
(Balsamiq lends itself to modular disposition of controls). And finally on top of the
controls you can layer Post-Its (or strips thereof) onto which users can type
(photograph the results after tests and then just replace the Post-Its).
Try it a few times and youll probably find youre not uncomfortable with the
process.

::PowerPoint or Keynote Prototype

This is the same basic idea as paper prototyping but youre simulating the
interaction with inter-slide links on PPT or Keynote. The advantage is that
everythingson the computer if youre not a glue-and-scisssors fan, and the
experience may feel more real to subjects. The disadvantage is that the linking
can get confusing, improvisation is harder, and if you want the user to fill out text
youll need to have a paper formfor them anyway.

Create the various (static) screens you want as slides within your application of
choice. Then add inter-slide links. In the current version of PowerPoint (Mac; Ill
guess its the same on PC but havent been able to check), you do this by:

two-finger (or right) clicking on a shape

selecting Hyperlink

-then selecting Document in the pop-up

and using the Locate button to find an Anchor (youll need to click the right
triangle to unfurl the list of slides).

On Keynote its simpler: two-finger/right click a shape, select Add Link or Edit
Link if you have one in place, and then select the target slide.

3. Prepare Test Infrastructure

When the alternative is doing nothing, you can finish a darn good test by sitting
someone down in front of what you have, giving them a few goals to complete,
and seeing what happens.

Few of youwill have access to observation booths etc., so Ill skip that.

If you have the team size, separating the facilitator and observer/note-taker
functions is very helpful, leaving the facilitator free to focus on the experience of
the subject.

Make sure the facilitatoris close by, but ideally not immediately visible or over the
shoulder of the subject. A good location is between the subjects 4/5-oclock and
2-4 distant. The observers will generally sit behind the subject- as far away as
possible where they can still see whats happening.
A simple PC/Mac with a web-cam will do fine. For recording screen activity and a
web-cam feed on a PC, I like Camtasia Studio. For the Mac, I use ScreenFlow.
Make sure you have everything recording and rendering the way you expect
beforehand.

Note: You have serious obligations (ethical and legal) to steward and safeguard
your subjects privacy and obtain their explicit agreement on participation,
particularly if youre recording. See the above note on Recording & Compliance.

4. Obtain Subjects

First off, if youre trying out a new test set up (not to mention if youve never done
this before), find some subjects where you can test the test. This is anyone who
could plausibly use the product, even if they dont well represent one of your
target personas. Things will break, youll fix them, dont worry, its natural.

Following this, prepare a screener- simple, factual question or questions to quality

the relevance of subjects. With usability you can be a little more lenient than with
development of your persona and problem hypothesis, but watch for the bias
towards subjects that are convenient & comfortable vs. relevant.

6. ExecuteTest Suite

If you have a research suite along the lines of whats above/in the template, then
you have a plan.

In working the plan, practice is the best tutor. Be careful not to coach subjects too
much or make them feel judged. Its painful to watch them struggle with
something they dont understand, but better to learn about that now than subject
every future user of your product to it! Give them time to work through confusion.
Eventually (set a threshold for yourself) youll need to help them move forward,
but make sure you dont do it too soon.

Dont forget to thank your subject, compensate them (in whatever way you plan),
and ask them if follow-ups are OK.

8. Make Your Notes (ASAP!)

I recommend doing this right away. Most of the important insights youll have,
youll have on the spot.

Validation (Invalidation)
As with any test, conclusions are the point. Success/a good result will vary by test
type.

Exploratory: The results should help you better understand the likely journey of a
typical user and, depending on where you are in designing/prototyping the
interface, whether youre headed in a workable direction. Comparison tests here
are highly desirable given their low cost and possible impact.

Assessment: The key question here is whether the pivot or persevere on a given
direction. Lots of stuck and/or frustrated users means no. Be ready to iterate a lot-
the change you need may not be radical. Comparison testing is also highly
economical here.

Validation: Here youll generally have a quantitative target for time spent per task
and in total on your major experience arcs. Validationis being within a reasonable
deviation from that.

Creating Effective Screeners

Its not hard to spend 45 minutes with a subject only to realize theyre really not
the subject youre looking for. Particularly if you dive into detail early, they may be
informative enough to keep you going even if youre on a road to nowhere. This
applies to all the hypothesis types and research techniques above.

For this, we create screeners. Basically the screener is a simple, factual question
or set of questions can you ask a potential subject to be sure theyre relevant. It
shouldnt take much.

For example, lets say were interested in problem scenarios around some aspect
of network management, with the idea of possibly building an application for
network engineers to manage transport elements like routers and switches. We
have a persona(s) for the end user that we want to develop and validate. A good
screener would be: How many times last week did you log into a switch or
router?. Lets say were building software for plumbers. A good screener would
be: How many plumbing jobs were you out on last week?.

The screeneris more important than you might guess at first. We have a natural
bias to go with subjects that are convenient & comfortable, which can dramatically
limit actionable learning. Dont blame yourself, but do screen yourself!

Youll find both the Enable Quiz example usability test plan as well as another that
tests automation platforms for social media (Hootsuite, Buffer, etc.) in the
References at the end of this page

WANT TO LEARN CUSTOMER DISCOVERY?

RECOMMENDED:
CHECK IT OUT
Course 1: Getting Started: Agile Meets Design Thinking


Reference A: Enable Quiz Usability Test Plan

Objectives & Methods
There are three general types of tests:
Exploratory: for learning about customer problem scenarios in greater detail,
sometimes with a paper or working prototype
Assessment: for testing the usability of an early direction on product
implementation
Validation: for later stage final testing of an implementation

This test suite is exploratory and were preceding the user testing with customer
discovery interviews to deepen and align our view of personas and problem
scenarios with the exploratory test results.

Product Version
Well be using version [0.1] of the product for this exploratory test. [NOTE: They
could easily be using paper or PowerPoint prototypes as this stage as well]

Subjects
Since enabling the HR manager persona to be more effective is central to our
value proposition, our target weighting of subjects should reflects that. An ideal
total and mix of subjects would be:

Helen (or Hank!) the HR Manager 4

Frank the Functional Manager 1-2

The screening question for both these subjects type are-

Helen (or Hank!) the HR Manager: How many technical job candidates did you
screen in the last month?

Research Composition

# Item Duration Notes

(min.)

1 Intro. & 5 Here we will explain the objectives of the test and
Explanation the parameters of their participation. Well also
obtain the designated release & consent form*.

2 Discovery 20 Using the interview guide, well spend a few minutes

Questions to discovery to improve our personas, problem
scenarios and user stories.
3 Test Tasks 15 Well introduce the test scenario and then ask them
to complete the Test Items.

4 Post-Test 5 Make sure we ask if it is OK to follow-up with

Debrief additional questions.

Pre-Session Checklist

# Item Notes

1 Make sure have written versions of discovery and

test questions to refer to

2 Make sure test instance is up and functional log in

make sure app is on
starting page

3 Make sure recording equipment* is up and

functional

Session Design
Intro

Thanks for making time to take part in our study. My names [name] and this is
[observer]. [Explain participation and deal with consent form/obtain written
consent]*

Well be using a test guide through the rest of this, so I hope you wont mind me
referring to that.

Were here to learn about [an early version of a solution that allows HR managers
to assess the technical skill set of a job candidate through an online quiz].

Im going to ask you some questions and give you some tasks. Feel free to think
out loud if youre comfortable with that. Were not here to test you and there are
no wrong answers. Our results just help us better understand our product and its
audience.

The session will take roughly [40-60] minutes.

Do you have any questions before we start?

Test Items

# Research Objective Est. v. Notes

Actual
(min.)
1 Exploratory Intro
2 Assess primary 2 MODERATOR GUIDE
navigation for new quiz
creation
3 How are we doing on 5 MODERATOR GUIDE
this user story:
As an HR manager, I
want to match an open
positions required skills
with quiz topics so I can
create a quiz relevant for
candidate screening.
? to the available affordances and their
NOTES ON TAKEAWAYS
Personas & Problem Scenarios
5 MODERATOR GUIDE
Lets say your job is to create one of
these quizzes for an open position.
Heres a description of the position
[Provide them sample job description and
let them review.]. Let me know when
youve finished reviewing it and if you
have any questions.
OUTPUT
Validation that the subject understands
their goal and the job description,
roughly as well as they would in their
current position.
Lets say you want to create a new quiz.
What would you do?
OUTPUT
Assessment of primary navigation for
new quiz creation
Tell me what you think youre seeing
here?
Lets say you wanted to choose a set of
quiz topics for the open position you just
reviewed. Show me how youd do that?
OUTPUT
An assessment of the users relationship
appropriateness to the current user
narratives and tasks.
[]
UI and User Stories []

Post-Test Debrief
Thanks so much. Well be using this to make the product and solution stuff like
documentation better.

Would you mind if we send you follow-up questions?

(if youre giving them some kind of tangible thank you, make sure that gets
done)

* I dont supply legal advice on this site and I dont warrant these notes as fit for
legal compliance. As well it should be, recording individuals is subject to various
laws and regulations depending on who you are, who they are, where you are,
and how the recording will be used and stored (among other factors). Its
important that you get advice from your legal counsel and maintain transparency
and applicable compliance with your subjects. At a minimum, this means securing
written releases for the recordings and making sure that the recordings are stored
and accessed securely (if you store them at all). Regarding releases and consent,
your specific compliance requirements will vary, but here are a few sample
consent forms from US institutions:

Usability.gov

Indiana University

Reference B: Usability Test Plan forSmall Business Social Media

Automation
Screener
How many times last month did you post to social media for your business? What
services did you use?

Objectives & Methods

There are three general types of tests:
Exploratory: for learning about customer problem scenarios in greater detail,
sometimes with a paper or working prototype
Assessment: for testing the usability of an early direction on product
implementation
Validation: for later stage final testing of an implementation

This test suite is an assessment test.

Product Version
Well be using version [x.y] of the product for this exploratory test. [NOTE: They
could easily be using paper or PowerPoint prototypes as this stage as well]

Subjects
Our core subject has a small company or personal brand theyre promoting. [XYZ],
is central to our value proposition of [ABC] so were targeting a composition of
subjects as follows (organized against our personas):

Sam the Small Business Owner 4

Research Composition

# Item Duration Notes

(min.)

1 Intro. & 5 Here we will explain the objectives of the test and
Explanation the parameters of their participation. Well also
obtain the designated release & consent form*.

2 Test Tasks 15 Well introduce the test scenario and then ask them
to complete the Test Items.

4 Post-Test 5 Make sure we ask if it is OK to follow-up with

Debrief additional questions.

Pre-Session Checklist

# Item Notes

1 Make sure have written versions of discovery and test

questions to refer to

2 Make sure test instance is up and functional log in

make sure app
is on starting
page

3 Make sure the subject doesnt have an account on [social

media automation system] already

4 Make sure they have accounts on at least two of: FB,

Twitter, LinkedIn, G+ and they know their username and
password

Session Design
Intro
Thanks for making time to take part in our study. My names [name]. [Explain
participation and deal with consent form/obtain written consent]*

Well be using a test guide through the rest of this, so I hope you wont mind me
referring to that.

Were here to learn about a product that helps individuals and teams manage
social media accounts.

Im going to ask you some questions and give you some tasks. Feel free to think
out loud if youre comfortable with that. Were not here to test you and there are
no wrong answers. Our results just help us better understand our product and its
audience.

The session will take roughly [40-60] minutes.

Do you have any questions before we start?

Test Items

# Research Objective Est. v. Notes

Actual
(min.)

1 How are we doing on this user 5 MODERATOR GUIDE

story: Lets say youve heard that what
As a Sam the Small Business youre seeing here [name] is a
Owner, I want to sign up for the great tool for managing social
service, so I can give it a try. media marketing for your
business or personal brand and
? you want to give it a try.
What would you do here?
OUTPUT

Validation that the subject

understands and can complete
the signup process.

2 Assess start page and major 2 MODERATOR GUIDE

navigation elements What do you think youre seeing
here?
OUTPUT

Assessment of users relationship

to the available affordances, their

3 How are we doing on this user
story:
As a Sam the Small Business
Owner, I want to I want to
connect my social media
accounts, so I can create
automated posts to them.
? would you do that?
4 How are we doing on this user
story:
As Sam the Small Business
Owner, I want to schedule some
posts for two days from now at
8AM [local time zone], so I know
its going to post at the time I
want.
? www.alexandercowan.com
prominence in the presentation,
and their appropriateness to the
top user narratives and tasks.
5 MODERATOR GUIDE
I have it in the notes here that
you post to LinkedIn, Twitter, a
Facebook page, and G+. Is that
right?
Lets say you wanted to add your
Twitter and LinkedIn accounts.
Would you show me how you
OUTPUT
An assessment of the users
relationship to the available
affordances and their
appropriateness to the current
user narratives and tasks.
MODERATOR GUIDE
Lets say you wanted to compose
and schedule a posting for 2
days now at 8AM [local time
zone]. The text is:
Free fries from 3-6pm today!
and the url is
Ill help you make sure its not
accidentally posted and well
delete it at the end. Lets say you
want it to post to both Twitter and
LinkedIn. How would you do
that?
OUTPUT
An assessment of the users
relationship to the available
affordances and their
 appropriateness to the current
user narratives and tasks.

5 How are we doing on this user MODERATOR GUIDE

story:
As Sam the Small Business
Owner, I want to update the time
on a scheduled event, so I know
its going to post at the time I
want.
Lets say you wanted to change
the items you just scheduled to
post at 10am instead of 8am.
Would you show me how you
would do that?
OUTPUT

? An assessment of the users

relationship to the available
affordances and their
appropriateness to the current
user narratives and tasks.

6 How are we doing on this user MODERATOR GUIDE

story:
As Sam the Small Business
Owner, I want to remove a
scheduled event, so I know its
not going to post.
Lets say you wanted to remove
that event so nothing goes out at
all. Would you show me how you
would do that?
OUTPUT

? An assessment of the users

relationship to the available
affordances and their
appropriateness to the current
user narratives and tasks.

NOTES ON TAKEAWAYS

Personas & Problem Scenarios []

UI and User Stories []

Post-Test Debrief
Thanks so much. Well be using this to make the product and solution stuff like
documentation better.
Would you mind if we send you follow-up questions?
(if youre giving them some kind of tangible thank you, make sure that gets
done)
 (make sure to delete their accounts and all login, password, and personal
identifying data)

* I dont supply legal advice on this site and I dont warrant these notes as fit for
legal compliance. As well it should be, recording individuals is subject to various
laws and regulations depending on who you are, who they are, where you are,
and how the recording will be used and stored (among other factors). Its
important that you get advice from your legal counsel and maintain transparency
and applicable compliance with your subjects. At a minimum, this means securing
written releases for the recordings and making sure that the recordings are stored
and accessed securely (if you store them at all). Regarding releases and consent,
your specific compliance requirements will vary, but here are a few sample
consent forms from US institutions:

Usability.gov

Indiana University

32 90

Filed Under: Product Design

6Comments AlexCowan Login

Recommend 2 Share SortbyBest

Jointhediscussion

j_mes5monthsago
I'dlovetoknowwhenyou'replanningtoreleasetheCustomerCreationHypothesis,asit'swhatI'mstrugglingwithmost.
Doyouhaveanyotherresourcesyoucouldrecommendinthemeantime.

ThanksagainforyourcontentAlex,Ican'tunderstandwhythisisn'tmorepopularit'sincredible!
1 Reply Share

AlexCowan Mod >j_mes 4monthsago

HiJ_MesThanks!I'msogladyoulikeit.CustomerCreationHypothesis:ImadeasmallupdateandmoreinFeb.,
butalotoftheactionishere:http://www.alexandercowan.c....

Whyisn'tthismorepopular?Whatagreatquestion!Partofitisbychoicethere'sdesignrealityanddesigntheater.
Thesetutorialsaredetailedbecausethey'reforseriouspractitioners.That'snotappealingtoahugeaudiencealot
ofpeopleprefersimpleranswers.I'mkindofremindedofthisTweetbyKentBeckaboutTDD:

t
.

Anyway,thanksforwritingandI'llhavemorematerialupsoon.
1 Reply Share

j_mes>AlexCowan4monthsago
ThanksfortheresponseAlex.Lookingforwardtowhatmoreyouhavetoshare!
1 Reply Share

Jaffy6monthsago
ThisISwhatIwassearchingfor.IsavedinEvernote,becausethisneedstoexistsinthefuture.Thanksfromarandom
person.
1 Reply Share

AlexCowan Mod >Jaffy 5monthsago

HiJaffyThanksforwriting,andI'msogladyoulikedit!
Reply Share

Alex4monthsago
HiAlex,
Thankyouforyourmaterials.
PleasecheckthefirstsentenceinPreparationsubsectionofValueHypothesissection:
"IhighlyrecommendhavingaworkingviewofyourPersonaHypothesis&PersonaHypothesisbeforeyouapproachthis
area(seeabove)"
ItseemssecondwordPersonashouldbereplacedwithProblem
Reply Share

ALSOONALEXCOWAN

PersonasforDesign,Development,&Growth BeyondDesignThinkingThinkLikeaShrink!
22comments3yearsago 4comments3yearsago
FelipeBassanAlex,Thankyousomuchforsharing.The AlexCowanHiPatricia,Thanks!I'msogladyoulikedit.
thingsI'mlearningonCourseraishelpingmetochange Alex
drasticallythewaymyteamswork.

CustomerDiscoveryinAction Theendofmegalomaniainhightechbranding?
3comments3yearsago 3comments3yearsago
BrennonBortz Howsoon??:) JoniPauloVeryinformative.Thisblogentryjust
confirmedthatIamontherighttrackforournewservices
andadditionalsites.ThanksAlex.:)

Subscribe d AddDisqustoyoursiteAddDisqusAdd Privacy

 THE VENTURE DESIGN PROCESS
You are here.

OVERVIEW

GO DEEPER:
Want to make innovation
an everyday thing?
RECOMMENDED:
Agile Development
Specialization on Coursera

CHECK IT OUT

LETS STAY IN TOUCH

JOIN ME ON:
AlexCowan@cowanSF
NewPoston@HuffPost:'HowHasSiliconValley
ChangedSincethe1990s?'buff.ly/2q3GXKo

HowHasSiliconValleyChangedSincethe1990s?
InwhatwayshasSiliconValleychangedsincethelate90sandhowdoesit
remainthesame?originallyappearedonQuoratheplacetogainand
sharekn...
huffingtonpost.com

19h

Copyright 2016 Alex Cowan All rights reserved.