Metrics That Matter

Creating custom analytics that quantify user behaviors

and drive business practices.
How do you measure success? Many online
businesses focus on dashboard-friendly metrics
that answer questions like: How many new views
did we have today? How many new daily active
users?How much revenue did we bring in last
week? Good information to haveand its never a
bad idea to get people thinking about databut
are you really getting the information you need
to make the best-possible decisions?

Define

Modern analytics is about so much more than defining highlevel KPIs and building simplistic dashboards to amplify
them. With very little additional work, you can create
custom analytics that inform product design and business
growth. Its as straightforward as defining the behaviors
you want to quantify, measuring them in a logical way, and

Quantify

then pushing them through product and process changes

ultimately making for happier customers, more efficient
operations, and higher revenues. At the same time, youll
learn what data you should be tracking, so you can refine
your metrics and monitor them over the long haul.
Take a look at three different scenarios where custom
analytics work to support business goals. By creating
specialized metrics that explain user behaviors, each of
these companies was able to quantify a negative customer
experience, gain valuable insights, and take action to solve
the problem.

Metrics That Matter

Act &
Refine

Improve user experience by tracking when its bad

Messages

Jason

Contact

Wednesday 10:17 AM

Lol sorry I messed up. I

paid you for the AC
instead of charging you
so Im gonna charge
you double now...

Todays online businesses now have huge quantities of data at their

fingertips, allowing product teams to adapt their designs to actual user
behaviors. When Venmo, which produces a peer-to-peer payment app,
realized that the vast majority of their users were sending money to
friends (rather than charging them), the product team redesigned the
interface to make it faster and easier to send money. The result? More
people sent more money, boosting engagement metricsjust as the
product team had hoped.

Lollll ok
Now it says you paid me
double...
OH GODDAMMIT
Yes I just paid you 3
times the amount for
the AC
Okay i can deal with this
iMessage

Send

But, oops, the redesign caused a problem for the small percentage of
people who were trying to request payments from their friends. By
making it easier to send money, the new interface had made it more
confusing to request payment, which resulted in user errors, double
payments, offsetting reversals, and all-around social awkwardness.
How big was the problem? To quantify it, the Venmo product team
developed a custom metric called pay-charge mixup, which measured
the number of users who made a payment and then immediately
requested double payment.

After the first redesign, many users sent money

to friends when they intended to charge them.
PAYMENT CHARGE SWITCH Percent Recipients Are Confi

% of iOS users that make a pay/charge mix up - app version

10

Pay-charge mixups after

the redeigned interface

Thu 15

Sat 17 Mon 19 Wed 21 Fri 23

Mon 25 Tue 27

Thu 29

Sat 3

Tue 03 Thu 05

pre-V6

Sat 07 Mon 09 Wed 11 Fri 13

Jun 15

Tue 17

Thu 19 Sat 21

Mon 23 Wed 25 Fri 27 Jun 29

v6.0-v6.1

Pay-charge mixups after

the redeigned interface

PAYMENT CHARGE SWITCH Percent Recipients Are Confi

With the new version they saw that the instance of this type of error
was unacceptably high. This information gave the team the support they
needed to launch another redesign, version 6.2, which fixed the user
experience.
% of iOS users that make a pay/charge mix up - app version
10

8
6
4

2
0
Mon 14

Tue 15 Wed 15 Thu 17

Fri 18

Sat 19

Jul 21

Mon 21 Tue 22

Wed 23 Thu 24

pre-V6

Fri 25

Sat 26

v6.0-v6.1

Jul 27

Mon 21 Tue 29

Wed 30

Thu 31 August

Sat 03

Aug 03

Mon 04 Tue 05

Wed 06 Thu

v6.2+

With the fixed version, this bad experience was now back to manageable
levels. Going forward, the Venmo product team has a useful metric to
watch as they continue to iterate and improve their app.

Metrics The Matter

Having quantified a bad

user experience, we can set
a goal of driving these down
to zero.

Track user happiness to focus the team

User ratings on app stores represent a crucial metric that can make or
break many startups. But think about apps that deliver local services
with local supply: innovators like Uber, Lyft, HotelTonight, Instacart, and
Shyp. They can often get an unfair rap: If people download the app in a
geography where the service doesnt exist, the user has an overall bad
experienceand may feel free to share it on the store site.

*****

Not worth it l downloaded this app several

times hoping that it would be changed...still as it
is, limited to cretin famous cities only. craaap.

*****

Worthless Only a few cities...not worth my time.

Good ratings on app stores are the lifeblood
of continuing success. And reviews like these
(even ones with bad spelling) can pull down an
excellent product or service thats not available
in the users area.

How can a business start small, refine both their product and their
operations, and then use data to expand to the right geographies in
stages? The solution lies in creating a custom metric that quantifies this
bad user experience, starting with: How many users open the app for
the first time more than N miles from a service area? N equals whatever
distance constitutes a poor experience. Then, having quantified a bad
user experience, we can set a goal of driving these down to zero.

100
90
80
70
60
50
40
30
20
10

ril

ay
M

Ap

ry
ua

ar
ch

br
Fe

15
20

be

be

em

em

D
Ec

er

ov
N

be

ob
ct

Se

pt
e

ly

st

Ju

gu
Au

ay

ne

Ju

Ap
ril

ar
ch

14

ar

20

br
u
Fe

be
r

be
em

D
ec
em

er

N
ov

be

ob

em

O
ct

Se
pt

ly

st
gu
Au

Ju

As

ia

Location Data Date

Asia

Europe

South America

United States

In the example above, a service provider has quantified a poor experience

as 40 miles from a service area (N= 40). The graph shows coverage
across several geos: from South America (purple), where coverage is still
lacking, down to the United States (green), where coverage continues to
improve over time.

Metrics The Matter

Define
What kinds of behaviors do we want
to measure?

To understand more local effects, and what caused a particularly good

improvement in the United States, its easy to drill down into one
particular geo of the United States.

Quantify
How can we use data to understand
behaviors?

100
90
80

Act
How can we create value from the
data and put insights to work in
our business?

70
60
50
40
30
20
10

M
ay

Ap
ril

20
15
Fe
br
ua
ry
M
ar
ch

Ju
ly
Au
gu
st
Se
pt
em
be
r
O
ct
ob
er
N
ov
em
be
r
D
Ec
em
be
r

M
ay

Ju
ne

Ap
ril

20
14
Fe
br
ua
ry
M
ar
ch

O
ct
ob
er
N
ov
em
be
r
D
ec
em
be
r

Ju
ly

As
ia

Moving forward, how can we track the

right data and drive more valuable
insights?

Au
gu
st
Se
pt
em
be
r

Refine

100

Location Data Date

90

Asia

Europe

South America

80

United States

70
60
50
40
30
20
10

ril

ay
M

arc

Ap

15

ary

be

20

bru
Fe

em
Ec

be

be

em

cto

ov
N

ly

st

be
O

pte

ne

Ju

gu
Se

Ju

Au

ril

ay
M

Ap

14

ary

arc
M

bru
Fe

20

be

be

be

em

em

cto
O

ec

ov
N

ly

st

be

gu

Au

Se

pte

Ju

As
ia

Notice the massive improvement in user experience in Puerto Rico when

the product was introduced, which translated into a visible improvement
in the U.S. overall. By tracking this custom metric, the business can
continue to improve the user interface by focusing on reducing this bad
user experience.

Dont let top line metrics mask the

detailed experience
Gaming companies that produce game apps like Candy Crush, FarmVille,
and Angry Birds love to track Daily Active Users (DAU) and Monthly
Active Users (MAU), and data collectors regularly put out rankings of top
apps based on these metrics.
But DAU and MAU can easily mask more important metrics related
to retention and growth. To ensure attention is paid to ongoing user
engagement, producers of a mobile gaming app, much like Candy Crush,
developed a quick cohort analysis and turned it into a custom metric that
they could track for each level of the game.

Metrics The Matter

Top 25 Facebook
Games by MAU
Rank

Game

MAU

Candy Crush Saga

43,469,305

Farmville 2

38,437,500

Texas Holdem Poker 30,218,750

Pet Rescue Saga4

24,717,139

Dragon City

23,343,750

Diamond Crush

22,731,250

Bubble Witch Saga

18,818,750

Coasterville

17,562,500

Bubble Safari

16,343,750

10

Criminal Case

15,862,500

11

Words With Friends

14,262,500

12

Chefville

12,762,500

13

Pool Live Tour

12,050,000

14

Farm Heroes Saga

12,050,000

15

8 Ball Pool

11,837,500

16

Top Eleven

10,934,750

17

Farmville

10,725,000

18

Village Life

10,087,500

19

Angry Birds Friends

9,781,250

20

Zynga Slingo

9,712,500

21

Bejeweled Blitz

9,350,000

22

SongPop

8,950,000

23

Bubble Safari Ocean 8,593,750

24

Tetris Battle

8,287,500

25

Bubble Island

8,156,250

Metrics The Matter

This level-completion custom metric tells them how many users who start
a given level complete that level. In the example below, they saw that
early, less-engaged users were struggling to get past Level 1, but loyalty
quickly jumped and increased steadily over time, with some mild variance.
More importantly, though, they also noticed that there were three levels
that seemed to be driving users away.

Fortunately, its easy for a mobile gaming company to quickly iterate and
update their app. By completely retuning these three levels, the team was
able to eliminate the roadblocks that users were experiencingand even
to encourage churned players to return.

Custom metrics show the devil in the details

Everyone loves dashboards. And many tech companies find it motivating
to have screens on the walls of the office that show how the company is
growing. But all too often, these at-a-glance metrics are too simplistic to
give a true view of how a product or a business is performing. By adding
custom analytics to your metrics mix, whether its quantifying a bad
experience or tracking retention at a more granular level, you can truly
illuminate your business challengesand gain the information you need
to address them.
As you evaluate your current analytics solution or begin to look for a
new analytics tool, make sure you have the flexibility to create reusable
custom metrics so you can employ these tactics. Your analytics solution
should absolutely be tailored to the unique aspects of your business so
that you can track and measure the specific custom information that is
critical to your companys success.

Ready to Love Your Analytics?

Come see a demo and
schedule your free trial.
Call 888-960-2331 or go
to:
looker.com/demo

About Looker
Looker is an inventive software company thats pioneering the next generation of
business intelligence (BI). We believe in bringing better insights and data-driven
decision-making to businesses of all sizes. The company has fast become the catalyst
that is creating data-driven cultures at hundreds of industry-leading companies such
as Yahoo!, Gilt, Warby Parker and Sony.

Looker is based in Santa Cruz, CA

Follow @lookerdata or visit www.looker.com

2015 Looker. All rights reserved. Looker and the Looker logo are trademarks of Looker Data Sciences, registered in the United States. Other trademarks are trademarks of their
respective companies. All services are subject to change or discontinuance without notice.

Metrics The Matter