2014 Las Session 11

Session 11 PD, Predictive Modeling Session 1 - A World's Eye View of Predictive Modeling
and Data Science

Moderator:
Brian D. Holland, FSA, MAAA
Presenters:
James Dillard
Andrew Gardner, Ph.D.
Nikolaos Vasiloglou, Ph.D.
2014 Life & Annuity Symposium

May 19, 2014
DRAFT
This information is confidential and was prepared by Bain & Company solely for the use of our client; it is not to be relied on by any 3rd party without Bain's prior written consent
Call for volunteers
Describe an analytics project you worked on that was

successful in producing results what happened and
why was it successful?
OR
Describe an analytics project you worked on that
wasnt successful in producing results what
happened and why wasnt it successful?
ATL
20140519 Life and Annuity Sypmosium
Topics for today
What elements turn analytics into business results?

How are companies using analytics to create a strategic
advantage?
ATL
Briefly: What we mean when we say Big Data

Commonly referred
to as BigData
Basic
analytics
Very large
data volume
A lot of data
(>terabytes), e.g.
- Transaction data
- Personalization
data
- Sensor data
- Machine to
machine logs
- Many years of
data sets
Traditional
analytics
Analytical
complexity
Large number of
variables and steps
required to
parameterize a
question/problem
Advanced statistical
methods and tools
Lack of right/
wrong, fuzzy logic
Advanced
analytics
Complex
data sets
BigData
Lots of unstructured
data: hard to sort,
evaluate or
compare (e.g.
language, sound
recordings, images,
video from social
media)
High velocity
of analytics
Need for real-time

results, (e.g. TSA
waiting line,
instances of
personalized
advertisement)
Incomplete data,
low data quality or
consistency
ATL
Companies with the best analytic capabilities are

outperforming their competition
Bain Primary Research
~2X more likely to be in the top quartile of
financial performance within their industry
~3X more likely to execute decisions as intended
~5X more likely to make decisions much faster
Source: Bain Survey, N=409

ATL
These results come from analytical talent, strategic

intent, data and tools
Resolving to be datadriven; creating
structures, processes
and incentives to
support analytical
decision making
Mix of data science

and business acumen,
technical expertise
-36% of companies have a
dedicated data insights
team
Talent
Quality, consistent
data stored in a
manner that is easy
to access
-Only 19% have high

quality, consistent
data in their
organization
Data
-23% have a clear

strategy for embedding
analytics effectively
Intent
Tools
State of the art tools

like Hadoop, NoSQL,
HPCC or automated
data scrubbing
algorithms
-38% are using state of
the art analytics tools
Only 4% of companies meet all of

these criteria; for these Top
Performers analytics pay off

ATL
Data/Tools: Analytics technology is widely

available; still, many have yet to master the basics
Why isnt your companys analytics program more effective?
Capture Data
Store and Access
Analytical tools
Lack of systems
to capture data
56% of companies
Data arent stored Lack capabilities

in a useful form:
to manage
66% of companies
databases/ other
data tech: 49% of
companies
Useful data arent

collected: 56% of
companies
Lack of
technology to
store and access
data: 47% of
companies
Lack capabilities
to develop data
driven insights:
56% of companies

ATL
Intent: Top performers embed analytics within

structures, processes and incentives
TOP PERFORMERS EMBED DATA
INTO DECISION MAKING
AND CONNECT INSIGHTS TO

TANGIBLE ACTIONS
You have a scalable

mechanism to work through all
that big data and inform the
clinician who is delivering
care, at any point in time,
when and where it is most
impactful. That's really where
the secret sauce lies.
- Brian LeClaire, Chief Services
and Information Officer,
Humana

ATL
Talent: The best data teams are mixing analytical,

technical and business talent
CRITICAL BIG DATA TALENT
LEGAL
BUSINESS
ENGINEERING
DATA
Critical role
Data
scientists
COMPOSITION OF ANALYTICS TEAM
Role description
Generate ideas, insights, and
questions to be tested
Synthesize conclusions using
business, statistics, and
programming expertise
DATA
(20-25, ~30%)
Set up initial architecture

BDA
engineers
Handle HW & SW platforms to

support complex analytics (e.g., for
cluster computing)
Design of internet-scale
infrastructure and operations
ENGINEERING
(30-35, ~45%)
Identify cross-functional BigData

opportunities
Business Drive prioritization of BigData
managers intensive initiatives
Legal
Coordinate access to BigData

resources (e.g., data and personnel)
BUSINESS
Advise on standards for data

privacy & security
LEGAL
(10-12, ~15%)
(5-10, ~10%)
Advise business leaders on projectspecific adaptations

ATL
Agenda
What elements turn analytics into business results?

How are companies using analytics to create a strategic
advantage?
ATL
Each industry has a different center of gravity for

analytical sophistication
Key
Typical
Range
Basic
Traditional
Advanced
Big Data
Healthcare
Payers
Humana/
Anvita
Emory
Providers
Pharma
Retail
Brick & Mortar
Home Depot
Wal-Mart
E-Commerce
Amazon
Financial services
USAA/
CapitalOne
Retail Banking
Retail Insurance
Personal Fin. Software
Progressive/
GEICO/USAA
Mint
TMT
Apple/
Google
Tech
Newspapers
TV
Weather Co.
Netflix
OTT content providers

Telecom
Energy/Industrials
Generation/Distribution
Retail
Source: Expert Interviews, Local Companies
Southern Co.
ATL
10
Within an industry, analytics can be both an offensive

EXAMPLE
and defensive capability
High
Impact
Allstate is
seeking to
improve its
analytic
capabilities by
acquiring
Esurance
Low
Basic
analytics
Low
Traditional
analytics
Advanced
analytics
Big Data
High
Data capabilities
Source: Expert Interviews

ATL
11
Four principal types of Big Data opportunities exist

1
Improve
internal
processes
Uses social media

listening and
complex algorithms
to identify brand
influencers on
important topics
Uses analytics to
find unexpected
product
relationships that
improve sales (e.g.,
beer next to diapers)
Improve
existing
products or
services
Maximizes
engagement by
selecting the optimal
time to send
messages on behalf
of its customers
Uses electronic park

passes to track
visitor movement
in its amusement
park and optimize
the park
experience
Build new
offerings
Transform
business model
Integrates realtime weather data

with internal client
data for retail,
insurance and CPG
clients
Detects phone
fraud in near realtime using audioanalysis and a
proprietary
fraudster database
Uses plug-in
devices to track
driver behavior and
proactively select
its customer base
Uses predictive
analytics to direct
preventative care,
making it profitable
to serve sick
patients
ATL
12
These opportunities are differentially difficult and

valuable
Hi
Accelerate
Wave 1
1
Ease of Execution
Improve internal processes

2
Build new products/

service offerings
Highest potential
opportunities to be
turbo-charged by
center
Improve
existing
products/
services
Transform/amend
business model
Low
Low
Opportunity Value
Hi
ATL
13
We use a proven approach of mapping Types and

Sources of Data to identify data assets and use cases
DATA CAN BE CATEGORIZED
BY TYPE AND SOURCE
Identity and
relationships
Context (incl. calendar
& location)
USER
Digital activity
Real world activity
Communications &
social presence
Content
Financial data
SOURCES OF DATA
PRODUCTS:
Hardware & Devices
TYPES OF DATA
Health data
Device interactions
Product status data
Manufacturing
Logistics
Sales
Marketing
Finance
TV
Computer
Tablet
Mobile
Photo camera
Music player
Home appliance
Operating System
SERVICES:
Software & Systems
ENTERPRISE
MACHINE
Institutional data
EACH COMPETITOR HAS A UNIQUE

DATA ASSET BASED ON DATA ACCESS
Search
Browsers & App Store
Maps
Voice
Mail, Calendar,
Contacts
Media
Storage
ATL
14
Example: Google has strength in data related to

search, mapping, and digital activity
Potential access to data

Access to data
Sources of differentiation
Sources of data
Products: Hardware & Device
PC/
Computer
laptop
Types of data
User
TV
Photo
Camera
cameras
Smart
M
obile
phone
Tablet
M
usic
Music
player
players
Services: Software & Systems

Home
appliance
Operating
system
Browsers/
Content
SVC/
SW
App store
Search
M aps
Storage
Declared identity
Senso r based identity
Demo graphic data
Interests
P erso nal devices
Relatio nships
Lo catio n
P eo ple
Objects
Events
B ro wser activity
Client app. & OS activity
So cial activity
Context
awareness
improves
2
2
2
search
2
2and advertising
2
2
Text
Speech
So cial M edia
P resence
P hysical go o ds interactio n
Co nsumed media
Digital reco rds o f phys. Go o ds
Virtual go o ds
Health care data
Financial data
Health insurance data

2
Health senso r data

Go vernment data
A cademic data
1
1
Emplo yer data
Machine
M edia
P rivate do cuments
Mail and calendar allow

Google to mine interests,
relationships, events for
advertising purposes
M o bile and TV interactio ns
Ho me eco system interactio ns
Device status
Device perfo rmance
Failure causes
M anufacturing
Lo gistics
Sales
M arketing
Finance
So ftware perfo rmance data
Enterprise
M
ail,
Mail/
calendar,
contact/
calendar
contacts
Voice
Browser,
OS, and
App
store
activity
2
2
2
2
2
2
allows
Google
2
2
2 to improve
2
2 search,
2
2
2
2
2
2
2
optimize advertising, create new
2
2
2
2
2
2
services
(trends,
etc.)
2
2
2
2
2
2
2
ATL
15
Recent acquisitions have helped fill in Googles data

map in areas where they lack strength

Access to data
Sources of data
PC/
Computer
laptop
User
TV
M
usic
Music
player
players
Home
appliance
Operating
system
M
ail,
Mail/
calendar,
contact/
calendar
contacts
Voice
M edia
Storage
Declared identity
Demo graphic data
Interests
P erso nal devices
Relatio nships
Lo catio n
P eo ple
Objects
Events
B ro wser activity
So cial activity
Text
Speech
P rivate do cuments
Co nsumed media
2
2
Mobile context
2
2
awareness and
2
2
recommendation
2
2
engine
2
1 Local2 point of1

interest
and 3user
3
2
2
2
visiting
history,
2
2
2
review,
rating
data
3
1
3
Virtual go o ds
Health care data
Financial data

Health senso r data
Go vernment data
Device hardware
and enterprise data
2
1
1
A cademic data
Emplo yer data
Machine
M aps
P resence
Device status
Device perfo rmance
Failure causes
M anufacturing
Lo gistics
Sales
M arketing
Finance
Enterprise
Browsers/
Content
SVC/
SW
App store
Search
So cial M edia
Types of data
Photo
Camera
cameras
Smart
M
obile
phone
Tablet
Home
appliance,
2
sensor based
identity and
presence data
ATL
16
Example: We applied this framework to Tech Co to

DISGUISED
identify strategic opportunities
Data sources
Shoppers
Consumers
Parents
Socializers
Seniors
Businesses
Gov/Law Enf
Kid-friendly
social media
Mobile
monitoring for
health/safety
Kid-friendly
social media
In-pharmacy
push reminders
at POS
Transport
Education
Strategy
Marketing
Mobility BU
TV BU
Media BU
IT BU
Plagiarism
detection/
prevention
Invest in content
that align with
customer
preferences
Adapt/update
segmentation in
real-time
Auto-alerts of
positive/negative
reviews
Closed-loop
feedback on device
performance
Understand
customer viewing
habits
Shopping
Digital
content
Emergency
notifications
Track
bottlenecks to
improve routes
Push ads based

on scan of
photos/videos
Geo/map
Market
alerts
Local health
warnings
Marketers
Calendar-linked
special offers
Financial
Event
location/
landmark
mapping
Automated
news sharing
Enable
unmanned
vehicles
Reviews
News/ent
Travel
Price alerts on
holiday gifts
Healthcare
Fin Svcs
Client
Local deals
Travelers
Professionals
Data users
Updates when
kids/seniors
leave/arrive home
Children
Students
Partner/supplier generated
Demographics
Device
problems
Home
control
Energy
related
Travel
related
Retail/
Inventory
Advertising
/ media
Service/
mtce
Machine generated
Rich media
Browsing
Purchasing
Search
history
Calendar
Time
Transform business
Location
New offerings
Social
User generated
Improve products
Weather
Internal processes
Determine customer
sources and
geographic radius
Identify
manufacturing
process defects
ATL
17
Big Data observations

1
Data and technology are only part of the story;

combining these with talent and strategic intent produces
the most powerful results
Analytics must be actionable to be valuable; anchoring

the data in critical processes is as important as getting the
right answer
A mix of analytical, technical and business skillsets

best equips a team to deliver actionable solutions
Analytical sophistication varies by industry; your

companys analytics strategy must include the industry and
strategic context
Follow a hypothesis-driven approach test ideas,

manage expectations and sequence investments as you
learn more
ATL
18
Thank you for joining us!
ATL
19
Each competitor has built a unique strategy around

their core data strengths
OPPORTUNITIES
SOURCES OF DATA
Products
SOURCES OF DATA
Services
Products
i1
SOURCES OF DATA
Services
Products
G1
User
Mach
Ent.
Ent.
Ent.
TYPES OF DATA
User
G2
Mach
TYPES OF DATA
User
i2
Mach
TYPES OF DATA
i2
Services
a1
a2
3
3
3
Differentiated data asset:

i1
i2
Apple differentiates through

applications, media and
services in a closed
ecosystem with increasing
customer stickiness
Strength in seamless crossdevice integration creates
positive feedback loops
G1
G2
Powerful context
awareness engine
improves search and
advertising
Mail and calendar
capabilities enable
Google to mine interests,
relationships, and events
for advertising purposes
a1
Data collected from all

aspects of business drives
recommendation engine
a2
Deep expertise in
logistics has become an
important source of
customer-facing
differentiation
ATL
20
Apple differentiates through its interconnected

mobile products and software capabilities

Access to data
Sources of data
PC/
Computer
laptop
TV
Home
appliance
Operating
system
Search
Browsers/
Content
SVC/
SW
App store
M aps
Declared identity
Demo graphic data
Interests
P erso nal devices
Relatio nships
Lo catio n
P eo ple
Objects
Events
2
1
2
Apple
differentiates
its industryApple
differentiates
with
2
2
2
2
leading products with seamless
1
1
1
seamlessintegration
verticalofintegration
services 2
2
User
M
usic
Music
player
players
B ro wser activity
Types of data
Photo
Camera
cameras
Smart
M
obile
phone
Tablet

M
ail,
Mail/
calendar,
contact/
calendar
contacts
Voice
M edia
its
across
devices, content and services
2
2
Storage
1
1
So cial activity
Speech
So cial M edia
P resence
Text
P rivate do cuments
Co nsumed media
Financial data
Virtual go o ds
Health care data
Health senso r data

Go vernment data
A cademic data
Emplo yer data
Machine
Strength in media driven by

iTunes media store, media players
(iPod, iPhone) and cross-device
integration through iCloud
Device status
Device perfo rmance
Failure causes
2
2
M anufacturing
Lo gistics
Enterprise
Sales
M arketing
Finance
ATL
21
Amazon uses data to create a powerful

recommendation engine

Access to data
Sources of data
PC/
Computer
laptop
Enterprise
Machine
Types of data
User
TV
Photo
Camera
cameras
MSmart
obile
phone
Tablet
M
usic
Music
player
players

Home
appliance
Operating
system
Browsers/
Content
SVC/
SW
App store
Search
M aps
M ail,
Mail/
calendar,
contact/
calendar
contacts
Voice
M edia
Storage
Declared identity
2
1
Demo graphic data
Interests
P erso nal devices
Relatio nships
Lo catio n
P eo ple
Objects
Events
B ro wser activity
So cial activity
Text
Speech
So cial M edia
0
Data
collected
Data
collected for
0
from
all aspects of
recommendation
0
business
to drive
0
from
all aspects of
recommendation
0
business
engine
0
P resence
P rivate do cuments
Co nsumed media
Financial data
Virtual go o ds
Health care data
Health senso r data
Go vernment data
A cademic data
Emplo yer data
Device status
Device perfo rmance
Failure causes
M anufacturing
Lo gistics
Deep expertise in
0
0
logistics
has become
an
0
0
0
important
source
of 0
0
0
differentiation
0
0
0
0
Little0 manufacturing
0
data
0
0
0
3
1
Sales
M arketing
Finance
ATL
22
Companies with the best analytic capabilities are

outperforming their competition
1
~2X MORE LIKELY TO BE TOP QUARTILE

FINANCIAL PERFORMERS
~3X MORE LIKELY TO BE HIGHLY

EFFECTIVE AT EXECUTING DECISIONS
~5X MORE LIKELY TO

MAKE DECISIONS MUCH FASTER
~2X MORE LIKELY TO USE DATA TO MAKE

DECISIONS VERY FREQUENTLY
Source: Bain Survey; see appendix for details

ATL
23
409 respondents from a range of companies were

surveyed about their data capabilities
Big Data Diagnostic Survey, profile of survey
respondents
100%
Other
Other
80
$10Bn or more
Other Mfg.
60
40
$5Bn to $10Bn
$1Bn to $5Bn
20
Financial
Intermediation
Manager
North America
Head of
department
Head of
business unit
Tech, Telco
Services, Telco
Manufacturing
SVP/VP/
Director
Europe
<$1Bn
C-Level & Board

Asia-Pacific
Company size
(global revenues)
Industry
Respondent
geography
Respondent job title
Note: N=409
Source: Bain Big Data Diagnostic Survey
ATL
24
Top data performers were defined by isolating key

data characteristics
CRITERIA FOR ANALYTICS
CAPABILITIES
MOST RESPONDENTS HAD AT LEAST

1 DATA CAPABILITY
Intent:
- Respondents were asked to indicated their level of
agreement with the statement We have a clearly
defined strategy for how to use big data effectively
- Respondents that answered strongly agree were
considered to have this capability
Talent:
- Respondents were asked to assess the statement We
lack a dedicated data insight team as a major, minor
or not a reason that they were not more effective
- Respondents who marked not a reason were
considered as having this capability
Data:
- Respondents were asked to indicated their level of
agreement with the statement We have high quality,
consistent data within our organization
- Respondents that answered strongly agree were
considered to have this capability
Tools:
- Respondents were shown a list of analytics tools and
asked which were used within their organization
- Those that selected Advanced analytics tools (e.g.,
HPCC, Hadoop or NoSQL) and Automated cleaning
tools/algorithms were considered to have this capability
ATL
25
The most common capability was use of advanced

analytic tools; the least common was consistent data
Percentage of respondents with each capability

40%
38
36
30
20
23
Top
Performers
19
High
10
Medium
0
Low
Org strategy
Talent
Tools
Data

ATL
26
Top performers were ~3X more likely to have top

quartile financials than bottom feeders
Likelihood of being top quartile financial performers

60%
Top performers are ~3X more likely

than bottom feeders to have top
quartile financial performance
46
40
40
And ~2X more

likely than the
average company
36
22
20
Average = 25%
16
Bottom feeders
Low
Medium
High
Top perfomers

ATL
27
Top performers were ~27X more likely to make

decisions much faster than bottom feeders
Top performers are ~3.3X more likely
to make decisions faster than bottom
feeders; ~27X more likely to make
decisions much faster
And ~2X more likely

to make decisions faster
than average; ~6.5X
more likely to make
decisions much faster

ATL
28
Top performers are ~7X more likely than bottom feeders

to execute decisions effectively most of the time
Percentage of respondents that
execute decisions as intended
more than 75% of the time
50%

than bottom to execute decisions as
intended most of the time
47
38
40
And ~3X more

likely than the
average company
30
25
20
Average
10
10
Bottom feeders
Low
Medium
High
Top Performers

ATL
29
Top performers are ~7X more likely than bottom

feeders to use data to make decisions
Percentage of respondents that
use data to make business decisions
"very frequently"
60%

than bottom to execute decisions as
intended most of the time
53
46
40
And ~2X more

likely than the
average company
37
27
20
Average
19
Bottom feeders
Low
Medium
High
Top Performers
Note: Very frequently defined as 6 times are more in the past 12 months
ATL
30
Top performers are more effective at the tactical

aspects of analytics across every category
Top performers:
0%
Average

ATL
31
Top performers are more likely to employ analytics

talent in each position
Average
Data
Tech
Business
ATL
32
Talent: Insights are driven by a mix of data science,

tech expertise and business acumen
Critical roles =
Basic
Traditional
Advanced
Big Data
As issues
arise
As issues
arise
Regularly
Daily
Business roles
Business manager
Business analyst
IT roles
Database manager
Infrastructure manager
and architect
Programmer
Data roles
Data analyst
Statistician
Data scientist
Interaction across
functions
Increasing need for coordination

ATL
33
Talent: Top performers are deliberately investing in

analytic talent
BACKUP
Data scientists
Tech expertise
Generate hypotheses,
business insights, and
questions to test
Set up initial architecture for

capturing, storing and analyzing
data
Synthesize conclusions using

business, statistics, and
programming expertise
Handle HW & SW platforms to

support complex analytics
Design of internet-scale
infrastructure and operations
Source: Bain Survey, N=409; see appendix for details

Business acumen
Validate hypotheses are
actionable, relevant and
aligned with business
priorities
Drive prioritization of
initiatives
Coordinate access to COE
resources (including data
and personnel)
ATL
34
Analytics opportunities often are specific to value

chain elements of our clients' industries
Key
N/A
Low
Med.
Service
R&D
Manufact.
Logistics
Marketing
Sales
Also: Product
development
Also:
Production,
Generation
Also:
Distribution,
Network ops
Also: Pricing,
Customer
segmenting
Also: Front
Office, Client
management
High
Back office
Also: Customer
Also: Risk
experience,
management
repairs, claims
Industries
Healthcare
Payers
Providers
Pharma
Retail
Brick and Mortar

E-commerce
Financial services
Retail banking
Personal insurance
Commercial insurance
Personal fin. Software
TMT
Tech
Newspapers
TV
OTT providers
Telecom
Energy/Industrials
Electricity distribution
Electricity retail
Source: Expert Interviews
ATL
35
For technology companies, analytics can be both a

strategic capability and a product offering
Typical sellside use cases:

Resource
optimization
Automated
decisioning
Risk and fraud
management
Customer
segmentation
Make It Easy:
Provide Analytic Offerings
to Facilitate Creating
Impactful Insights for
customers
Typical internal
use cases:
Personalized
product and
service
offerings
Stuck
in the
middle?
Predictive
maintenance
Supply chain
optimization
Make It Strategic:
Drive Internal Value
from Data Through
Advanced Analytics
ATL
36
Detail behind Atlanta Companies

Emory
Hospitals
Home Depot
Weather
Company
Detailed
search of
leading
hospitals for
Big Data didnt
turn up any
mentions of
Emory
Several articles
mentioned
Home Depot
using Big Data
to improve
performance
without
growing stores
Search for
Emory and Big
Data turned up
a partnership
with IBM to
pilot predictive
medical care
through realtime analytics
Differentiated
pricing
capabilities
were credited
for much of
this
Independently
verified a
general
understanding
of the
conversation
we had with
their
leadership
team
Since this is
experimental, I
triangulated
Acquisition of
Black Locus in
2012 is the
source of
pricing
capabilities
While dynamic
pricing is
mentioned, its
likely not realtime
Southern
Company
MailChimp
Found some
evidence of
them using
predictive
analytics to
improve
customer
collection
Email
marketing
company
based in
Atlanta with a
web-based app
for small
businesses
Discussed this
with folks close
to the client
who said that
they are
relatively close
to the median
within the
industry, if not
lagging a little
Provides small
businesses
with several
data based
services
Triangulated a
bit generously
This services
analyzes the
most effective
time to send
emails based
on individual
user habits
and their
larger data set
ATL
Pindrop
Security
This is Rajs
company
Read through
their website
to understand
their products
Settled in on
their fraud
detection
product, which
uses that
audio finger
print of a
particular call,
combined with
the expected
footprint and
their
proprietary
fraudster
database to
determine
fraud likelihood
in ~15
seconds
37
From linear regression to matrix

factorizations
Nikolaos Vasiloglou
nvasil@ismion.com
The statisticians toolkit
Hypothesis testing (Likelihood Ratios)

Correlations (Pearsons)
Linear Regression
Bootstrap
Confidence Intervals
Joint/Conditional probability estimators
NonParametric Tests
The 100+ year old predictor

Regression
o
o
o
o
o
o
Linear
Logistic
Quantile
Poisson
Polynomial
Nonparametric
The classic one
y=aixi
Why do people love regression?
They spent hours studying it in college

It is simple
Easy to interpret
People think they can do causal analysis
A lot of free software
Nice and rigorous confidence intervals
From good low to bad high

dimensions
Data in the past
o
o
o
Structured
Clean
Well designed and actively collected
Data now
o
o
o
o
Unstructured
Dirty
Passively collected
Exchange agencies
Unstructured Data
Text
Clickstream
Buying history
Location
Images
Weblogs
UPS shipment traffic
etc
The power of unstructured data

If you are not convinced about them look at
o
o
o
o
Google
Netflix
Amazon
Online Ad companies
Generate billions of revenues
The high-dimensional regression

The formula remains the same
Unstructured variables become categorical
How do you solve it now?
Scaling issues
1. Storage
2. Management
a.
b.
Querying
Joining
3. Solver
a.
b.
Training
Eliminate redundant variables
Scaling Solutions
The distributed storage GFS/HDFS (early 2000)

The birth of map-reduce Hadoop (2005)
BigSQL systems (Google BigQuery) (2012)
Vopal Wabit (2008)
Vopal Wabit (http://hunch.net/~vw/)

Ordinary Least Square is O(Nk2) prohibitive
o
o
Online training is the solution

One sample at a time
StepWise Regression is prohibitive

o
L1 regularization is the solution
When dimensionality is huge ~1B

o
o
coordinate gradient descent

feature hashing
Interactions
Linear models are good but they might be
insufficient
For example user recommendations
Given historical user-movie ratings predict
new ratings
user_rating= ai *user + bi*movie
This approach doesnt work
The power of 2nd order effects
Similar Items
Back to regression
Wouldnt be nice if we could use the pairwise
interactions in high-dimensional regression?
y=aixi+bijxixj ????
b is k x k
For k=1M b is ~1B !!!!
k
k
The factorization machine (www.libfm.org)
y=aixi+ <bi,bj> xixj

k
d
b
A few ingredients of a good recipe in

the science of data
Massive Data Feed aggregation

High Dimensionality
Pair-wise data interactions
Approximate methods for training

2014 Las Session 11

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

2014 Las Session 11

Hochgeladen von

Copyright:

Verfügbare Formate

Session 11 PD, Predictive Modeling Session 1 - A World's Eye View of Predictive Modeling

and Data Science

2014 Life & Annuity Symposium

Call for volunteers

Describe an analytics project you worked on that was

20140519 Life and Annuity Sypmosium

Topics for today

What elements turn analytics into business results?

20140519 Life and Annuity Sypmosium

Briefly: What we mean when we say Big Data

Need for real-time

20140519 Life and Annuity Sypmosium

Companies with the best analytic capabilities are

Source: Bain Survey, N=409

20140519 Life and Annuity Sypmosium

These results come from analytical talent, strategic

Mix of data science

-Only 19% have high

-23% have a clear

State of the art tools

Only 4% of companies meet all of

Source: Bain Survey, N=409

20140519 Life and Annuity Sypmosium

Data/Tools: Analytics technology is widely

Store and Access

Data arent stored Lack capabilities

Useful data arent

Source: Bain Survey, N=409

20140519 Life and Annuity Sypmosium

Intent: Top performers embed analytics within

AND CONNECT INSIGHTS TO

You have a scalable

Source: Bain Survey, N=409

20140519 Life and Annuity Sypmosium

Talent: The best data teams are mixing analytical,

COMPOSITION OF ANALYTICS TEAM

Set up initial architecture

Handle HW & SW platforms to

Identify cross-functional BigData

Coordinate access to BigData

Advise on standards for data

Advise business leaders on projectspecific adaptations

Source: Bain Survey, N=409

20140519 Life and Annuity Sypmosium

What elements turn analytics into business results?

20140519 Life and Annuity Sypmosium

Each industry has a different center of gravity for

OTT content providers

20140519 Life and Annuity Sypmosium

Within an industry, analytics can be both an offensive

Source: Expert Interviews

20140519 Life and Annuity Sypmosium

Four principal types of Big Data opportunities exist

Uses social media

Uses electronic park

Integrates realtime weather data

20140519 Life and Annuity Sypmosium

These opportunities are differentially difficult and

Improve internal processes

Build new products/

20140519 Life and Annuity Sypmosium

We use a proven approach of mapping Types and

EACH COMPETITOR HAS A UNIQUE

20140519 Life and Annuity Sypmosium