Real-time Streaming Analytics

Business Value, Use Cases and Architectural Considerations

Big Data Solutions and Services Partner for Enterprises

Anand Venugopal Sr. Director Business Development Big Data Solutions

Yue Cathy Chang Sr. Director Business Development Alliances and Partnerships

Picture your house

What if this was happening now to your home ?

Recorded version available at http://bit.ly/1i6OrwR

When do you want to know ?

Later

Now ?
or

Recorded version available at http://bit.ly/1i6OrwR

Your best buddy from school

You havent met in years

5
Recorded version available at http://bit.ly/1i6OrwR

Is in Vegas same time as you

Your buddy

You

Recorded version available at http://bit.ly/1i6OrwR

When do you want to know ?

After you return or

7

NOW ?

Recorded version available at http://bit.ly/1i6OrwR

Life is happening NOW

Whether individual or Business

Important things are always happening NOW NOW is the ONLY time life REALLY happens Maximize data value process and act
sooner!

Real-time insight preserves or creates value

8

Topics we will cover today

What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?

Use cases

How?
Architectural considerations

What next ?
Recommendations
9

Recorded version available at http://bit.ly/1i6OrwR

What is Real-time Streaming Analytics ?

What it is NOT
Quick-response interactive analytics on static data
Batch processing Could be close but still NOT Micro Batch processing

10

Recorded version available at http://bit.ly/1i6OrwR

What is Real-time Streaming Analytics ?

Data analyzed in motion as it arrives

Routine: Monitoring, Counting, Alerting, Reporting

Complex Decision Making with Predictive analytics Every incoming event is distinctly processible Receive, Inspect, Analyse, Store, Distribute Events may be stored later or in parallel Immediate actions possible after processing

11

Recorded version available at http://bit.ly/1i6OrwR

Real time vs. Batch analytics

BATCH
SENSE Days ANALYSE Week s ACT

Sec/ ms

Sec/ ms

Real time

SENSE

ANALYSE

ACT

12

Recorded version available at http://bit.ly/1i6OrwR

13

Topics we will cover today

What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?

Use cases

How?
Architectural considerations

What next ?
Recommendations
14

Recorded version available at http://bit.ly/1i6OrwR

Business Value
Diminishes with the age of data

$$$ ? Befor e

Predictive analytics based on current events Value depends on accuracy Real-time Certainty is high REAL Value based on quick response

$$ NOW

$ $ $ Later
Value of Data
Age of Data

Descriptiv e Diagnostic Least value

The drop is non-linear

15

Business Value from RTSA

Routine business operations (Real time systems)

Cutting preventable losses

Finding and monetizing missed opportunities
More revenue

Cost savings

Creating new opportunities

New Business models (Products, Services, Revenue)

16

Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today

What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?

Use cases

How?
Architectural considerations

What next ?
Recommendations
17

Recorded version available at http://bit.ly/1i6OrwR

Routine Operations
(RT systems)
Manufacturing Control Systems (Closed loop)

IT - Systems & Network Monitoring

Field Assets Monitoring and Alerting Trucks, Oil rigs, Vending machines, Radio towers

Financial Transactions Processing

Authentications, Validations, Fraud

18

Recorded version available at http://bit.ly/1i6OrwR

Cutting Preventable Losses

Many headline stories are failures in routine operations and were preventable losses

MH 370 Loss of lives and assets GM Manufacturing defects Target Major Security breach Stock Exchange Meltdown

19

Recorded version available at http://bit.ly/1i6OrwR

Cutting Preventable Losses (2)

Medical / Clinical Complex analytics in ICU

Disaster Warning Systems: Chile / Sandy

Brokerage - Fraudulent or Risky Trades Preventive Maintenance Machines, Plants

Customer Churn
Brand Reputation on Social Media

20

Recorded version available at http://bit.ly/1i6OrwR

Missed Opportunities - Revenue

Customer Service always happens in real-time

Listening and Learning from customers (Social)

Context sensitive inventory Products, Ads Recommend - Upsell Cross-sell

SENSE Arrival

ANALYSE Context

ACT Serve

21

Recorded version available at http://bit.ly/1i6OrwR

Missed Opportunities - Efficiency

Operational Efficiency of systems or processes Network Optimization for cost, quality of service Dynamic capacity management Dynamic re-routing of traffic, cargo Insurance Adjudication Drone image analysis
SENSE ANALYSE ACT

Operation

Efficiency

Change

22

Recorded version available at http://bit.ly/1i6OrwR

New Opportunities
Tractors are becoming soil sensors Information service to farmers

Nike becoming a healthcare company ??

Quantified self movement Telecom giants selling data and insights

SENSE What else ?

ANALYSE What else ?

ACT What else ?

23

Recorded version available at http://bit.ly/1i6OrwR

Business Value of RTSA Summary

24

Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today

What ?
Is Real-time Streaming Analytics and what it is not

Why ?
Business Value

Who and Where ?

Use cases

How ?
Architectural considerations

What next ?
Recommendations
25

Recorded version available at http://bit.ly/1i6OrwR

Architectural Considerations
(1/3)

SENSE SENSE

ACT ACT

ANALYSE ANALYSE

APPROPRIATELY

26

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics

Batch Processin g

Stream Processi ng Real-time Analytics

27

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics

Stream Processing

Reliable ingest of multiple hi-volume hi-speed streams Route Data to Indexing and/or Storing DBs Distribution of data and analytics results Dynamic rule based routing

28

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics

Hi-speed ingest of multiple hivolume streams Route to Indexing and Storing Distribution of data and analytics results over ESB Dynamic rule based routing

Stream Processin g

Real-time Analytics

Computations Rules-based Monitoring & Alerts Predictive Models live scoring of real time data with run-time model refresh

29

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics

Receive, store and active archive of data-feeds Offline analytics over ALL data Visual interactive BI/ Dashboards Batch Offline building and updating of Predictive Processin Models for real-time scoring

Hi-speed ingest of multiple hi-volume streams Route to Index and DB Distribution of data and analytics results over ESB Dynamic rule based routing

Stream Processin g Real-time Analytics

Computations Rules-based Monitoring & Alerts Predictive Models live scoring of real time data with run-time model refresh

30

Recorded version available at http://bit.ly/1i6OrwR

Real time + Batch Analytics

Receive, store and active archive of data-feeds Offline analytics over ALL data Visual interactive BI/ Dashboards Batch Offline building and updating of Processing Predictive Models for real-time scoring

Stream Processin g
Hi-speed ingest of multiple hivolume streams Route to Index and DB Distribution of data and analytics results over ESB Dynamic rule based routing

Real-time Analytics
Computations Rules-based Monitoring & Alerts Predictive Models live scoring of real time data with run-time model refresh

31

Recorded version available at http://bit.ly/1i6OrwR

Real-time streaming analytics pipeline and flow

32

Recorded version available at http://bit.ly/1i6OrwR

Real-time streaming analytics pipeline and flow

Scale and Robustness Reliability Guarantees Publish-Subscribe Flexibility Dynamic Integration with Batch Loose Coupling Visualization Ease of Administration
33

Recorded version available at http://bit.ly/1i6OrwR

StreamAnalytix

34

Recorded version available at http://bit.ly/1i6OrwR

Approaches to Stream Analytics

Proprietary platforms Vendor lock-in No leverage of open source movement Do it yourself

Open source stitch up

Integration and maintenance nightmare Significant delays in time-to-market

35

Recorded version available at http://bit.ly/1i6OrwR

Introducing StreamAnalytix
An App Server for real-time apps Based on best-of-breed Open source Focus on your Business logic leave infrastructure to the
platform

Handle all the 3Vs of Big Data in one platform Seamless integration with Hadoop, NoSQL or any other DB Rapidly operationalize pre-built analytical models or new ones Significant time to market acceleration

Impetus provides full product support and professional

36

services

Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today

What ?
Is Real time streaming anaytics and what it is not

Why ?
Business Value

Who and Where ?

Use cases

How?
Architectural considerations

What next ?
Recommendations
37

Recorded version available at http://bit.ly/1i6OrwR

RECAP
Important things are always happening NOW

Maximize data value process and act sooner!

There is value find it Improve Ops, Cut losses, Find
missed & new opportunities

Architecture: Sense Analyse Act Sense

Real time insight preserves and creates business value

38 38

RECOMMENDATIONS
Get Real time streaming analytics in your roadmap

Talk to experienced peers and consultants

Start now with opportunities search, solution architecture and
vendor conversations

Instrument (SENSE) everything find gaps and fill

Prove value with faster batch with current infra is possible Establish mechanisms to ACT on the insights

Close the loop Sense and Analyse effectiveness

DO IT
39
Recorded version available at http://bit.ly/1i6OrwR

Topics we will cover today

What ?
Is Real time streaming anaytics and what it is not

Why ?
Business Value

Who and Where ?

Use cases

How?
Architectural considerations

What next ?
Recommendations
40

Request a demo of StreamAnalytix

bigdata@impetus.com

Big Data Solutions and Services partner for Enterprises

@impetustec h

41

Recorded version available at http://bit.ly/1i6OrwR