Tuning Realtime Data Warehouses

Tuning Realtime Data Warehouses - A Guide from the Field
Peter Scott Principal Consultant

T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 18 February 13
Who Am I?
One of the first members of Rittman Mead Consulting Specializes in Data Warehouse Design and Performance Experience of several Realtime systems
Presentation Outline
Presentation Outline
Why Realtime? The Who and Where of Realtime Tuning Effective Realtime Tuning
Why Realtime?
Three main use cases

We are too busy reporting to load new data - 24x7 working day Large data volumes take too long to load What happened, I need to know, NOW! - Getting new data in as soon as possible
Why Realtime?
Why Realtime?
Things Happen
Things Appear in Reports
Why Realtime?
Things Happen
Magic
David Coppereld Model
Why Realtime?
Things Happen
Data Movement, Data Transformation
Real World Model
Presentation Aim
Things Happen
Presentation Aim
Things Happen
Making the arrow as short as possible

What is Realtime?
What is Realtime?
Data Movement Detect change Extract change Stage change
What is Realtime?
Data Movement Detect change Extract change Stage change Data Transformation Apply change Aggregate Publish
What is Realtime?
NOT all data in a Realtime Data Warehouse is Realtime
Who and Where of Performance

Design
Develop
Test
Operate
Performance ?

Design
Develop
Performance ?
Test
Operate

Design
Develop
Performance ?
Test
Operate
Develop
Design Test
Performance ?
Operate
Effective Realtime Tuning
Plz do the needful
Dont process too much data Noise

- Not all change is real change
Poor change detection algorithms

- Dont forget to use simple math to check that data volumes are on a par with query results
Plz do the needful
Late filtering of data Avoid unnecessary processing in realtime database


Realtime is not traditional Data Warehousing Office hours tune for query Out of hours tune for batch

Realtime is not traditional Data Warehousing Office hours tune for query Out of hours tune for batch We need to tune for both query and transactional data load
Quick Straw Poll
Quick Straw Poll

Do you run your data warehouse in Archive Log Mode?
Quick Straw Poll

Do you run your data warehouse in Archive Log Mode? For realtime need to run database in archive log mode traditional recovery restore DB and reapply batches will not work

Use DBA tools to monitor where queries stick AWR, ASH, SQL Monitor Look at query plans to determine whether the database is being wise poor plans mean that the database is making a wrong assumption about the data - Make sure we describe the data to the database with constraints and adequate statistics

Move data in sets Direct path, set operators (union, minus, intersect) Avoid update and delete activities If you must update then consider using indexes - watch out for row migration Use database features to reduce undo Global Temporary Tables Partition Exchange
Real World Examples

Database Change Data Capture for a e-retailer JMS message services for a high transaction environment
Database Change Data Capture for a e-retailer

Major on-line seller of entertainment media Computer Games, Video, Music, Digital Download Exadata (V1) data warehouse, Oracle 10g/11gR1 and legacy MSFT SQL Server sources Realtime need around sales processing from 11g source - Must use OWB as ETL platform - Must use Oracle Asynchronous CDC (but not GoldenGate)
Challenges to performance
Building complete records from partial information Use analytics and set based SQL - extra challenge to handle data becoming NULL! The web-based source systems often issued DB commits (SELECT FOR UPDATE) without data change (aka NOISE!) Need set based code (filtered MINUS)
JMS message services for a high transaction environment
Confidential customer Large volume of financial transactions Exadata (V2) data warehouse Realtime feeds through corporate JMS message bus (Tibco) Realtime need to exposure to risk immediately before events - ODI as ETL platform - JMS XML Messages
Challenges to performance
Building up complete records from multiple messages Use analytics and set based SQL Merging data into the reporting layer Avoid updates by using insert + partition exchange Need to multiple-thread some ETL steps for performance Use Global Temporary Tables and run threads in their own sessions ETL tool generates large amounts of REDO Move ETL meta data to its own database and run in nor archive log mode
Questions???
?
Presentation Takeaways
Performance starts in the design and Development Everyone has a part to play - not just the DBA Avoid doing the unnecessary Monitor, measure, evaluate, prove, never guess Exploit the power of the database

Tuning Realtime Data Warehouses

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Tuning Realtime Data Warehouses

Hochgeladen von

Copyright:

Verfügbare Formate

Tuning Realtime Data Warehouses - A Guide from the Field

Peter Scott Principal Consultant

Three main use cases

Things Appear in Reports

Things Appear in Reports

David Coppereld Model

Data Movement, Data Transformation

Things Appear in Reports

Real World Model

Data Movement, Data Transformation

Things Appear in Reports

Data Movement, Data Transformation

Things Appear in Reports

Making the arrow as short as possible

NOT all data in a Realtime Data Warehouse is Realtime

Who and Where of Performance

Who and Where of Performance

Who and Where of Performance

Who and Where of Performance

Effective Realtime Tuning

Plz do the needful

Effective Realtime Tuning

Dont process too much data Noise

Poor change detection algorithms

Plz do the needful

Late filtering of data Avoid unnecessary processing in realtime database

Effective Realtime Tuning

Effective Realtime Tuning

Effective Realtime Tuning

Quick Straw Poll

Quick Straw Poll

Quick Straw Poll

Effective Realtime Tuning

Effective Realtime Tuning

Real World Examples

Database Change Data Capture for a e-retailer

JMS message services for a high transaction environment

Das könnte Ihnen auch gefallen