ETIS08 - Process Neutral Data Modelling Concepts - Presentation

Data Management & Warehousing
PROCESS NEUTRAL DATA MODELLING

CONCEPTS
DAVID M WALKER
ETIS COMMUNITY GATHERING
13-14 NOVEMBER 2008 - BRUSSELS
© 2008 Data Management & Warehousing ETIS Community Gathering, Brussels Page 1
David M Walker 14 November 2008
Agenda
•  The Issues With Conventional Data Warehouse

Data Models
•  Assumptions About The Data Model To Be
Constructed
•  Requirements Of A Data Warehouse Data Model
•  Constructing The Data Warehouse Data Model
THE ISSUES WITH

CONVENTIONAL DATA WAREHOUSE
DATA MODELS
Issues
•  Data models take a long time to develop

•  Data models are expensive to change
–  Affects Source -> Data Warehouse ETL
–  Affects Data Warehouse -> Data Mart ETL
•  The design often reflects the first or largest
source system
–  This makes it difficult to add other systems
•  They often reflect current working practice
–  Making it difficult to change when the business does
Issues
•  A struggle to keep up with rapidly changing

source system data models
•  Reference data is often not stored in a time
variant way
•  History is lost with data model changes
•  Queries directly on the data warehouse are
complex
•  Different rules apply to query each table
•  Different database platforms have different
needs
ASSUMPTIONS ABOUT
THE DATA MODEL
TO BE CONSTRUCTED
Assumptions
•  Used in data warehouse

–  Not in the operational systems or the data marts
–  Different style of modelling required
•  Users not going to query the data model
–  Users will query separate dependent data marts
•  Data will be extracted from the model to
populate the data marts by ETL tools
•  Data will be loaded into the model from the
source systems by ETL tools
Assumptions
•  Direct updates will be prohibited

–  A separate application or applications will exist as a
surrogate source and ETL used to load the data
•  Not a ‘mixed mode’ database
–  Some parts using one data modelling convention and
other parts using another
–  This is bad practice with any modelling technique!
REQUIREMENTS OF A
DATA WAREHOUSE
DATA MODEL
Requirements
•  Uses A Design Pattern

–  General reusable approaches and solutions to
commonly occurring problems that can be used in
many different situations
•  Convention Over Configuration
–  Decrease the number of decisions that designers /
developers need to make, gaining simplicity, without
losing flexibility
–  Achieved by ensuring that tables and columns use a
standard structures, naming convention, etc. and are
populated and queried in a consistent fashion
Requirements
•  DRY (Don’t Repeat Yourself)

–  Reduce duplication because it:
•  Increases the difficulty of changing the model
•  Decreases the clarity of the model
•  Leads to opportunities for inconsistency
•  Static over a long period of time
–  No need to add or modify tables on a regular basis
–  Note: There is a difference between designed and
implemented, it is possible to have designed a table
but not to implement it until it is actually required
Requirements
•  The data model should store data at the lowest

possible level
–  Information stored at the transaction level
–  Avoid the storage of aggregates
•  Supports the best use of platform specific
features without compromising the design
–  Where available supports:
•  Partitioning
•  Column Storage
•  Many Insert/Few Update strategies
Requirements
•  Completely time-variant
–  It should be possible to reconstruct all information at
any point in time
•  Communication tool
–  Aids the refinement of requirements
–  Aids the explanation of possibilities
–  Develops confidence from the user
Requirements
•  Uses Standard BI Relational Databases

–  Ensure that the solution can be deployed on any
current platform and, if necessary, re-deployed on a
future platform
•  Process Neutral
–  It will not reflect past, current or planned business
processes, practices or dependencies
–  Stores the data items and relationships as defined by
their use at the point in time when the information is
created and acquired
CONSTRUCTING THE
DATA WAREHOUSE
DATA MODEL
Who is the customer?
•  Everyone has a
different definition
•  Everyone needs a
different information
•  Users have conflicting
definitions
•  Customer can be
individuals or
businesses
More problems …
•  Some of the customers are suppliers as well

•  Some businesses have separate divisions that
have to be handled separately
•  Some customers interact with different divisions
within our organisation
•  Some individuals or organisations also perform
other roles
–  e.g. legal, re-sellers, partners, etc.
The Party
•  These problem arises because the data is being

looked at in terms of current business process
•  In fact there is no customer entity, just different
types of party
–  Individuals, Organisations, Organisational Units
–  Concept of Party identical to that in contract law
•  The role of customer is defined not by the table
definition but by the usage of party data with
other information held (e.g. the purchase
transaction relating to a product)
Attributes of Party
•  The attributes of ‘Party’ will be those that remain

static over the life of the record
–  State ID Number, Name, Start Date, End Date
–  These attributes have ‘lifetime value’
•  Attributes that change need to be stored
elsewhere
•  The Party table needs to be categorised or typed
–  Individual, Organisation, Organisation Unit
PARTIES Data Model
PARTIES
• PARTY_DWK
• PARTY_ID
• PARTY_NAME
• PARTY_START_DATE
PARTY_TYPES • PARTY_END_DATE
• PARTY_TYPE_DWK • PARTY_TYPE_DWK
• PARTY_TYPE
• PARTY_TYPE_DESC
• PARTY_TYPE_GROUP
• PARTY_TYPE_START_DATE
• PARTY_TYPE_END_DATE
Supporting Non-Lifetime Attributes
•  Need to add data for different Party Types

–  Marital Status for Individuals
–  Number of Children for Individuals
–  Number of Employees for Organisations
–  Turnover for Organisations
•  Need to add data that changes over the lifetime
of the party
–  Usually the same attributes that are needed for
different Party Types
PARTY_PROPERTIES Data Model
PARTIES
• PARTY_DWK
• PARTY_ID
• PARTY_NAME
• PARTY_START_DATE PARTY_PROPERTIES
• PARTY_END_DATE • PARTY_DWK
• PARTY_TYPE_DWK • PARTY_PROPERTY_TYPE_DWK
• PARTY_END_DATE
PARTY_PROPERTY_TYPES • PARTY_PROPERTY_VALUE
• PARTY_PROPERTY_TYPE_DWK
• PARTY_PROPERTY_TYPE
• PARTY_PROPERTY_TYPE_DESC
• PARTY_PROPERTY_TYPE_GROUP
• PARTY_PROPERTY_TYPE_START_DATE
• PARTY_PROPERTY_TYPE_END_DATE
Relationships between Parties
•  Parties have relationships

–  David Walker works in Professional Services
–  David Walker is employed by Data Management &
Warehousing
–  David Walker is married to Helen walker
•  This is known as a Peer-To-Peer relationship
•  This is the first place that we see a role defined
by a relationship
PARTY_LINKS Data Model
PARTIES
• PARTY_DWK
• PARTY_ID
• PARTY_NAME
• PARTY_END_DATE PARTY_LINKS
• PARTY_TYPE_DWK • PARTY_DWK
• LINKED_PARTY_DWK
• PARTY_LINK_TYPE_DWK
PARTY_LINK_TYPES • PARTY_START_DATE
• PARTY_LINK_TYPE_DWK • PARTY_END_DATE
• PARTY_LINK_TYPE
• PARTY_LINK_TYPE_DESC
• PARTY_LINK_TYPE_GROUP
• PARTY_LINK_TYPE_START_DATE
• PARTY_LINK_TYPE_END_DATE
Segments of Parties
•  Grouping Parties together because at some

point in time they shared characteristics
•  This is known as a Peer Group Relationship
•  Examples
–  Married people with two or more children
–  IT companies with less than <100 employees
•  Usually generated by analysis and the results
stored
•  Most commonly seen in market segmentation
type applications
PARTY_SEGMENTS Data Model
PARTIES
• PARTY_DWK
• PARTY_ID
• PARTY_NAME
• PARTY_END_DATE PARTY_SEGMENTS
• PARTY_TYPE_DWK • PARTY_DWK
• PARTY_SEGMENT_TYPE_DWK
PARTY_SEGMENT_TYPES • PARTY_END_DATE
• PARTY_SEGMENT_TYPE_DWK
• PARTY_SEGMENT_TYPE
• PARTY_SEGMENT_DESC
• PARTY_SEGMENT_GROUP
• PARTY_SEGMENT_START_DATE
• PARTY_SEGMENT_END_DATE
Understanding The Conventions
•  All Type tables have the same format

–  Categorisation
•  All Property tables have the same format
–  Time Variant Attributes
•  All Link tables have the same format
–  Peer-To-Peer Relationships
•  All Segment tables have the same format
–  Peer Group Relationships
•  There are no other significant clusters of data about a
single entity such as Party
Introducing Major Entities
•  Party is a Major Entity

–  These are entities that exist regardless of the
business process
–  It is the relationships between major entities that are
defined by business processes
–  Major Entity attributes differ from one another
•  All Organisations only need a finite number of
major entities including:
–  Campaign –  Asset –  Geography
–  Account –  Channel –  Product/Service
–  Electronic Address –  Contract –  Calendar
Data Models For Other Major Entities
•  Geography
–  Geography Types
•  Postal Addresses, GPS Co-ordinates, ELR
–  Geographic Property Types
–  Geographic Properties
–  Geographic Link Types
–  Geographic Links
–  Geographic Segment Types
–  Geographic Segments
•  and so on for every major entity
Major Entity Sub Model
Major Entity Sub-Model
MAJOR ENTITY MAJOR ENTITY
PROPERTIES PROPERTY TYPES
MAJOR ENTITY
MAJOR ENTITY MAJOR ENTITY

LINKS LINK TYPES
MAJOR ENTITY MAJOR ENTITY MAJOR ENTITY

TYPES SEGMENTS SEGMENT TYPES
Relationships Between Major Entities
•  Storing names with multiple addresses and

multiple electronic addresses (e-mail, telephone
numbers, etc.)
–  Billing, Contact, Home, Work, etc
•  Usage
–  Party -> Contract -> Account -> Electronic Address ->
A Number -> Usage
–  Party -> Contract -> Account -> Electronic Address ->
B Number -> Usage
–  Product/Service -> Tariff - Usage
Party -> (Electronic) Addresses
PARTY_
ADDRESS_
HISTORY_
TYPES
PARTY_ ADDRESS
ADDRESS_
HISTORY
PARTY
PARTY_ ELECTRONIC
ELECTRONIC_ADDRESS_ ADDRESS
HISTORY
PARTY_
ELECTRONIC_ADDRESS_
HISTORY_
TYPES
Party -> Usage (Simplified)
PARTY CONTRACT ACCOUNT
PRODUCT SERVICE PRODUCT ACCOUNT

TARIFF SERVICE ELECTRONIC ADDRESS
HISTORY & TYPE HISTORY & TYPE
A Number
TARIFF ELECTRONIC
USAGE HISTORY ADDRESS
B Number
Extending the Data Model
•  Identify as many Major Entities as possible

–  But remember there are only a finite number so don’t
invent things for the sake of it
•  Define the standard sub-model around them
•  Put appropriate data in the sub-model
•  Create the relationships to _HISTORY tables for
the transaction the business wants to analyse
Does this help meet requirements?
  Uses A Design Pattern

  Convention Over Configuration
  DRY (Don’t Repeat Yourself)
  Static over a long period of time
  The data model should store data at the lowest possible
level
  Supports the best use of platform specific features
without compromising the design
  Completely time-variant
  Communication tool
Some Key Elements
•  Self Similar modelling

–  All _TYPE tables have the same structure, etc.
–  Naming conventions are consistent everywhere
–  Easy to create standard algorithms for load and extraction
–  Easy to partition on type and/or date
•  Insert ‘heavy’ / Update ‘light’
–  Most ETL will result in an insert, there will be very few updates
•  Manages ‘Slowly Changing Dimensions’
–  Inherent in the Major Entity Sub-Model design
–  Significantly reduces overhead in the Data Mart build
•  Data Driven
–  Types provide extensible metadata
–  Prevents un-necessary updating of the data model itself
•  Natural Star Schemas
–  Histories will map to FACTS,
–  Major Entity Collections will collapse into DIMENSIONS
Is this all there is to it ?
•  At a high level – YES

•  BUT:
–  There are methods for dealing with data quality
–  Special case methods for some lifetime attributes
•  e.g. Handling women changing their names at marriage
–  Insert/Update methods for performance
–  Design Patterns for implementation
–  Other detailed techniques
•  This talk could only ever be:
“An introduction to
Process Neutral Data Modelling”
Further Reading
•  Available From http://www.datamgmt.com

•  White Papers
–  Overview Architecture for Enterprise Data Warehouses
•  March 2006 - 32 pages
–  Data Warehouse Documentation Roadmap
•  April 2007 – 28 pages
–  How Data Works
•  June 2007 – 32 Pages
–  Data Warehouse Governance
•  April 2007 – 24 Pages
–  Data Warehouse Project Management
•  October 2008 – 32 Pages
–  Process Neutral Data Modelling
Thank you !!
Website: http://www.datamgmt.com
Phone: +44 7050 028 911
E-mail: davidw@datamgmt.com
Skype/MSN: datamgmt

ETIS08 - Process Neutral Data Modelling Concepts - Presentation

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

ETIS08 - Process Neutral Data Modelling Concepts - Presentation

Hochgeladen von

Copyright:

Verfügbare Formate

Data Management & Warehousing

PROCESS NEUTRAL DATA MODELLING

• The Issues With Conventional Data Warehouse

THE ISSUES WITH

• Data models take a long time to develop

• A struggle to keep up with rapidly changing

• Used in data warehouse

• Direct updates will be prohibited

• Uses A Design Pattern

• DRY (Don’t Repeat Yourself)

• The data model should store data at the lowest

• Uses Standard BI Relational Databases

• Some of the customers are suppliers as well

• These problem arises because the data is being

• The attributes of ‘Party’ will be those that remain

• Need to add data for different Party Types

• Parties have relationships

• Grouping Parties together because at some

• All Type tables have the same format

• Party is a Major Entity

MAJOR ENTITY MAJOR ENTITY

MAJOR ENTITY MAJOR ENTITY MAJOR ENTITY

• Storing names with multiple addresses and

PARTY CONTRACT ACCOUNT

PRODUCT SERVICE PRODUCT ACCOUNT

• Identify as many Major Entities as possible

 Uses A Design Pattern

• Self Similar modelling

• At a high level – YES

• Available From http://www.datamgmt.com

Das könnte Ihnen auch gefallen

•  The Issues With Conventional Data Warehouse

•  Data models take a long time to develop

•  A struggle to keep up with rapidly changing

•  Used in data warehouse

•  Direct updates will be prohibited

•  Uses A Design Pattern

•  DRY (Don’t Repeat Yourself)

•  The data model should store data at the lowest

•  Uses Standard BI Relational Databases

•  Some of the customers are suppliers as well

•  These problem arises because the data is being

•  The attributes of ‘Party’ will be those that remain

•  Need to add data for different Party Types

•  Parties have relationships

•  Grouping Parties together because at some

•  All Type tables have the same format

•  Party is a Major Entity

•  Storing names with multiple addresses and

•  Identify as many Major Entities as possible

  Uses A Design Pattern

•  Self Similar modelling

•  At a high level – YES

•  Available From http://www.datamgmt.com