Beruflich Dokumente
Kultur Dokumente
BI INTEGRATION
CHALLENGES:
DATA ERRORS,
BIG DATA AND
REAL-TIME NEEDS
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
PA G E 2 O F 1 2
SPONSORED BY
Alan Earls
Business intelligence (BI) systems and their supporting data warehouses are
only as good as the data that goes into them. And if you arent properly handling
the BI data integration process, your end users -- and ultimately, your organization -- may be in for trouble.
With BI tools becoming more and more pervasive in organizations, and
more critical to the success of business operations, making sure that you have
a well-designed and well-executed process for integrating BI data is of paramount importance, according to data management analysts such as Ted Friedman of Gartner Inc.
Friedman said Gartner sees data integration challenges related to BI as a
drag on the success of BI and analytics initiatives -- and a big reason for outright
project failures.
As the data that organizations are trying to harness gets more and more
complex, with more kinds and sources of data and now big data thrown into
PA G E 3 O F 1 2
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
the mix, a significant amount of time and effort is involved in matching, cleaning and preparing data for BI applications, he said. Its a darned hard problem,
particularly when you add in older, legacy systems where you sometimes need
to do archaeology first in order to interpret the data.
Another complicating factor is that things are changing in the world of data
integration technology as business users demand faster access to BI data.
ETL STILL BEST BET FOR BI DATA INTEGRATION?
The traditional workhorse technology for managing BI data integration is extract, transform and load (ETL) software that pulls data from source systems
in bulk batch processing jobs. Friedman said newer data integration techniques
offer lower latency than ETL tools do. For example, change data capture software and other real-time data integration tools let you push new or modified
information to data warehouse and BI systems in real or near real time, which
can be particularly useful for tasks like fraud detection. It is streaming [data]
in granular form rather than big chunks in batch, which is what ETL is using,
he said.
Another option: federated and virtualized approaches to data integration
and delivery that dont move the data out of source systems at all but instead
PA G E 4 O F 1 2
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
create consolidated views of data from multiple sources for BI uses. With data
virtualization tools, the integrated data doesnt persist anywhere, Friedman
said. Youre grabbing it in real time and joining it together and making it seem
as if it is one database somewhere to the applications using it.
Despite the emergence of this new wave of data integration and delivery
tools, though, Friedman thinks it would be a mistake to view ETL software as
obsolete or no longer valuable. ETL is still relevant, he said. We think there
will always be a role for ETL-style processing because not all data can or should
be delivered in real time.
Indeed, Friedman warned that data integration vendors are pushing sexy
real-time options for BI data integration when many organizations can still get
what they need from a batch approach. Real-time [integration] costs money
and it requires a change from what organizations have been doing, so there
needs to be a strong business case for it, he said.
ETL still has a role -- it is the heavy lifter of data integration, agreed Claudia Imhoff, president of Intelligent Solutions Inc., a consultancy in Boulder,
Colo. Still, she noted that its newer competitors can be more flexible and faster
to deploy and are better suited to delivering timely data to business users for
operational BI applications.
PA G E 5 O F 1 2
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
should look ahead and make sure theyre prepared for the data integration challenges ahead, added Kobielus, who has since taken a job at IBM. You need to
be ready, he said, for things like massive data inputs from social media and
start to budget and staff up.
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
users need real- or near-real-time access to data, is a must for BI and data integration teams, Imhoff said.
Good data quality is just as important, according to Imhoff, who said that
correcting and cleaning up bad data shouldnt be a function solely of the BI data
integration process. Errors are happening everywhere else along the way, so
you need to figure out where they are coming from, she said -- and then work
to prevent data mistakes from finding their way into source systems in the first
place. In effect, Imhoff added, data integration and BI professionals are given
the job of consolidating faulty data and then get the blame when it isnt perfect.
We need to get people to understand that they shouldnt just shoot the messenger, she said.
Ted Friedman, an analyst at Gartner Inc. in Stamford, Conn., thinks that
not paying enough attention to data quality is the biggest BI data integration
danger companies face. Ive been following data integration for more than
10 years, he said. And I still spend days talking to organizations that are not
getting the usage and trust and acceptance and value out of their BI efforts
because the quality of the data is not good enough, and they havent done the
right things to fix that.
Data quality problems clearly affect more than BI data in wayward
PA G E 8 O F 1 2
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
organizations, Friedman said, but he sees poor data quality as one of the primary barriers to successful BI programs. The shortcomings, he added, typically
result from not focusing [on data quality] early and often enough, and simply
not doing enough to mitigate quality issues as information is moved into data
warehouses.
James Kobielus, who was an analyst at Forrester Research Inc. in Cambridge, Mass., before taking a job with a technology vendor earlier this year,
also pointed to missteps on data quality as a common source of trouble for BI
data integration efforts.
Organizations think they can simply load data from their various back-end
applications into a data warehouse and it will be usable without cleansing it or
doing match-and-merge or transform [processes], Kobielus said while he was
still at Forrester. But doing so sets up companies for some nasty surprises, he
added. For example, they end up with six records on the same person and dont
know which one is the right one, Kobielus said.
BI DATA INTEGRATIONS DRAMATIC EFFECT
Another big source of inconsistent data, and drama, stems from internal debates over what constitutes a system of record, said Jill Dyche, co-founder of
PA G E 9 O F 1 2
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
Baseline Consulting in Sherman Oaks, Calif. For example, she noted, there can
be arguments about which transaction system should be used as the source of
customer addresses. Such conversations often then turn to the definition of
address: Is it a customers billing address or shipping address, or its headquarters location if that differs from the other two?
Thats when the arguments ensue and business people become disaffected
with the BI teams ability to understand and deliver the right data, Dyche said.
So then someone just decides to forklift everything into a single database,
which the business people then refuse to use.
Barry Devlin, founder of 9sight Consulting in Cape Town, South Africa,
thinks the most problematic mistake is not including the right people in the
process of crafting a BI data integration strategy and plan. The people who
really understand data and what it means are a particular subset of the business community who have been playing with data over the years -- they are
the gurus and the power users, Devlin said. As a result, he added, theyre best
equipped to define what data needs to be integrated in order to create effective
BI applications.
But in many cases, its left to the IT department to develop the data integration plan in addition to doing the implementation work, Devlin said. While IT
PA G E 1 0 O F 1 2
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
PA G E 1 1 O F 1 2
SPONSORED BY
Home
BI data integration
process challenged
by real-time needs,
big data
Data errors, other
missteps can waylay
BI data integrationstrategy
PA G E 1 2 O F 1 2
SPONSORED BY