Beruflich Dokumente
Kultur Dokumente
E-Book
such as real-time data integration. Read more about the pros and cons of hand-coding
Sponsored By:
Bolstering business intelligence with a sound data integration plan Table of Contents
E-Book
Sponsored by:
Page 2 of 15
Bolstering business intelligence with a sound data integration plan The real deal on data integration for business intelligence
other obstacles, see to that. BI done right requires carefully selected data sources, particular attention to data quality, and an understanding of the ultimate business uses of data.
experts. But successful data integration is no easy task. Poorly labeled data fields and siloed data sources, among
tive BI without good data integration or else youll be running off different sets of books and encountering various Ensuring a successful BI initiative requires attention to some key data integration considerations, according to
management at East Hanover, N.J.-based consulting firm Conversion Services International. You cant have effec-
Business intelligence is all about data integration, said William McKnight, senior vice president of information
McKnight and Rick Sherman, founder of Stow, Mass.-based business intelligence consulting firm Athena IT Solutions.
said. How they are answered will largely depend on a companys business model and corporate culture.
hold customer account? Questions like these must be addressed before integrating data into a BI system, McKnight
another member of that household makes a purchase, is he or she a new customer or part of the existing house-
times with their own IT departments or their own rogue IT departments. Before data can be integrated, he said,
business, McKnight said. That means there are different cuts of the data taken off in different departments, some-
Sponsored by:
Page 3 of 15
Bolstering business intelligence with a sound data integration plan The real deal on data integration for business intelligence
Athenas Sherman also stressed the importance of data quality in any integration initiative. Prior to the implementation of a BI system, he said, users in separate departments probably routinely make changes to data to suit their purposeschanges that are not relayed to other business units. Thats fine when the data never leaves a given They massage the data to make up for data quality issues, or gaps in the data, or things they dont think are department, but can wreak havoc on BI systems that access enterprise-wide data.
data, from across different systems. Thats when you start to see data quality issues.
right in the data, Sherman said. Once you [undertake] a BI initiative, then you want to look at more detailed
You need to pick data sources appropriately for the stakeholders of BI, McKnight said. And they should underUnderstanding how data will be used is essential to successful BI data integration, Sherman agrees, but underThe lifecycle of your data is very closely related to your business processes, Sherman said. Data used by a
manufacturing company, for instance, undergoes very different processes throughout its lifetime than data used integrate data for BI purposes during its lifecycle is a key success factor.
by a retailer. Even within industries, data lifecycles often vary. Both Sherman and McKnight said knowing when to Only after data source options are clear, data quality issues have been addressed, and BI business uses determined, can an organization actually begin integrating data with BI systems, McKnight and Sherman agreed. The reason why some BI projects take so long, sometimes nine months to a year or longer, is due to getting
data governance rules, Sherman said. Business intelligence really raises the bar for data integration.
business users to determine how theyre going to define and use the data, plus assuring data quality and writing
Sponsored by:
Page 4 of 15
Gophers burrow through life without seeing the havoc they create.
They cant help having tunnel vision. But you can. With proven business intelligence and analytic software from SAS.
www.sas.com/gophers
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 2008 SAS Institute Inc. All rights reserved. 00000US.0408
better-informed business-critical decisions. Experts agree that real-time data integration is gaining popularity but
the goal is the same: to communicate accurate, timely data from point A to point B in real time so users can make
with Stamford, Conn.-based Gartner Inc. Any given company is going to have data integration requirements that oriented mode, and there [are] going to be other things where real-time data integration really does have value.
Recognize that the world is not a black-and-white place, said Ted Friedman, vice president and distinguished analyst
span the latency spectrum. There are going to be pieces that are best suited to be delivered in a high-latency, batch-
Bloor Researchs Philip Howard explains: As a change is made to a database record in your transactional system, whatever youve got running, so its ready to answer real-time queries.
for instance, its also actively captured and fed through to your data warehouse or business intelligence system, or
CDC is used most often to synchronize operational applications and for real-time business intelligence (BI) purposes, according to Gartners Friedman. Indeed, business intelligence is a major driver of real-time data integration adoption, he said, especially among businesses that require BI reports at a moments notice.
is performing, [for example] Friedman said, then you need to be delivering data from some data sources to your
If youve got some type of short-cycle business and you need up-to-the-second analysis of how your supply chain
CDC is less ideal, however, if the goal is a comprehensive, real-time view of a single entity via data housed in multiple sources. For that, users more often turn to data federation, sometimes called enterprise information integration. Data federation is better suited to people that are looking at a more narrow slice of the data landscape, employeeas opposed to somebody whos doing historical trending in the data warehouse.
Friedman said. They want to get a complete view of a single instance of an entitya customer, a product, an
ogy to search multiple data sources to obtain a comprehensive view of that customer while still on the call. That
For example, an insurance agent on a customer call might use an application supported by data federation technol-
Sponsored by:
Page 6 of 15
Both the CDC and data federation markets are well established, Howard said, having already gone through the players like Teradata and GoldenGate Software, offer a variety of solid CDC and data federation real-time data integration tools, he said.
consolidation phase that you tend to get once products start to mature. Large vendors like IBMwhich acquired
data integration specialist DataMirror last yearand Oraclewhich scooped up Sunopsis in 2006as well as smaller
Friedman also identified a third approach, what he calls the messaging-middleware method, in which real-time data integration is achieved through middleware technologies that connect applications. Think of IBM WebSphere MQ and Microsoft BizTalk Server, and products like that, that are really meant to do
application in a low-latency fashion. So its sort of like the data replication idea, but working at the application layer as opposed to the database layer. This middleware approach is ideal for inter-enterprise scenarios, when theres a need for real-time data integration among organizations that may not have access to one anothers data sources, Friedman said. A vendor might communicate an important data change to a supplier in real time using this method, for instance.
of something meaningful that happened, and these technologies propagate that message to another system or
granular, message-oriented propagation of data, Friedman said. An application on one end spits out a message
cleanse that data, Friedman said. In the world of real-time data integration, theres less opportunity to apply integrated in real time will be of poorer quality, incorrect or misleading.
very sophisticated files for analyzing the quality and cleansing the data. There is a higher risk, then, that data
Friedman said current real-time data integration tools are better at data transformation and cleansing than theyve time data integration quality could be achieved, he said, as the problem is more technological than conceptual. been in the past, but there is still plenty of room for improvement. It is possible that someday near-perfect real-
organizations, and in some cases may even prove detrimental. Friedman advises users to match their data integrabasis, for example, would in that case have no need for real-time data integration, which could actually cause more harm than good, partly because of the already mentioned data quality concerns. tion method to their latency requirements. An organization that routinely analyzes certain data sets on a weekly
Both analysts said it is also important to recognize that real-time data integration isnt ideal for all companies and
Sponsored by:
Page 7 of 15
Organizational structure and corporate politics also play a role in determining the appropriateness of real-time data integration, Friedman said. If users arent ready to accept and use real-time data, theres little point in integrating data in real time in the first place.
decisions, the culture and the politics of the organization are not set up for them to act on real-time information, Friedman said. I think thats a limiting factor for many organizations today. Howard agreed, pointing to what he called decision-making latency. have a meeting with five other people and it takes two days to arrange that, or even two hours to arrange that, then you dont need real-time [data integration], Howard said.
Frankly, I know some companies that if they had real-time BI it wouldnt matter at all because the way they make
How soon can you as a human being make a decision based on new information that youre given? If you have to
He added: If you can make a decision instantly Ah, this has happened, therefore I know to do such-and-such
Sponsored by:
Page 8 of 15
Chickens are hypnotized by drawing a line in the dirt over and over.
They find it hard to step past the obvious. But you can. With proven business intelligence and analytic software from SAS.
www.sas.com/chickens
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 2007 SAS Institute Inc. All rights reserved. 00000US.04076
Bolstering business intelligence with a sound data integration plan Gartner data integration Magic Quadrant 2007: Platforms, market expand
forms, according to Ted Friedman, vice president and distinguished analyst with the Stamford, Conn.-based analyst
firm and co-author of the study. Big vendors are getting bigger through acquisition, as evidenced by IBMs purchase There are additions to the study this yearWoodcliff Lake, N.J.-based Syncsort Inc. and Hummingbird Connectivity, a division of Waterloo, Ontario-based Open Text Corp., as well as a host of other vendors that didnt meet the incluBuyers are demanding data integration platforms that support a range of different styles of data delivery, a trend
support ETL and at least one other integration style, like real-time data replication or data federation, Friedman tooling, metadata and runtime architectures, according to the study.
noted in last years study and mirrored in this years inclusion criteria. The study looked only at platforms that
said. Platforms are converging at a deeper technology level as well. That means theyre based on common design
capture (CDC), Friedman said. CDC tools help companies recognize and capture important changes that are occurit acquired Data Mirror, and Friedman has often been asked about it recently. ring in one data source and then propagate that changed data to another target. IBM talked up that feature when
The data integration method garnering more interest now is real-time data integration, particularly change data
The terminology thats emerging is event-driven approaches to data integration, Friedman said. Its growing of synchronizing data across different systems and databases.
more than that, people are interested in the technology because it enables a more time-effective, low-latency way Theres increased interest in data federation, too, though Friedman said inquiries about it are a distant third to stantial use of open source data integration tools on an enterprise-level scale, he said, if the freely downloaded which in the longer term could affect the market in a fairly dramatic way, he said. Though hes not seeing sub-
fairly substantially in popularity. Real-time, operational business intelligence is pushing some of that interestbut
questions about ETL or event-driven approaches. And buyers are also asking about open-source data integration,
tools mature to offer more functionality, commercial vendors may begin to feel some pricing pressure.
Sponsored by:
Page 10 of 15
Bolstering business intelligence with a sound data integration plan Gartner data integration Magic Quadrant 2007: Platforms, market expand
Enterprise data integration tool vendor rankings 2007: Leaders lead, others jockey for position
Informatica. Both were in positions similar to those of last year, thanks to continuing to meet market demands with The leaders quadrant of the study, for vendors with high marks for vision and execution, hosted IBM and
Business Objects, Microsoft and Oracle all appeared in the challengers quadrant, for vendors with less vision but said, these vendors need to do more to articulate a comprehensive vision.
strong customer execution. The latter two have vision challenges, Friedman said. While their position as challengers is largely a result of increased customer adoption of SQL Server Integration Services and Oracle Data Integrator, he
its breadth of integration capabilities and vision, particularly around the role of data integration in service-oriented architectures, Friedman said. SAS Institute was also in the visionaries quadrant with a strong, versatile platform.
this year. iWay, a subsidiary of business intelligence vendor Information Builders, made it into this section thanks to
The visionaries quadrant, for vendors with good vision but not as much execution experience, had just two vendors
from its position in the challengers section last year. Pervasive has a strong product with a mix of capabilities, and the cost-to-functionality ratio is good, Friedman said, but the vendor didnt have enough enterprise-class implementations or execution experience to be in the challengers quadrant this year. Others in the niche quadrant were Cognos, Friedman is interested to see how SAPs acquisition of Business Objects affects the space, since both vendors have Sybase, Tibco Software, SAP, ETI, Sun, Pitney Bowes Group 1 Software, Syncsort, and Hummingbird Connectivity.
still met the studys inclusion criteria. Near the top of the section was Pervasive Software, which dropped back a bit
Most vendors were clustered in the niche quadrant, reserved for those not as strong on vision and execution but which
integration products and little is known yet about how they may rationalize these capabilities.
Organizations should also look for strong metadata management and modeling functions, Friedman said. Basic intethat can help them truly understand data assets, relationships between data, and how data changes over time. tures (SOA), Friedman said. gration capabilities may become commoditized in the long term, he said, so organizations should align with vendors
Finally, buyers should seriously think about the importance of data and data integration in service-oriented architec-
find thats a concept that too few organizations are thinking about today.
Consider how these types of tools can be used to help facilitate and add value in your SOA work, he said. We
Sponsored by:
Page 11 of 15
Need to turn your information stampede into a logical path for success? You can.
Find your direction with assets like this using the SAS Platform Pathfinder. www.sas.com/pathfinder1
Bolstering business intelligence with a sound data integration plan Data integration software vs. hand coding: Balancing costs and benefits
Data integration software vs. hand coding: Balancing costs and benefits
By Hannah Smalltree, SearchDataManagement.com Senior Site Editor As organizations consider investing in data integration software, they encounter a familiar question: When and why does it make sense to buy technology to do something that you could do manually? Organizations have long relied on in-house developers to hand-code scripts for data integration projects. Yet, for Canadian retailer Home Hardware Stores Ltd. tion platforms. But many companies still use hand coding for data integration projectsincluding, until recently,
almost 20 years, theyve had alternatives in the form of extract, transform and load (ETL) tools and data integra-
pany to consider data integration software, according to Will Buddell, senior systems analyst with the St. Jacobs, was no small integration task.
Two years ago, when Home Hardware was upgrading its merchandising system, a consultant encouraged the comOnt.-based retailer. The hardware, building materials and furniture retailer had to merge its product database with
an acquired companys product database. With about 100,000 different products, combining two Oracle databases
from Austin, Texas-based Pervasive Software Inc. But first, Home Hardware ran a comparison test.
would provide productivity gains, according to the consultant. The team zeroed in on the Data Integrator software
Though Home Hardware had the in-house skills necessary to manually hand-code scripts, data integration software
We had an internal development team that produced a test XML file. Time on the books for them to produce it was used that as the basis to go forward, because it showed that we could reduce our development time drastically. Hand coding wouldnt have been difficult, Buddell saidjust time-consuming. In contrast, it took only a day or so to target mapping made it especially user-friendly, he said. When it was actually time to merge the data for phase one not as long as hand coding would have taken, Buddell said. Plus, time is money. Home Hardware estimates that the of the project, it only took about two hours. Integration for other phases of the project sometimes took longer, but set up the Data Integrator software. The pre-built database connectors and a drag-and-drop interface for source-toabout 20 hours, Buddell said. With [Data Integrator], we produced the same XML file in about four hours. We
software helped it save anywhere from $200,000 to $500,000 (Canadian dollars) during the project, he said.
repeatable functions are coded into the software, so that companies dont have to reinvent the wheel every time they do an integration project. Thats not all.
Sponsored by:
Page 13 of 15
Bolstering business intelligence with a sound data integration plan Data integration software vs. hand coding: Balancing costs and benefits
aged for version control, Sherman said. With hand coding, you have code all over the place, little management and little documentation.
An ETL tool or data integration platform means that you have all of the code in one place, documented and man-
And then there are issues like auditability and transparency, since regulations like SOX or HIPAA demand that companies be able to trace data back to their original source systems. Data integration software helps by automatically documenting information about data sources and transformations, making for an easier audit trail.
Thats a problem when companies need to replicate integration procedures, troubleshoot or change code, he said.
robust integration platform can be a six-figure proposition for a large company, he explained. Thats because the Add this all up and many companies see in-house development as the path of least resistance.
The top inhibitor is likely cost, Sherman said. While available ETL tools run the functionality and pricing gamut, a
scale of the project affects the price. Data integration software is often either priced by concurrent CPU usage or by the number of source and target systems. Beyond licensing costs, training is often required for platforms, he said.
an ETL tool.
mer leaveswhos going to maintain that persons code? If you worry about what you would do, you probably need
Theres seemingly no cost to coding, just the person you have doing it, Sherman said. But, when that program-
productivity gains of software. Systems accessed by many business users, or applications that will be expanded
candidate. For example, a business intelligence system, with data loaded on a regular basis, could benefit from the
not a straightforward ROI equation. But the soft benefitsespecially around auditability and transparencyare often compelling enough on their own to convince CFOs and CIOs to sign off on purchases, Sherman said.
Sponsored by:
Page 14 of 15
Bolstering business intelligence with a sound data integration plan Resources from SAS
successful 2007 Competing on Analytics event tour, this compilation features analytics expert, Thomas H. Davenport and highlights his groundbreaking findings. Pick up best practices and experiences from executives on how their organizations are building effective competitive strategies around data driven insights and trumping their rivals. Online Assessment: Platform Pathfinder ages and uses information. Your journey includes: organization.
Want to learn how specific organizations are using analytics? You can. Derived from summary reports of the hugely
Do you want to turn your information stampede into a logical path for success? You can. SAS invites you to visit our Platform Pathfinderan innovative Web site that provides an opportunity to determine how your organization man A five-minute Business Intelligence Assessment survey
An objective look at the current roles of analytics, business intelligence and data integration in your demos. Learning resources tailored to your survey responses, including white papers, Webcasts and product
Discover more about the organizations who use SAS. You can. SAS customers represent many of the most Global 500 List.
innovative and successful organizations in the world, including 96 of the top 100 companies on the 2007 FORTUNE
About SAS
software to improve performance through insight from data, resulting in faster, more accurate business decisions; SAS is the leader in business intelligence and analytical software and services. Customers at 43,000 sites use SAS
more profitable relationships with customers and suppliers; compliance with governmental regulations; research business intelligence applications within a comprehensive enterprise intelligence platform. Since 1976, SAS has been giving customers around the world The Power to Know. http://www.sas.com/
breakthroughs; and better products and processes. Only SAS offers leading data integration, storage, analytics and
Sponsored by:
Page 15 of 15